Speech recognition method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a voice recognition method and device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, after current audio stream data is acquired and segmented to obtain multiple segments of target audio data, feature extraction is performed on the target audio data to obtain acoustic features of the current audio stream data, and an initial text corresponding to the current audio stream data is generated based on the acoustic features; the method comprises the following steps: detecting at least one text key frame in current audio stream data, identifying a segmentation position of at least one audio data block in the current audio stream data based on the text key frame, then segmenting acoustic features according to the segmentation position to obtain a target acoustic feature of each audio data block, and obtaining a target acoustic feature of each audio data block based on the target acoustic features. Updating the initial text to obtain a target text corresponding to the current audio stream data; according to the scheme, the accuracy of speech recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition method, device, electronic device and computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technology, the recognition methods for streaming speech have become more and more diverse. In order to improve the recognition efficiency and accuracy of streaming speech, speech recognition can be performed through a cascade recognition method. The current cascade speech recognition often recognizes the initial text of the streaming speech through a real-time recognition network, recognizes the target text of the streaming speech through a non-real-time recognition network, and refreshes the target text to overwrite the initial text.

[0003] In the process of research and practice of current technology, the inventor of the present application found that both real-time recognition networks and non-recognition networks extract acoustic features by segmenting the data into chunks with fixed window lengths. However, the segmentation of fixed chunks may be in the middle of a word, resulting in inaccurate recognition of the boundaries of each chunk, thereby affecting the recognition rate of the entire sentence, and thus resulting in a low accuracy rate in speech recognition. Summary of the invention

[0004] Embodiments of the present invention provide a speech recognition method, device, electronic device, and computer-readable storage medium, which can improve the accuracy of generating labels.

[0005] A speech recognition method, comprising:

[0006] Acquire current audio stream data, and segment the current audio stream data to obtain multiple segments of target audio data;

[0007] Performing feature extraction on the target audio data to obtain acoustic features of the current audio stream data, and generating initial text corresponding to the current audio stream data based on the acoustic features;

[0008] Detecting at least one text key frame in the current audio stream data, and identifying a segmentation position of at least one audio data block in the current audio stream data based on the text key frame;

[0009] Segmenting the acoustic feature according to the segmentation position to obtain a target acoustic feature of each audio data block;

[0010] Based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

[0011] Accordingly, an embodiment of the present invention provides a speech recognition device, comprising:

[0012] An acquisition unit, used to acquire current audio stream data and segment the current audio stream data to obtain multiple segments of target audio data;

[0013] A generating unit, configured to extract features from the target audio data to obtain acoustic features of the current audio stream data, and generate an initial text corresponding to the current audio stream data based on the acoustic features;

[0014] an identification unit, configured to detect at least one text key frame in the current audio stream data, and identify a segmentation position of at least one audio data block in the current audio stream data based on the text key frame;

[0015] A segmentation unit, configured to segment the acoustic feature according to the segmentation position to obtain a target acoustic feature of each audio data block;

[0016] An updating unit is used to update the initial text based on the target acoustic feature to obtain a target text corresponding to the current audio stream data.

[0017] In some embodiments, the recognition unit can be specifically used to predict the posterior probability of the text content corresponding to each audio frame based on the acoustic features; detect the peak position of at least one peak in the current audio stream data according to the posterior probability; and filter out at least one audio frame corresponding to the peak position in the audio frame to obtain the text key frame corresponding to each peak.

[0018] In some embodiments, the identification unit can be specifically used to group the peaks according to the peak positions to obtain multiple peak groups, wherein the peak groups include a preset number of peaks; based on the text key frames, identify the frame number interval of the audio data block corresponding to each peak group in the current audio stream data; amplify the frame number interval to obtain a target frame number interval, and use the target frame number interval as the segmentation position of the audio data block.

[0019] In some embodiments, the identification unit can be specifically used to screen out a starting peak and an ending peak in the peak group according to the peak position; identify the frame number of the text key frame corresponding to the starting peak in the current audio stream data to obtain the starting frame number; identify the frame number of the text key frame corresponding to the ending peak in the current audio stream data to obtain the ending frame number; and use the interval formed by the starting frame number and the ending frame number as the frame number interval of the audio data block corresponding to the peak group.

[0020] In some embodiments, the identification unit can be specifically used to subtract a preset first frame number from the starting frame number to obtain a target starting frame number; add a preset second frame number to the ending frame number to obtain a target ending frame number; and use the interval formed by the target starting frame number and the target ending frame number as the target frame number interval.

[0021] In some embodiments, the segmentation unit can be specifically used to filter out at least one audio frame corresponding to the target frame number interval in the audio frame, and obtain a set of candidate audio frames corresponding to each audio data block; extract the acoustic sub-features corresponding to each audio frame in the candidate audio frame set from the acoustic features, and obtain a set of acoustic sub-features corresponding to each audio data block; and use the set of acoustic sub-features as the target acoustic features corresponding to the audio data block.

[0022] In some embodiments, the update unit can be specifically used to determine the target field corresponding to each audio data block based on the target acoustic features; splice the target fields according to the segmentation position to obtain a spliced ​​text; replace the initial text with the spliced ​​text to obtain the target text corresponding to the current audio stream data.

[0023] In some embodiments, the generation unit can be specifically used to extract features of the target audio data using a speech recognition model to obtain initial acoustic features corresponding to each target audio data; and concatenate the initial acoustic features to obtain acoustic features corresponding to the current audio stream data.

[0024] In some embodiments, the speech recognition device may further include a training unit, which may be specifically used to obtain multiple audio data samples corresponding to the speech sample, and use a preset speech recognition model to perform feature extraction on the audio data sample to obtain sample acoustic features of the speech sample; based on the sample acoustic features, predict the text content corresponding to the speech sample to obtain a predicted text; based on the predicted text, determine the recognition loss of the speech sample, and based on the recognition loss, converge the preset speech recognition model to obtain a speech recognition model.

[0025] In some embodiments, the training unit can be specifically used to obtain an initial speech recognition model, which includes a time series classification network; when the time series classification network includes a real-time time series classification network and a non-real-time time series classification network, the initial speech recognition model is used as a preset speech recognition model; when the initial speech recognition model is a pre-trained model and the time series classification network is the non-real-time time series classification network, the non-real-time time series classification network is copied, and the copied non-real-time time series classification network is added to the initial speech recognition model as a real-time time series classification network to obtain a preset speech recognition model.

[0026] In some embodiments, the training unit can be specifically used to predict the text content of the speech sample based on the sample acoustic features using a real-time time series classification network to obtain real-time predicted text, and identify the sample segmentation position of at least one sample audio data block in the speech sample; according to the sample segmentation position, the sample acoustic features are segmented to obtain the target sample acoustic features corresponding to each sample audio data block; based on the target sample acoustic features, the non-real-time time series classification network is used to predict the text content of the speech sample to obtain the current predicted text, and the real-time predicted text and the current predicted text are used as predicted text.

[0027] In some embodiments, the training unit can be specifically used to detect at least one sample text key frame corresponding to a sample peak in the speech sample based on the sample acoustic features; determine the target number of groups corresponding to the sample peak in a preset number set, and group the sample peaks based on the target number of groups to obtain at least one sample peak group; and identify the sample segmentation position of the sample audio data block corresponding to each sample peak group in the speech sample based on the sample text key frame.

[0028] In some embodiments, the training unit can be specifically used to extract text features of multiple dimensions from the acoustic features of the target sample; based on the text features, use the non-real-time time series classification network to predict the text content of the speech sample to obtain non-real-time predicted text; splice the text features to obtain the current text features corresponding to each sample audio data block, and determine the attention weight of the current text features; based on the attention weight, determine the text content corresponding to the current text features to obtain auxiliary predicted text, and use the auxiliary predicted text and the non-real-time predicted text as the current predicted text.

[0029] In some embodiments, the training unit can be specifically used to obtain the annotated text of the speech sample, and compare the annotated text with the real-time prediction text to obtain the real-time prediction loss; compare the annotated text with the non-real-time prediction text to obtain the non-real-time prediction loss, and compare the annotated text with the auxiliary prediction text to obtain the auxiliary prediction loss; merge the real-time prediction loss, the non-real-time prediction loss and the auxiliary prediction loss to obtain the recognition loss of the speech sample.

[0030] In some embodiments, the training unit can be specifically used to obtain loss fusion weights, and based on the loss fusion weights, weight the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss respectively; calculate the average of the weighted real-time prediction loss and the weighted non-real-time prediction loss to obtain the target prediction loss; fuse the target prediction loss with the weighted auxiliary prediction loss to obtain the recognition loss of the speech sample.

[0031] In some embodiments, the training unit can be specifically used to segment the speech sample by a preset audio mask and a preset number of segmentation frames to obtain multiple sample audio frame sets; filter out at least one historical sample audio frame corresponding to each sample audio frame set from the sample audio frames; add the historical sample audio frames to the corresponding sample audio frame set to obtain multiple target sample audio frame sets, and use the target sample audio frame sets as audio data samples.

[0032] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to implement the speech recognition method provided by the embodiment of the present invention.

[0033] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the speech recognition methods provided in the embodiments of the present invention.

[0034] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the speech recognition method provided in the embodiment of the present application.

[0035] The embodiment of the present invention obtains the current audio stream data, divides the current audio stream data, obtains multiple segments of target audio data, extracts features from the target audio data to obtain acoustic features of the current audio stream data, generates initial text corresponding to the current audio stream data based on the acoustic features, detects at least one text key frame in the current audio stream data, and identifies the segmentation position of at least one audio data block in the current audio stream data based on the text key frame, then divides the acoustic features according to the segmentation position to obtain the target acoustic features of each audio data block, updates the initial text based on the target acoustic features, and obtains the target text corresponding to the current audio stream data; because the scheme can not only identify the initial text, but also detect at least one text key frame in the current audio stream data during the real-time recognition of the corresponding streaming speech, and then identifies the segmentation position of the audio data chunk in the current audio stream data through the text key frame, thereby improving the accuracy of chunk boundary recognition, and further improving the recognition rate of the sentence corresponding to the audio data chunk, and therefore, can improve the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a schematic diagram of a scenario of a speech recognition method provided by an embodiment of the present invention;

[0038] Figure 2 is a flow chart of a speech recognition method provided by an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram comparing the chunk-based cascade model conversion and the preset speech recognition model provided by an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of the network structure of the speech recognition model provided by an embodiment of the present invention;

[0041] Figure 5 is another flowchart of the speech recognition method provided by an embodiment of the present invention;

[0042] Figure 6 is a structural diagram of a speech recognition device provided by an embodiment of the present invention;

[0043] Figure 7 is another structural schematic diagram of a speech recognition device provided by an embodiment of the present invention;

[0044] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] The embodiment of the present invention provides a speech recognition method, device, electronic device and computer-readable storage medium. The speech recognition device can be integrated in an electronic device, which can be a server or a terminal.

[0047] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0048] For example, see Figure 1 Taking the speech recognition device integrated in an electronic device as an example, after the electronic device obtains the current audio stream data and segments the current audio stream data to obtain multiple segments of target audio data, it extracts features of the target audio data to obtain the acoustic features of the current audio stream data, and based on the acoustic features, generates an initial text corresponding to the current audio stream data, detects at least one frame of text keyframes in the current audio stream data, and based on the text keyframes, identifies the segmentation position of at least one audio data block in the current audio stream data, and then, according to the segmentation position, segments the acoustic features to obtain the target acoustic features of each audio data block, and based on the target acoustic features, updates the initial text to obtain the target text corresponding to the current audio stream data, thereby improving the accuracy of speech recognition.

[0049] It can be understood that in the specific implementation of the present application, when the audio stream data or voice sample and other related data of the object are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0050] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0051] This embodiment will be described from the perspective of a speech recognition device, which can be specifically integrated into an electronic device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device, or other smart devices that can perform speech recognition.

[0052] A speech recognition method, comprising:

[0053] The current audio stream data is obtained, and the current audio stream data is segmented to obtain multiple segments of target audio data, and features are extracted on the target audio data to obtain acoustic features of the current audio stream data, and based on the acoustic features, an initial text corresponding to the current audio stream data is generated, at least one frame of text keyframes is detected in the current audio stream data, and based on the text keyframes, a segmentation position of at least one audio data block is identified in the current audio stream data, and according to the segmentation position, the acoustic features are segmented to obtain target acoustic features of each audio data block, and based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

[0054] like Figure 2 As shown, the specific process of the speech recognition generation method is as follows:

[0055] 101. Obtain current audio stream data, and segment the current audio stream data to obtain multiple segments of target audio data.

[0056] The current audio stream data may be audio stream data that needs to be voice recognized. The so-called audio stream data may be understood as streaming audio data or audio stream. The audio stream data may include multiple audio frames that are continuous or last for a period of time.

[0057] There are many ways to obtain the current audio stream data, which can be as follows:

[0058] For example, audio stream data uploaded by a terminal or client may be received to obtain the current audio stream data, or the sound or audio emitted by a target object may be collected in real time by a sound collection device to obtain the current audio stream data, or multiple consecutive audio frames may be obtained from the Internet or an audio stream database to obtain the current audio stream data, and so on.

[0059] After obtaining the current audio stream data, the current audio stream data can be segmented to obtain multiple segments of target audio data. There are many ways to segment the current audio stream data. For example, the current audio stream data can be segmented based on a preset segmentation frame number to obtain multiple audio frame sets, and at least one historical audio frame corresponding to each audio frame set is screened out from the audio stream data, and the historical audio frame is added to the corresponding audio frame set to obtain multiple target audio frame sets, and the target audio frame sets are used as target audio data.

[0060] There are multiple ways to segment the current audio stream data based on the preset segmentation frame number. For example, when receiving the audio frames in the current audio stream data, the number of received audio frames can be counted. When receiving the audio frames with the preset segmentation frame number, the audio frames with the preset segmentation frame number can be taken as an audio frame set, and then the counting of the number of received audio frames is returned until all the audio frames in the current audio stream data are received, thereby obtaining multiple audio frame sets. Alternatively, when receiving the audio frames corresponding to the preset segmentation frame number in the current audio stream data, these audio frames can be taken as an audio frame set, and then, the same process is repeated until all the audio frames in the current audio stream data are received, thereby obtaining multiple audio frame sets. For example, the current audio stream data includes N audio frames (X={X1,…,X N}), taking the preset number of segmentation frames as m frames as an example, the segmented audio frame set may be X[t, t+m].

[0061] After the current audio stream data is segmented based on the preset segmentation frame number, at least one historical audio frame corresponding to each audio frame set can be screened out in the audio stream data. The historical audio frame can be at least one audio frame before the first audio frame in the audio frame set. There can be multiple ways to screen out at least one historical audio frame corresponding to each audio frame set in the audio stream data. For example, the first audio frame is screened out in the audio frame set as the target audio frame, and a preset number of audio frames before the target audio frame are screened out in the audio stream data, so as to obtain the historical audio frames corresponding to each audio frame set. For example, taking the audio frame set as X[t, t+m] and the preset number as r, the target audio frame can be the tth frame, and the historical audio frames corresponding to the target audio frame can be the (tr)th to (t-1)th frames.

[0062] After filtering out the historical audio frames corresponding to each audio frame set, the historical audio frames can be added to the corresponding audio frame set to obtain multiple target audio frame sets, and the target audio frame sets are used as target audio data to obtain multiple segments of target audio data. For example, taking the audio frame set as X[t, t+m], the historical audio frames can be frames (tr) to (t-1), and the target audio frames can be [tr, t+m].

[0063] 102. Perform feature extraction on the target audio data to obtain acoustic features of the current audio stream data, and generate an initial text corresponding to the current audio stream data based on the acoustic features.

[0064] The acoustic feature may be feature information representing acoustics in the current audio stream data.

[0065] The initial text may be text obtained by performing real-time speech recognition on the current audio stream data based on acoustic features.

[0066] The target audio data is feature extracted to obtain the acoustic features of the current audio stream data, and the initial text corresponding to the current audio stream data is generated based on the acoustic features. Specifically, the method may be as follows:

[0067] S1. Extract features of target audio data to obtain acoustic features of current audio stream data.

[0068] For example, a speech recognition model may be used to extract features from target audio data to obtain initial acoustic features corresponding to each target audio data, and the initial acoustic features may be concatenated to obtain acoustic features corresponding to the current audio stream data.

[0069] Among them, the speech recognition model may include a real-time coding network (Streaming Encoder). There may be multiple ways to use the speech recognition model to extract features of the target audio data. For example, a real-time coding network may be used to perform feature encoding on audio frames in the target audio data to obtain audio features corresponding to each audio frame in the target audio data, and use the audio features as initial acoustic features corresponding to the target audio data.

[0070] There may be multiple network structures of the real-time coding network, for example, it may include Causal Conformer (a speech coding network), or other network structures that can perform audio coding or speech coding, etc. Taking the real-time coding network as Causal Conformer as an example, the initial acoustic feature corresponding to the target audio data may be C[t,t+m], which may be shown in formula (1), specifically as follows:

[0071] C[t, t+m] = Causal Conformer Encoder( X[tr, t+m] ) (1)

[0072] Among them, C[t,t+m] is the initial acoustic feature and X[tr,t+m] is the target audio data.

[0073] After the initial acoustic features are extracted from the target audio data, the initial acoustic features can be spliced ​​to obtain the acoustic features of the current audio stream data. There are many ways to splice the initial acoustic features. For example, the initial acoustic features can be sorted based on the timing information of the target audio data in the current audio stream data, and based on the sorting results, the initial acoustic features can be spliced ​​to obtain the acoustic features of the target audio stream data. For example, taking the current audio stream data including N audio frames and the initial acoustic features being C[t,t+m] as an example, the initial acoustic features can be spliced ​​to obtain the acoustic features C={C1,…,C N}.

[0074] Optionally, the speech recognition model may be pre-set by an operation and maintenance personnel, or may be obtained by self-training of the speech recognition device, that is, after the step of "using the speech recognition model to extract features from the target audio data to obtain initial acoustic features corresponding to each target audio data", the speech recognition method includes:

[0075] Acquire multiple audio data samples corresponding to the speech samples, and use a preset speech recognition model to extract features from the audio data samples to obtain sample acoustic features of the speech samples. Based on the sample acoustic features, predict the text content corresponding to the speech samples to obtain predicted text. According to the predicted text, determine the recognition loss of the speech samples, and based on the recognition loss, converge the preset speech recognition model to obtain a speech recognition model. Specifically, the method may be as follows:

[0076] (1) Acquire multiple audio data samples corresponding to the speech samples, and use a preset speech recognition model to extract features from the audio data samples to obtain sample acoustic features of the speech samples.

[0077] The speech sample includes multiple sample audio frames. There are many ways to obtain multiple audio data samples corresponding to the speech sample, which can be as follows:

[0078] For example, the speech sample can be segmented by a preset audio mask and a preset number of segmentation frames to obtain multiple sample frame sets, at least one historical sample audio frame corresponding to each sample audio frame set can be screened out from the sample audio frames, the historical sample audio frames can be added to the corresponding sample audio frame set to obtain multiple target sample audio frame sets, and the target sample audio frame sets can be used as audio data samples.

[0079] Among them, the preset audio mask can be understood as a mask (mask or mask) with a fixed length of m frames, where m is the preset number of segmentation frames. By using the preset audio mask, m frames of sample audio frames can be segmented out from the audio frame of the speech sample, thereby obtaining a set of sample audio frames. The role of the preset audio mask here is to simulate the segmentation of audio stream data, because the segmentation of audio stream data cannot predict the audio frames in the future. The specific segmentation process is similar to the process of segmenting audio stream data, as described above, and will not be repeated here.

[0080] After segmenting the speech sample, at least one historical sample audio frame corresponding to each sample audio frame set can be screened out from the sample audio frame. The method of screening the historical sample audio frame is similar to the method of screening the historical audio frame, as described above, and will not be repeated here.

[0081] After filtering out at least one historical sample audio frame corresponding to each sample audio frame set in the sample audio frames, the historical sample audio frame can be added to the corresponding sample audio frame set to obtain multiple target sample audio frame sets, and the target sample audio frame sets are used as audio data samples.

[0082] After obtaining multiple audio data samples corresponding to the speech sample, the preset speech recognition model can be used to extract features from the audio data samples to obtain sample acoustic features of the speech sample. The method of extracting features from the audio data samples using the preset speech recognition model is similar to the method of extracting features from the target audio data using the speech recognition model, as described above, and will not be repeated here.

[0083] Optionally, in some embodiments, before using a preset speech recognition model to extract features from audio data samples to obtain sample acoustic features of speech samples, a preset speech recognition model can also be obtained. There can be multiple ways to obtain a preset speech recognition model, for example, an initial speech recognition model can be obtained, the initial speech recognition model includes a time series classification network, when the time series classification network includes a real-time time series classification network and a non-time series classification network, the initial speech recognition model is used as the preset speech recognition model, when the initial speech recognition model is a pre-trained model, and the time series classification model is a non-real-time time series classification network, the non-real-time time series classification network is copied, and the copied non-real-time time series classification network is added to the initial speech recognition model as a real-time time series classification network to obtain the preset speech recognition model.

[0084] Among them, the temporal classification network can be understood as a temporal classification based on a neural network. In the speech recognition scenario, the temporal classification network can be a CTC (connectionist temporal classification, a temporal classification network) network. The training of the speech recognition acoustic model belongs to supervised learning. It is necessary to know the label corresponding to each frame in order to conduct effective training. In the data preparation stage of training, the speech must be forced to align. The introduction of CTC can relax this one-to-one correspondence restriction requirement, and only one input sequence and one output sequence are required for training. On the one hand, the introduction of CTC does not require data alignment and one-to-one labeling. On the other hand, CTC directly outputs the probability of sequence prediction without external post-processing. The real-time temporal classification network can be understood as a temporal classification network that performs speech recognition processing in real time, and the non-real-time temporal classification network can be understood as a temporal classification network that performs speech recognition processing in non-real time. The traditional chunk-based (audio data block) speech recognition model (cascade model) often shares a time series classification network in the process of real-time speech recognition and non-real-time speech recognition, and in the process of sharing the time series classification network, the time series classification network is essentially a non-real-time time series network, or it implements the function of the non-real-time time series classification network in this solution. Therefore, the traditional chunk-based speech recognition model can be regarded as containing a non-real-time time series classification network. When the initial speech recognition model includes a real-time time series classification network and a non-real-time time series classification network, the initial speech recognition model can be directly used as the preset speech recognition model. When the initial speech recognition model is a pre-trained model and the time series classification network is a non-real-time time series classification network, the initial speech recognition model can be based on the chunk cascade model, the non-real-time time series classification network can be copied, and the copied non-real-time time series classification network can be added to the initial speech recognition model as a real-time time series classification network to obtain a preset speech recognition model.

[0085] Among them, the preset speech recognition model can include a real-time temporal classification network and a non-temporal classification network. When the initial speech recognition model is a traditional chunk-based speech recognition model, according to the needs of the optimization scenario, in order to quickly optimize the corresponding scenario, it is not necessary to train the preset speech recognition model (CPBC-ASR) from scratch. Instead, an external or newly added CTC is connected to the chunk-based cascade model trained with big data as a real-time temporal classification network, so that the chunk-based cascade model can be turned into the preset speech recognition model in this solution. Then, the external or newly added CTC is fine-tuned using general data or data in specific scenarios with big data to obtain the trained speech recognition model. The schematic diagram of the comparison between the chunk-based cascade model and the preset speech recognition model is shown in the figure. Figure 3As shown. Compared with the chunk-based cascade model, the Streaming Encoder (real-time encoding network) and the Non-streaming Encoder (non-real-time encoding network) share a CTC Decoder (time series classification network). By fine-tuning the chunk-based cascade model, a CTC-pre Decoder (real-time time series classification network) is added to the cascade model, and a peak (wave peak or spike) is generated in real time for the Non-Streaming Encoder (non-real-time encoding network) at the end of the Streaming Encoder (real-time encoding network), thereby achieving fine-tuning of the chunk-based cascade model. It should be noted that the newly added CTC-pre Decoder (real-time time series classification network) is initialized by the CTC Decoder (non-real-time time series classification network), so that the peak generated after fine-tuning is more accurate, thereby improving the training accuracy of the preset speech recognition model.

[0086] After determining the preset speech recognition model, the preset speech recognition model can be used to extract features from the audio data sample to obtain the sample acoustic features of the speech sample. The method of extracting features from the audio data sample using the preset speech recognition model is similar to the method of extracting features from the target audio data using the speech recognition model, as described above, and will not be repeated here.

[0087] (2) Based on the acoustic features of the sample, predict the text content corresponding to the speech sample to obtain the predicted text.

[0088] The predicted text may be text obtained by predicting the text content of the speech sample.

[0089] There are many ways to predict the text content corresponding to the speech sample based on the acoustic features of the sample, which can be as follows:

[0090] For example, based on the sample acoustic features, a real-time time series classification network can be used to predict the text content of a speech sample to obtain real-time predicted text, and the sample segmentation position of at least one sample audio data block can be identified in the speech sample. According to the sample segmentation position, the sample acoustic features are segmented to obtain the target sample acoustic features corresponding to each sample audio database. Based on the target sample acoustic features, a non-real-time time series classification network is used to predict the text content of the speech sample to obtain the current predicted text, and the real-time predicted text and the current predicted text are used as predicted text.

[0091] Among them, there are many ways to predict the text content of speech samples based on the sample acoustic features using a real-time time series classification network. For example, based on the sample acoustic features, a real-time time series classification network can be used to predict the posterior probability of the text content corresponding to each sample text frame, and based on the candidate probability, at least one text character corresponding to each sample text frame is determined to obtain a first predicted text character, and the first predicted text characters are spliced ​​to obtain real-time predicted text corresponding to the speech sample.

[0092] Among them, the sample audio data block can be regarded as a chunk, and the speech sample can include multiple chunks, and the number of words or text characters contained in each chunk can be the same. There can be multiple ways to identify the sample segmentation position of at least one sample audio data block in the speech sample. For example, based on the sample acoustic features, a sample text key frame corresponding to at least one sample peak can be detected in the speech sample, and the target number of groups corresponding to the sample peak can be determined in a preset number set. Based on the target number of groups, the sample peaks are grouped to obtain at least one sample peak group, and the segmentation position of the sample audio data block corresponding to each sample peak group is identified in the speech sample according to the sample key frame.

[0093] The sample peak can be a peak in the speech sample, and the peak can also be called a wave crest. In the speech recognition scenario, according to the characteristics of CTC, the position of the peak is the position where a modeling unit (text character / word) exists. Therefore, different fields (sample audio data blocks) can be formed according to the number of modeling units. z1 ],[P z1+1 ,…P z1+z2 ],…,[ Pz1+…+zn-1+1 …P L},z1,z2,…,z n The number of words corresponding to each field. The sample text key may include at least one sample audio frame where a sample peak exists or corresponds. According to the sample acoustic features, there may be multiple ways to detect at least one sample text key frame corresponding to a sample peak in a speech sample. For example, based on the posterior probability of the text content corresponding to each sample audio frame, the position of at least one sample peak may be detected in the speech sample, and at least one sample audio frame corresponding to the position of the sample peak may be screened out in the sample audio frame, thereby obtaining a sample text key frame corresponding to each sample peak.

[0094] There are many ways to determine the target number of groups corresponding to the sample peak in the preset number set. For example, a group number can be randomly selected from the preset number set as the target number of groups corresponding to the sample peak, or the number of groups corresponding to the round of iterations of the preset speech recognition model can be screened out from the preset number set as the target number of groups corresponding to the sample peak. It should be noted that in the process of iterative training of the preset speech recognition model, the target number of groups corresponding to the sample peak in different iteration rounds before convergence can be the same or different. In addition, it should also be noted that after the preset speech recognition model is trained, the number of groups in the test or application of the semantic recognition model after training can be the same as or different from the number of groups in the training process.

[0095] After determining the target number of groups corresponding to the sample peaks, the sample peaks can be grouped based on the target number of groups to obtain at least one sample peak group. There are many ways to group the sample peaks. For example, the sample peaks can be sorted based on the peak positions of the sample peaks, and according to the sorting results, the sample peaks of the target number of groups are grouped together to obtain at least one sample peak group.

[0096] After the sample peaks are grouped, the segmentation position of the sample audio data block corresponding to each sample peak group can be identified in the speech sample according to the sample text key frame. There are many ways to identify the segmentation position of the sample audio data block corresponding to each sample peak group in the speech sample. For example, based on the position of the sample text key frame, the frame number interval of the sample audio data block corresponding to each sample peak group can be identified in the speech sample to obtain the sample frame number interval, the sample frame number interval is amplified to obtain the target sample frame number interval, and the target sample frame number interval is used as the sample segmentation position of each sample audio data block.

[0097] There are many ways to identify the frame number interval of the sample audio data block corresponding to each sample peak group in the speech sample. For example, according to the position of the key frame of the sample file, the first sample peak is screened out in the sample peak group as the sample start peak, and the last sample peak is screened out in the sample peak group as the sample end peak. The frame number of the sample text frame corresponding to the sample start peak is identified in the speech sample to obtain the sample start frame number, and the frame number of the sample text frame corresponding to the sample end peak is identified in the speech sample to obtain the sample end frame number. The interval composed of the sample start frame number and the sample end frame number is used as the frame number interval of the sample audio data block corresponding to the sample peak group.

[0098] After identifying the frame number interval of the sample audio data block corresponding to each sample peak group in the speech sample, the identified sample frame number interval can be expanded, and the expanded target sample frame number interval is used as the sample segmentation position of each sample audio data block. There are many ways to expand the identified sample frame number interval. For example, the sample start frame number can be subtracted from the preset first frame number to obtain the target sample start frame number, and the sample end frame number can be added to the preset second frame number to obtain the target sample end frame number. The interval consisting of the target sample start frame number and the target sample end frame number is used as the target frame number interval. For example, taking the sample start frame number as a ps frame, the sample end frame number as a pl frame, the first preset frame number as u, and the second preset frame number as s as an example, the target frame number interval can be a frame number interval consisting of the u frame before the ps frame to the s frame after the pl frame, that is, the target frame number interval [ps-u,pl+s]. The target frame number interval can be regarded as a calculation window for a sample audio data block. The target frame number interval is used as the sample segmentation position of the corresponding sample audio data block.

[0099] After identifying the sample segmentation position of at least one sample audio data block in the speech sample, the sample acoustic feature can be segmented according to the sample segmentation position to obtain the target sample acoustic feature corresponding to each sample audio data block. There are many ways to segment the sample acoustic feature. For example, at least one sample audio frame corresponding to the target sample frame number interval can be screened out from the sample audio frame of the speech sample to obtain a candidate sample audio frame set corresponding to each sample audio data block, and the sample acoustic sub-features corresponding to each sample audio frame in the candidate sample audio frame set are extracted from the sample acoustic feature to obtain a sample acoustic sub-feature set corresponding to each sample audio data block, and the sample acoustic sub-feature set is used as the target sample acoustic feature corresponding to the sample audio data block. For example, taking the sample acoustic feature as C={C1,…,CN} and the target sample frame number interval corresponding to the sample audio data block as [ps-u,pl+s], the target sample acoustic feature corresponding to the sample audio data block can be C[ps-u,pl+s].

[0100] After segmenting the acoustic features of the sample, the text content of the speech sample can be predicted based on the segmented acoustic features of the target sample using a non-real-time time series classification network to obtain the current predicted text. There are many ways to use a non-real-time time series classification network to predict the text content of a speech sample, such as extracting text features of multiple dimensions from the acoustic features of the target sample, using a non-real-time time series classification network to predict the text content of the speech sample based on the text features, obtaining a non-real-time predicted text, splicing the text features to obtain the current text features corresponding to each sample audio data block, and determining the attention weight of the current text features, based on the attention weight, determining the text content corresponding to the current text features, obtaining auxiliary predicted text, and using the auxiliary predicted text and the non-real-time predicted text as the current predicted text.

[0101] Among them, the text feature can be feature information that characterizes text characters or fields. There are many ways to extract text features of multiple dimensions from the acoustic features of the target sample. For example, a non-real-time encoding network can be used to extract text features of multiple dimensions from the acoustic features of the target sample, or other text encoding networks can be used to extract text features of multiple dimensions from the acoustic features of the target sample, and so on.

[0102] There are many network structures of non-real-time coding networks, for example, Non-causalConformer (a target detection network), or other networks that can perform text coding, etc. Taking the non-real-time coding network as Non-causal Conformer and the acoustic feature of the target sample as Y[ps-u,pl+s] as an example, the process of extracting the acoustic feature of the target sample can be shown in formula (2), which can be as follows:

[0103] Y[ps-u, pl+s] = Non-Causal Conformer Encoder(C[ps-u, pl+s]) (2)

[0104] Among them, Y[ps-u,pl+s] is the text feature, and C[ps-u,pl+s] is the acoustic feature of the target sample.

[0105] After extracting text features of multiple dimensions from the acoustic features of the target sample, the text content of the speech sample can be predicted based on the text features using a non-real-time time series classification network to obtain a non-real-time predicted text. There are many ways to predict the text content of a speech sample based on text features using a non-real-time time series classification network. For example, a non-real-time time series classification network can be used to predict the text content corresponding to each text feature to obtain the initial text corresponding to each sample audio data block. Based on the sample segmentation position corresponding to the sample audio data block, the initial text is spliced ​​to obtain the non-real-time predicted text corresponding to the speech sample.

[0106] The attention weight may be the association weight between the current text features corresponding to different audio data blocks identified by the attention network. There are many ways to determine the attention weight. For example, the attention network may be used to extract the associated features from the current text features, and the attention weight of the current text features may be determined based on the associated features.

[0107] After determining the attention weight of the current text feature, the text content corresponding to the current text feature can be determined based on the attention weight. There are many ways to determine the text content corresponding to the current text feature. For example, the current text feature can be weighted based on the attention weight, and the weighted current text feature can be fused to obtain the target text feature, and the target text feature can be feature decoded to obtain the text content corresponding to the current text feature, and the text content corresponding to the current text feature can be used as the auxiliary predicted text. The auxiliary predicted text and the non-real-time predicted text are used as the current predicted text.

[0108] After the current predicted text is determined, the real-time predicted text and the current predicted text may be used as the predicted text corresponding to the speech sample.

[0109] (3) According to the predicted text, the recognition loss of the speech sample is determined, and based on the recognition loss, the preset speech recognition model is converged to obtain the speech recognition model.

[0110] The recognition loss may be the loss generated when a preset speech recognition model performs speech recognition on a speech sample. There are many ways to determine the recognition loss of a speech sample according to the predicted text, which may be as follows:

[0111] For example, the annotated text of the speech sample can be obtained, and the annotated text can be compared with the real-time prediction text to obtain the real-time prediction loss, the annotated text can be compared with the non-real-time prediction text to obtain the non-real-time prediction loss, and the annotated text can be compared with the auxiliary prediction text to obtain the auxiliary prediction loss. The real-time prediction loss, the non-real-time prediction loss and the auxiliary prediction loss are fused to obtain the recognition loss of the speech sample.

[0112] Among them, the annotated text can be the text content corresponding to the pre-annotated speech sample. There are many ways to compare the annotated text with the real-time predicted text, for example, the CTC loss function (the loss function corresponding to the CTC network) can be used to compare the annotated text with the real-time predicted text to obtain the real-time prediction loss, or the cross entropy loss function can be used to compare the annotated text with the real-time predicted text to obtain the real-time prediction loss, or other types of loss functions can be used to compare the annotated text with the real-time predicted text to obtain the real-time prediction loss, and so on.

[0113] Among them, the method of determining non-real-time prediction loss and auxiliary prediction loss is similar to the method of determining real-time prediction loss, as described above, and will not be repeated here.

[0114] After determining the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss, the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss can be fused to obtain the recognition loss of the speech sample. There are many ways to fuse the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss. For example, the loss fusion weight can be obtained, and the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss can be weighted respectively. The mean of the weighted real-time prediction loss and the weighted non-real-time prediction loss is calculated to obtain the target prediction loss. The target prediction loss is fused with the weighted auxiliary prediction loss to obtain the recognition loss of the speech sample, which can be shown in formula (3). Specifically, it can be as follows:

[0115] loss=ctc_weight*0.5*(causa_loss+non-causa_loss)+(1-ctc_weight)*att_loss (3)

[0116] Among them, loss is the recognition loss of the speech sample, ctc_weight is the loss fusion weight, causa_loss is the real-time prediction loss, non-causa_loss is the non-real-time prediction loss, and att_loss is the auxiliary prediction loss.

[0117] After the real-time prediction loss, the non-real-time prediction loss and the auxiliary prediction loss are fused, the preset speech recognition model can be converged based on the fused recognition loss to obtain the speech recognition model. There are many ways to converge the preset speech recognition model. For example, a gradient descent algorithm can be used to update the network parameters of the preset speech recognition model based on the recognition loss to obtain an updated speech recognition model. The updated speech recognition model is used as the preset speech recognition model, and the step of obtaining multiple audio data samples corresponding to the speech sample is returned until the preset speech recognition model converges to obtain the trained speech recognition model. Alternatively, other network parameter update algorithms can be used to update the network parameters of the preset speech recognition model based on the recognition loss to obtain an updated speech recognition model. The updated speech recognition model is used as the preset speech recognition model, and the step of obtaining multiple audio data samples corresponding to the speech sample is returned until the preset speech recognition model converges to obtain the trained speech recognition model, and so on.

[0118] S2. Generate the initial text corresponding to the current audio stream data based on the acoustic features.

[0119] For example, based on acoustic features, a real-time time series classification network can be used to predict the posterior probability of the text content corresponding to each audio frame. According to the posterior probability, the initial text content corresponding to each audio frame is determined, and the initial text content is fused to obtain the initial text corresponding to the current audio stream data.

[0120] 103. Detect at least one text key frame in the current audio stream data, and identify a segmentation position of at least one audio data block in the current audio stream data based on the text key frame.

[0121] The audio data block is equivalent to a chunk of current audio stream data in a speech recognition scenario, and the text content corresponding to the audio data block may include a preset number of text characters.

[0122] The current audio stream data may include multiple audio frames. The text key frame may be at least one audio frame indicating that text characters can be recognized in the current audio stream data. There are multiple ways to detect at least one text key frame in the current audio stream data, which may be as follows:

[0123] For example, based on acoustic features, the posterior probability of the text content corresponding to each audio frame can be predicted. According to the posterior probability, the peak position of at least one peak is detected in the current audio stream data, and at least one audio frame corresponding to the peak position is identified in the audio frame to obtain the text key frame corresponding to each peak.

[0124] The method of detecting the peak position of at least one peak in the current audio stream data may be similar to the method of detecting the peak position of the sample peak, as described above, and will not be described in detail here.

[0125] After detecting the peak position of at least one peak in the current audio stream data, at least one audio frame corresponding to the peak position can be identified in the audio frame, thereby obtaining a text key frame corresponding to each peak. The method of identifying the text key frame can be similar to the method of identifying the sample text key frame in the speech sample, as described above, and will not be repeated here.

[0126] After detecting at least one frame of text keyframe in the current audio stream data, the segmentation position of at least one audio data block can be identified in the current audio stream data based on the text keyframe. There can be multiple ways to identify the segmentation position of at least one audio data block in the current audio stream data. For example, peaks can be grouped according to peak positions to obtain multiple peak groups, each peak group including a preset number of peaks. Based on the text keyframe, the frame number interval of the audio data block corresponding to each peak group is identified in the current audio stream data, the frame number interval is amplified to obtain a target frame number interval, and the target frame number interval is used as the segmentation position of the audio data block.

[0127] Among them, the number of peaks in the peak group can be the same as or different from the number of peaks in the sample peak group. It should be noted that in the process of training the preset speech recognition model, in order to enable the preset speech recognition model to have the ability to determine the complete semantics of randomly cut complete sentences, the number of peaks in each round of iteration is randomly or in a specific order selected from the preset number set, that is, the number of modeling units (text characters) contained in each chunk can be set to a random value or a specific order of integers from a preset number set or a numerical range. After the training of the preset speech recognition model is completed, when the trained speech recognition model is used for testing or application, in the process of grouping the peaks, the number of peaks in each peak group is a fixed number, and the number of peaks can be any value, for example, it can be 9, 10 or any other integer greater than 0, and so on. The method of grouping the peaks according to the peak position is similar to the method of grouping the sample peaks, as described above, and will not be repeated here.

[0128] After the peaks are grouped, the frame number interval of the audio data block corresponding to each peak group can be identified in the current audio stream data based on the text key frame. There can be many ways to identify the frame number interval of the audio data block corresponding to each peak group in the current audio stream data. For example, the starting peak and the ending peak can be screened out in the peak group according to the peak position, the frame number of the text frame corresponding to the starting peak can be identified in the current audio stream data to obtain the starting frame number, the frame number of the text key frame corresponding to the ending peak can be identified in the current audio stream data to obtain the ending frame number, and the interval composed of the starting frame number and the ending frame number can be used as the frame number interval of the audio data block corresponding to the peak group.

[0129] The method of selecting the start spike and the end spike in the spike group is similar to the method of selecting the sample start spike and the sample end spike in the sample spike group, as described above, and will not be described in detail here.

[0130] After filtering out the starting peak and the ending peak, the number of text key frames corresponding to the starting peak and the number of text key frames corresponding to the ending peak can be identified in the current audio stream data, thereby obtaining the starting frame number and the ending frame number. Then, the interval formed by the starting frame number and the ending frame number is used as the frame number interval of the audio data block corresponding to the peak group.

[0131] After identifying the frame number interval of the audio data block corresponding to each peak group in the current audio stream data, the frame number interval can be expanded to obtain a target frame number interval. There are many ways to expand the frame number interval, for example, the start frame number can be subtracted from the preset first frame number to obtain the target start frame number, the end frame number can be added to the preset second frame number to obtain the target end frame number, and the interval consisting of the target start frame number and the target end frame number is used as the target frame number interval.

[0132] The preset first frame number and the preset second frame number may be the same or different. By expanding the frame number interval, the size of the calculation window is increased, so that the accuracy of the chunk boundary can be improved, thereby improving the accuracy of speech recognition. The preset first frame number and the preset second frame number can be any integer greater than or equal to 0.

[0133] After the frame number interval is expanded, the expanded target frame number interval can be used as the segmentation position of the audio data chunk corresponding to the peak group.

[0134] 104. Segment the acoustic features according to the segmentation positions to obtain target acoustic features of each audio data block.

[0135] The acoustic feature may include acoustic sub-features corresponding to each audio frame. There are multiple ways to segment the acoustic feature according to the segmentation position, which may be as follows:

[0136] For example, at least one audio frame corresponding to a target frame number interval can be screened out from the audio frames to obtain a set of candidate audio frames corresponding to each audio data block, and the acoustic sub-features corresponding to each audio frame in the candidate audio frame set can be extracted from the acoustic features to obtain a set of acoustic sub-features corresponding to each audio data block, and the set of acoustic sub-features can be used as the target acoustic features corresponding to the audio data block.

[0137] 105. Based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

[0138] For example, based on the target acoustic features, the target field corresponding to each audio data block can be determined, and the target fields can be spliced ​​according to the segmentation position to obtain the spliced ​​text, and the initial text can be replaced with the spliced ​​text to obtain the target text corresponding to the current audio stream data.

[0139] Among them, there are many ways to determine the target field corresponding to each audio data block based on the target acoustic features. For example, the non-real-time coding network (Non-Streaming Encoder) in the speech recognition model can be used to extract multiple dimensions of text features from the target acoustic features to obtain the target text features of each audio data block, and the non-real-time time series classification network can be used to determine the text content corresponding to the target text features to obtain the target field corresponding to each audio data block.

[0140] After determining the target field corresponding to each audio data block, the target field can be spliced ​​based on the segmentation position to obtain the spliced ​​text. There are many ways to splice the target field based on the segmentation position. For example, the target field can be sorted based on the segmentation position, and the target field can be spliced ​​based on the sorting result to obtain the spliced ​​text, or the frame number interval of the audio frame corresponding to each target field can be determined based on the segmentation position, and the target field can be spliced ​​or combined based on the frame number interval to obtain the spliced ​​text, and so on.

[0141] After the target fields are spliced, the initial text can be replaced with the spliced ​​text, thereby obtaining the target text corresponding to the current audio stream data. It can be understood that the target text corresponding to the current audio stream data can be the spliced ​​text.

[0142] After the initial text is replaced with the concatenated text, the concatenated text can be used as the target text corresponding to the current audio stream data. Then, the target text is output. There are many ways to output the target text, for example, the target text can be directly displayed, or the target text can be sent to the terminal corresponding to the current audio stream data so that the terminal displays the target text, etc.

[0143] Among them, the speech recognition of this solution can be applied to various input methods or speech recognition related products, and the speech recognition is performed on the audio stream data input by the user through voice, so as to obtain the input target text. Compared with the traditional cascade model for speech recognition, which has a low accuracy rate for chunk boundary recognition, the speech recognition model of this solution can be regarded as a cascade model (peak base cascade model) that introduces peak calculation. By calculating the peak position, a more accurate or longer fixed chunk window can be obtained for segmentation, thereby improving the accuracy of chunk boundary recognition. The model structure of the speech recognition model of this solution in the speech recognition process can be as follows: Figure 4 As shown, after obtaining the current audio stream data, the acoustic feature encoding is performed according to the traditional fixed chunk calculation window through the real-time encoding network (Streaming Encoder) to obtain the acoustic features of the current audio stream data. Then, based on the acoustic features, at least one peak in the current audio stream data and the initial text corresponding to the current audio stream data are calculated through the real-time temporal classification network (CTC-pre Decoder). Based on the peak position, the calculation window (segmentation position) of the chunk is calculated, and the acoustic features are segmented through the calculation window to obtain the target acoustic features of each chunk. The non-real-time encoding network (Non-Streaming Encoder) is used to extract the text features from the target acoustic features. Then, the non-real-time temporal classification network (CTC Decoder) is used to determine the target field of each chunk corresponding to the text features, and the target fields are spliced ​​to obtain the spliced ​​text, and the initial text is replaced with the spliced ​​text, so as to obtain the target text corresponding to the current audio stream data.

[0144] Among them, it should be noted that the speech recognition model in this scheme can be regarded as a real-time part of a causal conformer encoder (a coding network of a speech recognition model) and a non-real-time part of a non-causal conformer encoder (a coding network of a speech recognition model). The causal Conformer of the real-time part calculates the window according to the traditional fixed chunk, and the posterior probability of the ctc calculated by the calculation output can be used to calculate the CTC Peak (peak position). The non-causal Conformer of the non-real-time part uses the window calculated by the CTC Peak calculated by the real-time part as input to calculate the final text output. Compared with the non-real-time part of the traditional cascade model, the Non-causal Conformer encoder model uses a longer fixed chunk window as input. The window calculation based on the CTC Peak solves the problem of inaccurate sentence boundary recognition, thereby improving the recognition accuracy. In addition, a simulation comparison experiment was conducted on the sentence error rate (SER) of the traditional cascade model and the speech recognition model of this scheme. The sentence error rate of the speech recognition model of this scheme can be absolutely reduced by 2%. The sentence error rate is proportional to user dissatisfaction. A decrease in the sentence error rate means an increase in user satisfaction. Therefore, the speech recognition model in this solution can improve the recognition accuracy and thus improve user satisfaction by optimizing the speech recognition boundary.

[0145] The sentence error rate can be understood as the probability of judging that a sentence is incorrectly recognized if only one word in the sentence is incorrectly recognized.

[0146] As can be seen from the above, after the embodiment of the present application obtains the current audio stream data and divides the current audio stream data to obtain multiple segments of target audio data, feature extraction is performed on the target audio data to obtain the acoustic features of the current audio stream data, and based on the acoustic features, the initial text corresponding to the current audio stream data is generated, at least one frame of text key frame is detected in the current audio stream data, and based on the text key frame, the segmentation position of at least one audio data block is identified in the current audio stream data, and then, according to the segmentation position, the acoustic features are segmented to obtain the target acoustic features of each audio data block, and based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data; because the scheme can not only identify the initial text, but also detect at least one frame of text key frame in the current audio stream data during the real-time recognition of the corresponding streaming speech, and then, the segmentation position of the audio data block (chunk) is identified in the current audio stream data through the text key frame, thereby improving the accuracy of chunk boundary recognition, and then the recognition rate of the sentence corresponding to the audio data block can be improved, and therefore, the accuracy of speech recognition can be improved.

[0147] According to the method described in the above embodiment, the following is further described in detail with examples.

[0148] In this embodiment, the speech recognition device is specifically integrated in an electronic device, the electronic device is a server, the time series classification network is a CTC network, the audio data block is a chunk, and the segmentation position is a calculation window.

[0149] (A) Training of speech recognition model.

[0150] (1) The server obtains multiple audio data samples corresponding to the speech sample, and uses a preset speech recognition model to extract features from the audio data samples to obtain sample acoustic features of the speech sample.

[0151] For example, the server can segment the speech sample according to a preset audio mask and a preset number of segmentation frames to obtain multiple sample frame sets, filter out at least one historical sample audio frame corresponding to each sample audio frame set in the sample audio frame, add the historical sample audio frame to the corresponding sample audio frame set, obtain multiple target sample audio frame sets, and use the target sample audio frame sets as audio data samples.

[0152] An initial speech recognition model is obtained, which includes a CTC network. When the CTC network includes a real-time CTC network and a non-CTC network, the initial speech recognition model is used as a preset speech recognition model. When the initial speech recognition model is a pre-trained model and the time series classification model is a non-real CTC network, the non-real-time CTC network is copied and the copied non-real-time CTC network is added to the initial speech recognition model as a real-time CTC network to obtain a preset speech recognition model.

[0153] The server can obtain an initial speech recognition model, which includes a CTC network. When the CTC network includes a real-time CTC network and a non-CTC network, the initial speech recognition model is used as a preset speech recognition model. When the initial speech recognition model is a pre-trained model and the time series classification model is a non-real CTC network, the non-real-time CTC network is copied and the copied non-real-time CTC network is added to the initial speech recognition model as a real-time CTC network to obtain a preset speech recognition model.

[0154] The server can use a real-time coding network of a preset speech recognition model to extract features from the audio data samples to obtain sample acoustic features of the speech samples.

[0155] (2) The server predicts the text content corresponding to the speech sample based on the acoustic features of the sample to obtain the predicted text.

[0156] For example, the server can use a real-time CTC network to predict the posterior probability of the text content corresponding to each sample text frame based on the sample acoustic features, determine at least one text character corresponding to each sample text frame based on the candidate probability, obtain the first predicted text character, and splice the first predicted text characters to obtain the real-time predicted text corresponding to the speech sample.

[0157] The server can detect the position of at least one sample peak in the speech sample based on the posterior probability of the text content corresponding to each sample audio frame, and filter out at least one sample audio frame corresponding to the position of the sample peak in the sample audio frame, so as to obtain the sample text key frame corresponding to each sample peak.

[0158] The server randomly selects a group number from a preset number set as the target group number corresponding to the sample peak, or, based on the iterative rounds of a preset speech recognition model, selects the group number corresponding to the round from the preset number set as the target group number corresponding to the sample peak.

[0159] The server can sort the sample peaks based on the peak positions of the sample peaks, and group the sample peaks of the target grouping number according to the sorting results, thereby obtaining at least one sample peak group. According to the position of the key frame of the sample file, the first sample peak is screened out in the sample peak group as the sample start peak, and the last sample peak is screened out in the sample peak group as the sample end peak, and the frame number of the sample text frame corresponding to the sample start peak is identified in the voice sample to obtain the sample start frame number, and the frame number of the sample text frame corresponding to the sample end peak is identified in the voice sample to obtain the sample end frame number, and the interval composed of the sample start frame number and the sample end frame number is used as the frame number interval of the sample chunk corresponding to the sample peak group. Then, the sample start frame number is subtracted from the preset first frame number to obtain the target sample start frame number, and the sample end frame number is added to the preset second frame number to obtain the target sample end frame number, and the interval composed of the target sample start frame number and the target sample end frame number is used as the target frame number interval. The target frame number interval is used as the sample calculation window of the corresponding sample chunk.

[0160] The server can filter out at least one sample audio frame corresponding to the target sample frame number interval from the sample audio frames of the speech sample, obtain a set of candidate sample audio frames corresponding to each sample chunk, extract sample acoustic sub-features corresponding to each sample audio frame in the set of candidate sample audio frames from the sample acoustic features, obtain a set of sample acoustic sub-features corresponding to each sample chunk, and use the set of sample acoustic sub-features as the target sample acoustic features corresponding to the sample chunk.

[0161] The server may use a non-real-time encoding network to extract text features of multiple dimensions from the acoustic features of the target sample, or may use other text encoding networks to extract text features of multiple dimensions from the acoustic features of the target sample, and so on.

[0162] The server can use a non-real-time CTC network to predict the text content corresponding to each text feature to obtain the initial text corresponding to each sample chunk, and concatenate the initial text based on the sample calculation window corresponding to the sample chunk to obtain the non-real-time predicted text corresponding to the speech sample.

[0163] The server can concatenate the text features to obtain the current text features corresponding to each sample chunk, use the attention network (Attention) to extract the associated features from the current text features, and determine the attention weight of the current text features based on the associated features. Based on the attention weights, the current text features are weighted, and the weighted current text features are fused to obtain the target text features, and the target text features are feature decoded to obtain the text content corresponding to the current text features, and the text content corresponding to the current text features is used as the auxiliary predicted text. The auxiliary predicted text and the non-real-time predicted text are used as the current predicted text. Then, the real-time predicted text and the current predicted text are used as the predicted text corresponding to the speech sample.

[0164] (3) The server determines the recognition loss of the speech sample according to the predicted text, and based on the recognition loss, converges the preset speech recognition model to obtain the speech recognition model.

[0165] For example, the server can obtain the annotated text of the speech sample, and can use the cross entropy loss function to compare the annotated text with the real-time prediction text to obtain the real-time prediction loss, or can also use other types of loss functions to compare the annotated text with the real-time prediction text to obtain the real-time prediction loss, and so on.

[0166] The server may also compare the annotated text with the non-real-time prediction text to obtain the non-real-time prediction loss, and compare the annotated text with the auxiliary prediction text to obtain the auxiliary prediction loss.

[0167] The server can obtain the loss fusion weight, weight the real-time prediction loss, non-real-time prediction loss and auxiliary prediction loss respectively, calculate the average of the weighted real-time prediction loss and the weighted non-real-time prediction loss, obtain the target prediction loss, and fuse the target prediction loss with the weighted auxiliary prediction loss to obtain the recognition loss of the speech sample, which can be shown as formula (3):

[0168] The server can use a gradient descent algorithm to update the network parameters of the preset speech recognition model based on the recognition loss to obtain an updated speech recognition model, use the updated speech recognition model as the preset speech recognition model, and return to execute the step of obtaining multiple audio data samples corresponding to the speech sample until the preset speech recognition model converges to obtain the trained speech recognition model. Alternatively, other network parameter updating algorithms can be used to update the network parameters of the preset speech recognition model based on the recognition loss to obtain an updated speech recognition model, use the updated speech recognition model as the preset speech recognition model, and return to execute the step of obtaining multiple audio data samples corresponding to the speech sample until the preset speech recognition model converges to obtain the trained speech recognition model, and so on.

[0169] (ii) The server uses the trained speech recognition model to recognize the current audio stream data, thereby obtaining the target text corresponding to the current audio stream data.

[0170] Among them, the speech recognition model may include a real-time encoding network, a non-real-time encoding network, a real-time CTC network and a non-real-time CTC network.

[0171] like Figure 5 As shown, a speech recognition method, the specific process is as follows:

[0172] 201. The server obtains current audio stream data.

[0173] For example, the server can receive audio stream data uploaded by the terminal or client to obtain the current audio stream data, or it can collect the sound or audio emitted by the target object in real time through a sound collection device to obtain the current audio stream data, or it can obtain multiple consecutive audio frames from the Internet or an audio stream database to obtain the current audio stream data, and so on.

[0174] 202. The server divides the current audio stream data into multiple segments of target audio data.

[0175] For example, when the server receives audio frames in the current audio stream data, it can count the number of received audio frames. When receiving audio frames with a preset number of segmented frames, it can take the audio frames with the preset number of segmented frames as an audio frame set, and then return to execute the frame counting of the received audio frames until all the audio frames in the current audio stream data are received, thereby obtaining multiple audio frame sets. Alternatively, when receiving audio frames corresponding to the preset number of segmented frames in the current audio stream data, these audio frames can be taken as an audio frame set, and then, and so on, until all the audio frames in the current audio stream data are received, thereby obtaining multiple audio frame sets.

[0176] The server selects the first audio frame from the audio frame set as the target audio frame, and selects a preset number of audio frames before the target audio frame from the audio stream data, thereby obtaining historical audio frames corresponding to each audio frame set. The historical audio frames are added to the corresponding audio frame set to obtain multiple target audio frame sets, and the target audio frame sets are used as target audio data.

[0177] 203. The server performs feature extraction on the target audio data to obtain acoustic features of the current audio stream data.

[0178] For example, the server may use a real-time coding network to perform feature encoding on the audio frames in the target audio data, obtain the audio features corresponding to each audio frame in the target audio data, and use the audio features as the initial acoustic features corresponding to the target audio data. Based on the timing information of the target audio data in the current audio stream data, the initial acoustic features are sorted, and based on the sorting result, the initial acoustic features are spliced ​​to obtain the acoustic features of the target audio stream data.

[0179] 204. Generate an initial text corresponding to the current audio stream data based on the acoustic features.

[0180] For example, the server can use a real-time CTC network to predict the posterior probability of the text content corresponding to each audio frame based on acoustic features, determine the initial text content corresponding to each audio frame based on the posterior probability, and fuse the initial text content to obtain the initial text corresponding to the current audio stream data.

[0181] 205. The server detects at least one text key frame in the current audio stream data.

[0182] For example, the server can predict the posterior probability of the text content corresponding to each audio frame based on the acoustic features, detect the peak position of at least one peak in the current audio stream data according to the posterior probability, identify at least one audio frame corresponding to the peak position in the audio frame, and obtain the text key frame corresponding to each peak.

[0183] 206. The server identifies a calculation window of at least one chunk in the current audio stream data based on the text key frame.

[0184] For example, the server can group the peaks according to the peak positions to obtain multiple peak groups, which include a preset number of peaks. According to the peak positions, the starting peak and the ending peak are screened out in the peak group, the frame number of the text frame corresponding to the starting peak is identified in the current audio stream data to obtain the starting frame number, the frame number of the text key frame corresponding to the ending peak is identified in the current audio stream data to obtain the ending frame number, and the interval composed of the starting frame number and the ending frame number is used as the frame number interval of the chunk corresponding to the peak group. The starting frame number is subtracted from the preset first frame number to obtain the target starting frame number, the ending frame number is added to the preset second frame number to obtain the target ending frame number, and the interval composed of the target starting frame number and the target ending frame number is used as the target frame number interval. The target frame number interval is used as the calculation window of the chunk (chunk) corresponding to the peak group.

[0185] 207. The server divides the acoustic features according to the calculation window to obtain the target acoustic features of each chunk.

[0186] For example, the server can filter out at least one audio frame corresponding to the target frame number interval in the audio frame, obtain a set of candidate audio frames corresponding to each chunk, extract the acoustic sub-features corresponding to each audio frame in the candidate audio frame set from the acoustic features, obtain a set of acoustic sub-features corresponding to each chunk, and use the set of acoustic sub-features as the target acoustic features corresponding to the chunk.

[0187] 208. The server updates the initial text based on the target acoustic features to obtain the target text corresponding to the current audio stream data.

[0188] For example, the server can use the non-real-time encoding network (Non-StreamingEncoder) in the speech recognition model to extract text features of multiple dimensions from the target acoustic features to obtain the target text features of each chunk, and use the non-real-time CTC network to determine the text content corresponding to the target text features to obtain the target field corresponding to each chunk.

[0189] The server can sort the target fields based on the calculation window, and splice the target fields based on the sorting results to obtain the spliced ​​text. Alternatively, the server can determine the frame number interval of the audio frames corresponding to each target field based on the calculation window, and splice or combine the target fields based on the frame number interval to obtain the spliced ​​text, and so on.

[0190] The server replaces the initial text with the concatenated text, thereby obtaining the target text corresponding to the current audio stream data.

[0191] Optionally, in some embodiments, the server may also directly display the target text, or may also send the target text to a terminal corresponding to the current audio stream data so that the terminal displays the target text, and so on.

[0192] As can be seen from the above, after the server of this embodiment obtains the current audio stream data and divides the current audio stream data to obtain multiple segments of target audio data, it extracts features from the target audio data to obtain the acoustic features of the current audio stream data, and generates the initial text corresponding to the current audio stream data based on the acoustic features, detects at least one frame of text key frame in the current audio stream data, and identifies at least one chunk calculation window in the current audio stream data based on the text key frame, and then, according to the calculation window, divides the acoustic features to obtain the target acoustic features of each chunk, and updates the initial text based on the target acoustic features to obtain the target text corresponding to the current audio stream data; because the scheme can not only identify the initial text, but also detect at least one frame of text key frame in the current audio stream data during the real-time recognition of the corresponding streaming speech, and then, through the text key frame, the chunk (chunk) calculation window is identified in the current audio stream data, thereby improving the accuracy of chunk boundary recognition, and then improving the recognition rate of the sentence corresponding to the chunk, and therefore, can improve the accuracy of speech recognition.

[0193] In order to better implement the above method, an embodiment of the present invention further provides a speech recognition device, which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer.

[0194] For example, Figure 6 As shown, the speech recognition device may include an acquisition unit 301, a generation unit 302, a recognition unit 303, a segmentation unit 304 and an update unit 305, as follows:

[0195] (1) Acquisition unit 301;

[0196] The acquisition unit 301 is used to obtain the current audio stream data and segment the current audio stream data to obtain multiple segments of target audio data.

[0197] For example, the acquisition unit 301 can be specifically used to obtain current audio stream data, split the current audio stream data based on a preset number of split frames to obtain multiple audio frame sets, filter out at least one historical audio frame corresponding to each audio frame set in the audio stream data, add the historical audio frame to the corresponding audio frame set to obtain multiple target audio frame sets, and use the target audio frame sets as target audio data.

[0198] (2) generating unit 302;

[0199] The generating unit 302 is used to extract features from the target audio data to obtain acoustic features of the current audio stream data, and generate an initial text corresponding to the current audio stream data based on the acoustic features.

[0200] For example, the generation unit 302 can be specifically used to use a speech recognition model to extract features from the target audio data, obtain the initial acoustic features corresponding to each target audio data, splice the initial acoustic features to obtain the acoustic features corresponding to the current audio stream data, and based on the acoustic features, use a real-time time series classification network to predict the posterior probability of the text content corresponding to each audio frame, determine the initial text content corresponding to each audio frame based on the posterior probability, and fuse the initial text content to obtain the initial text corresponding to the current audio stream data.

[0201] (3) identification unit 303;

[0202] The identification unit 303 is used to detect at least one text key frame in the current audio stream data, and identify the segmentation position of at least one audio data block in the current audio stream data based on the text key frame.

[0203] For example, the recognition unit 303 can be specifically used to predict the posterior probability of the text content corresponding to each audio frame based on the acoustic features, detect the peak position of at least one peak in the current audio stream data according to the posterior probability, identify at least one audio frame corresponding to the peak position in the audio frame, obtain the text key frame corresponding to each peak, group the peaks according to the peak positions to obtain multiple peak groups, the peak group includes a preset number of peaks, and based on the text key frame, identify the frame number interval of the audio data block corresponding to each peak group in the current audio stream data, expand the frame number interval to obtain the target frame number interval, and use the target frame number interval as the segmentation position of the audio data block.

[0204] (4) segmentation unit 304;

[0205] The segmentation unit 304 is used to segment the acoustic features according to the segmentation positions to obtain target acoustic features of each audio data block.

[0206] For example, the segmentation unit 304 can be specifically used to filter out at least one audio frame corresponding to a target frame number interval in the audio frame, obtain a set of candidate audio frames corresponding to each audio data block, extract the acoustic sub-features corresponding to each audio frame in the candidate audio frame set from the acoustic features, obtain the acoustic sub-feature set corresponding to each audio data block, and use the acoustic sub-feature set as the target acoustic feature corresponding to the audio data block.

[0207] (5) Update unit 305;

[0208] The updating unit 305 is used to update the initial text based on the target acoustic features to obtain the target text corresponding to the current audio stream data.

[0209] For example, the update unit 305 can be specifically used to determine the target field corresponding to each audio data block based on the target acoustic features, splice the target field according to the segmentation position to obtain the spliced ​​text, replace the initial text with the spliced ​​text, and obtain the target text corresponding to the current audio stream data.

[0210] Optionally, in some embodiments, the speech recognition device may further include a training unit 306, such as Figure 7 As shown, the specific details can be as follows:

[0211] The training unit 306 is used to train the preset speech recognition model to obtain a speech recognition model.

[0212] For example, the training unit 306 can be specifically used to obtain multiple audio data samples corresponding to the speech sample, and use a preset speech recognition model to extract features from the audio data samples to obtain sample acoustic features of the speech sample, and based on the sample acoustic features, predict the text content corresponding to the speech sample to obtain the predicted text, determine the recognition loss of the speech sample based on the predicted text, and based on the recognition loss, converge the preset speech recognition model to obtain the speech recognition model.

[0213] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0214] As can be seen from the above, in this embodiment, after the acquisition unit 301 acquires the current audio stream data and segments the current audio stream data to obtain multiple segments of target audio data, the generation unit 302 extracts features from the target audio data to obtain the acoustic features of the current audio stream data, and based on the acoustic features, generates the initial text corresponding to the current audio stream data, the recognition unit 303 detects at least one frame of text keyframes in the current audio stream data, and based on the text keyframes, identifies the segmentation position of at least one audio data block in the current audio stream data, and then the segmentation unit 304 segments the acoustic features according to the segmentation position to obtain each segment of the target audio data. The target acoustic features of an audio data block, the updating unit 305 updates the initial text based on the target acoustic features to obtain the target text corresponding to the current audio stream data; since the scheme can not only identify the initial text but also detect at least one frame of text key frame in the current audio stream data during the real-time recognition of the corresponding streaming speech, and then identify the segmentation position of the audio data block (chunk) in the current audio stream data through the text key frame, thereby improving the accuracy of chunk boundary recognition, and then can improve the recognition rate of the sentence corresponding to the audio data block, therefore, can improve the accuracy of speech recognition.

[0215] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present invention, specifically:

[0216] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0217] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0218] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0219] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0220] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0221] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0222] The current audio stream data is obtained, and the current audio stream data is segmented to obtain multiple segments of target audio data, and features are extracted on the target audio data to obtain acoustic features of the current audio stream data, and based on the acoustic features, an initial text corresponding to the current audio stream data is generated, at least one frame of text keyframes is detected in the current audio stream data, and based on the text keyframes, a segmentation position of at least one audio data block is identified in the current audio stream data, and according to the segmentation position, the acoustic features are segmented to obtain target acoustic features of each audio data block, and based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

[0223] For example, the electronic device can obtain the current audio stream data, segment the current audio stream data based on a preset segmentation frame number to obtain multiple audio frame sets, select at least one historical audio frame corresponding to each audio frame set in the audio stream data, add the historical audio frame to the corresponding audio frame set to obtain multiple target audio frame sets, and use the target audio frame sets as target audio data. Use a speech recognition model to extract features from the target audio data to obtain initial acoustic features corresponding to each target audio data, concatenate the initial acoustic features to obtain acoustic features corresponding to the current audio stream data, and use a real-time time series classification network to predict the posterior probability of the text content corresponding to each audio frame based on the acoustic features. According to the posterior probability, determine the initial text content corresponding to each audio frame, and fuse the initial text content to obtain the initial text corresponding to the current audio stream data. Based on the acoustic features, the posterior probability of the text content corresponding to each audio frame is predicted. According to the posterior probability, the peak position of at least one peak is detected in the current audio stream data, and at least one audio frame corresponding to the peak position is identified in the audio frame to obtain the text key frame corresponding to each peak. The peaks are grouped according to the peak positions to obtain multiple peak groups, and the peak groups include a preset number of peaks. Based on the text key frame, the frame number interval of the audio data block corresponding to each peak group is identified in the current audio stream data, and the frame number interval is amplified to obtain the target frame number interval, and the target frame number interval is used as the segmentation position of the audio data block. At least one audio frame corresponding to the target frame number interval is selected from the audio frame to obtain a set of candidate audio frames corresponding to each audio data block, and the acoustic sub-features corresponding to each audio frame in the candidate audio frame set are extracted from the acoustic features to obtain a set of acoustic sub-features corresponding to each audio data block, and the set of acoustic sub-features is used as the target acoustic features corresponding to the audio data block. Based on the target acoustic features, the target field corresponding to each audio data block is determined, and the target field is spliced ​​according to the segmentation position to obtain the spliced ​​text. The initial text is replaced with the spliced ​​text to obtain the target text corresponding to the current audio stream data.

[0224] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0225] As can be seen from the above, after the embodiment of the present invention obtains the current audio stream data and divides the current audio stream data to obtain multiple segments of target audio data, the target audio data is feature extracted to obtain the acoustic features of the current audio stream data, and based on the acoustic features, the initial text corresponding to the current audio stream data is generated, at least one frame of text key frame is detected in the current audio stream data, and based on the text key frame, at least one audio data block segmentation position is identified in the current audio stream data, and then, according to the segmentation position, the acoustic features are segmented to obtain the target acoustic features of each audio data block, and based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data; because the scheme can not only identify the initial text, but also detect at least one frame of text key frame in the current audio stream data during the real-time recognition of the corresponding streaming speech, and then, the segmentation position of the audio data chunk is identified in the current audio stream data through the text key frame, thereby improving the accuracy of chunk boundary recognition, and then improving the recognition rate of the sentence corresponding to the audio data chunk, and therefore, can improve the accuracy of speech recognition.

[0226] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0227] To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the speech recognition methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0228] The current audio stream data is obtained, and the current audio stream data is segmented to obtain multiple segments of target audio data, and features are extracted on the target audio data to obtain acoustic features of the current audio stream data, and based on the acoustic features, an initial text corresponding to the current audio stream data is generated, at least one frame of text keyframes is detected in the current audio stream data, and based on the text keyframes, a segmentation position of at least one audio data block is identified in the current audio stream data, and according to the segmentation position, the acoustic features are segmented to obtain target acoustic features of each audio data block, and based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

[0229] For example, the current audio stream data is obtained, and the current audio stream data is segmented based on a preset segmentation frame number to obtain multiple audio frame sets, and at least one historical audio frame corresponding to each audio frame set is screened out from the audio stream data, and the historical audio frame is added to the corresponding audio frame set to obtain multiple target audio frame sets, and the target audio frame sets are used as target audio data. The target audio data is feature extracted using a speech recognition model to obtain the initial acoustic features corresponding to each target audio data, and the initial acoustic features are concatenated to obtain the acoustic features corresponding to the current audio stream data. Based on the acoustic features, a real-time time series classification network is used to predict the posterior probability of the text content corresponding to each audio frame, and the initial text content corresponding to each audio frame is determined based on the posterior probability, and the initial text content is fused to obtain the initial text corresponding to the current audio stream data. Based on the acoustic features, the posterior probability of the text content corresponding to each audio frame is predicted. According to the posterior probability, the peak position of at least one peak is detected in the current audio stream data, and at least one audio frame corresponding to the peak position is identified in the audio frame to obtain the text key frame corresponding to each peak. The peaks are grouped according to the peak positions to obtain multiple peak groups, and the peak groups include a preset number of peaks. Based on the text key frame, the frame number interval of the audio data block corresponding to each peak group is identified in the current audio stream data, and the frame number interval is amplified to obtain the target frame number interval, and the target frame number interval is used as the segmentation position of the audio data block. At least one audio frame corresponding to the target frame number interval is selected from the audio frame to obtain a set of candidate audio frames corresponding to each audio data block, and the acoustic sub-features corresponding to each audio frame in the candidate audio frame set are extracted from the acoustic features to obtain a set of acoustic sub-features corresponding to each audio data block, and the set of acoustic sub-features is used as the target acoustic features corresponding to the audio data block. Based on the target acoustic features, the target field corresponding to each audio data block is determined, and the target field is spliced ​​according to the segmentation position to obtain the spliced ​​text. The initial text is replaced with the spliced ​​text to obtain the target text corresponding to the current audio stream data.

[0230] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0231] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0232] Since the instructions stored in the computer-readable storage medium can execute the steps in any one of the speech recognition methods provided in the embodiments of the present invention, the beneficial effects that can be achieved by any one of the speech recognition methods provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0233] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the methods provided in various optional implementations of the above-mentioned speech recognition or audio recognition aspects.

[0234] The above is a detailed introduction to a speech recognition method, device, electronic device and computer-readable storage medium provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A speech recognition method, characterized in that: include: Acquire current audio stream data, and segment the current audio stream data to obtain multiple segments of target audio data; Performing feature extraction on the target audio data to obtain acoustic features of the current audio stream data, and generating initial text corresponding to the current audio stream data based on the acoustic features; Detecting at least one text key frame in the current audio stream data, and identifying a segmentation position of at least one audio data block in the current audio stream data based on the text key frame; Segmenting the acoustic feature according to the segmentation position to obtain a target acoustic feature of each audio data block; Based on the target acoustic features, the initial text is updated to obtain the target text corresponding to the current audio stream data.

2. The speech recognition method according to claim 1, characterized in that: The audio stream data includes multiple audio frames, and the detecting at least one text key frame in the current audio stream data includes: Based on the acoustic features, predicting the posterior probability of the text content corresponding to each audio frame; Detecting a peak position of at least one peak in the current audio stream data according to the posterior probability; At least one audio frame corresponding to the peak position is selected from the audio frames to obtain a text key frame corresponding to each peak.

3. The speech recognition method according to claim 2, characterized in that: The step of identifying a segmentation position of at least one audio data block in the current audio stream data based on the text key frame comprises: Grouping the peaks according to the peak positions to obtain a plurality of peak groups, wherein the peak groups include a preset number of peaks; Based on the text key frame, identifying the frame number interval of the audio data block corresponding to each peak group in the current audio stream data; The frame number interval is enlarged to obtain a target frame number interval, and the target frame number interval is used as a segmentation position of the audio data block.

4. The speech recognition method according to claim 3, characterized in that: The step of identifying, based on the text key frame, a frame number interval of an audio data block corresponding to each peak group in the current audio stream data comprises: According to the peak positions, selecting a starting peak and an ending peak from the peak group; Identifying the frame number of the text key frame corresponding to the starting peak in the current audio stream data to obtain the starting frame number; Identifying the frame number of the text key frame corresponding to the end peak in the current audio stream data to obtain the end frame number; The interval formed by the starting frame number and the ending frame number is used as the frame number interval of the audio data block corresponding to the peak group.

5. The speech recognition method according to claim 4, characterized in that: The step of amplifying the frame number interval to obtain a target frame number interval includes: Subtract a preset first frame number from the starting frame number to obtain a target starting frame number; Add the end frame number to a preset second frame number to obtain a target end frame number; The interval formed by the target start frame number and the target end frame number is used as the target frame number interval.

6. The speech recognition method according to claim 3, characterized in that: The acoustic feature includes an acoustic sub-feature corresponding to each audio frame, and the acoustic feature is segmented according to the segmentation position to obtain a target acoustic feature of each audio data block, including: Filtering out at least one audio frame corresponding to the target frame number interval from the audio frames to obtain a set of candidate audio frames corresponding to each audio data block; Extracting acoustic sub-features corresponding to each audio frame in the candidate audio frame set from the acoustic features, and obtaining an acoustic sub-feature set corresponding to each audio data block; The acoustic sub-feature set is used as the target acoustic feature corresponding to the audio data block.

7. The speech recognition method according to any one of claims 1 to 6, characterized in that: The updating of the initial text based on the target acoustic feature to obtain the target text corresponding to the current audio stream data includes: Based on the target acoustic feature, determining a target field corresponding to each audio data block; splicing the target fields according to the segmentation positions to obtain a spliced ​​text; The initial text is replaced with the concatenated text to obtain the target text corresponding to the current audio stream data.

8. The speech recognition method according to any one of claims 1 to 6, characterized in that: The extracting features of the target audio data to obtain acoustic features of the current audio stream data includes: Using a speech recognition model to extract features from the target audio data to obtain initial acoustic features corresponding to each target audio data; The initial acoustic features are concatenated to obtain acoustic features corresponding to the current audio stream data.

9. The speech recognition method according to claim 8, characterized in that: Before extracting features from the target audio data using the speech recognition model to obtain initial acoustic features corresponding to each target audio data, the method further includes: Acquire multiple audio data samples corresponding to the speech samples, and use a preset speech recognition model to perform feature extraction on the audio data samples to obtain sample acoustic features of the speech samples; Based on the acoustic features of the sample, predict the text content corresponding to the speech sample to obtain a predicted text; According to the predicted text, the recognition loss of the speech sample is determined, and based on the recognition loss, the preset speech recognition model is converged to obtain a speech recognition model.

10. The speech recognition method according to claim 8, characterized in that: Before extracting features from the audio data sample using a preset speech recognition model to obtain sample acoustic features of the speech sample, the method further includes: Acquire an initial speech recognition model, wherein the initial speech recognition model includes a temporal classification network; When the time series classification network includes a real-time time series classification network and a non-real-time time series classification network, the initial speech recognition model is used as a preset speech recognition model; When the initial speech recognition model is a pre-trained model and the time series classification network is the non-real-time time series classification network, the non-real-time time series classification network is copied, and the copied non-real-time time series classification network is added to the initial speech recognition model as a real-time time series classification network to obtain a preset speech recognition model.

11. The speech recognition method according to claim 10, characterized in that: The predicting text content corresponding to the speech sample based on the acoustic features of the sample to obtain predicted text includes: Based on the acoustic features of the sample, a real-time time series classification network is used to predict the text content of the speech sample to obtain real-time predicted text, and a sample segmentation position of at least one sample audio data block is identified in the speech sample; Segmenting the sample acoustic features according to the sample segmentation positions to obtain target sample acoustic features corresponding to each sample audio data block; Based on the acoustic features of the target sample, the non-real-time temporal classification network is used to predict the text content of the speech sample to obtain the current predicted text, and the real-time predicted text and the current predicted text are used as the predicted text.

12. The speech recognition method according to claim 11, characterized in that: The step of identifying a sample segmentation position of at least one sample audio data block in the speech sample comprises: According to the sample acoustic features, detecting a sample text key frame corresponding to at least one sample peak in the speech sample; Determining a target number of groups corresponding to the sample peaks in a preset number set, and grouping the sample peaks based on the target number of groups to obtain at least one sample peak group; According to the sample text key frame, the sample segmentation position of the sample audio data block corresponding to each sample peak group is identified in the speech sample.

13. The speech recognition method according to claim 11, characterized in that: The method of predicting the text content of the speech sample based on the acoustic features of the target sample by using the non-real-time time series classification network to obtain the current predicted text includes: Extracting text features of multiple dimensions from the target sample acoustic features; Based on the text features, using the non-real-time time series classification network to predict the text content of the speech sample to obtain non-real-time predicted text; The text features are concatenated to obtain current text features corresponding to each sample audio data block, and an attention weight of the current text features is determined; Based on the attention weight, the text content corresponding to the current text feature is determined to obtain auxiliary predicted text, and the auxiliary predicted text and the non-real-time predicted text are used as the current predicted text.

14. The speech recognition method according to claim 13, characterized in that: The step of determining the recognition loss of the speech sample according to the predicted text includes: Acquire the annotated text of the speech sample, and compare the annotated text with the real-time predicted text to obtain the real-time predicted loss; Comparing the annotated text with the non-real-time prediction text to obtain the non-real-time prediction loss, and comparing the annotated text with the auxiliary prediction text to obtain the auxiliary prediction loss; The real-time prediction loss, the non-real-time prediction loss and the auxiliary prediction loss are integrated to obtain the recognition loss of the speech sample.

15. The speech recognition method according to claim 14, characterized in that: The step of fusing the real-time prediction loss, the non-real-time prediction loss and the auxiliary prediction loss to obtain the recognition loss of the speech sample includes: Obtaining a loss fusion weight, and based on the loss fusion weight, weighting the real-time prediction loss, the non-real-time prediction loss, and the auxiliary prediction loss respectively; Calculate the average of the weighted real-time prediction loss and the weighted non-real-time prediction loss to obtain the target prediction loss; The target prediction loss and the weighted auxiliary prediction loss are fused to obtain the recognition loss of the speech sample.

16. The speech recognition method according to claim 9, characterized in that: The speech sample includes multiple sample audio frames, and the step of obtaining multiple audio data samples corresponding to the speech sample includes: The speech sample is segmented by a preset audio mask and a preset segmentation frame number to obtain a plurality of sample audio frame sets; Filtering out at least one historical sample audio frame corresponding to each sample audio frame set from the sample audio frames; The historical sample audio frames are added to the corresponding sample audio frame sets to obtain multiple target sample audio frame sets, and the target sample audio frame sets are used as audio data samples.

17. A speech recognition device, characterized in that: include: An acquisition unit, used to acquire current audio stream data and segment the current audio stream data to obtain multiple segments of target audio data; A generating unit, configured to extract features from the target audio data to obtain acoustic features of the current audio stream data, and generate an initial text corresponding to the current audio stream data based on the acoustic features; an identification unit, configured to detect at least one text key frame in the current audio stream data, and identify a segmentation position of at least one audio data block in the current audio stream data based on the text key frame; A segmentation unit, configured to segment the acoustic feature according to the segmentation position to obtain a target acoustic feature of each audio data block; An updating unit is used to update the initial text based on the target acoustic feature to obtain a target text corresponding to the current audio stream data.

18. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the speech recognition method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the speech recognition method according to any one of claims 1 to 16 are implemented.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the speech recognition method according to any one of claims 1 to 16.

Images