Speech recognition method and device, computer readable storage medium

Abstract

Disclosed are a speech recognition method and device, and a computer readable storage medium. In an embodiment of the present application, the speech recognition method can include: obtaining a speech to be recognized; segmenting the speech to be recognized to obtain a plurality of speech segments, adjacent speech segments of the plurality of speech segments overlapping at a tail of a preceding speech segment and a head of a following speech segment; using a speech recognition model based on an attention mechanism to obtain attention data and an initial text segment of each speech segment in the plurality of speech segments; extracting voice data of each speech segment from the attention data of each speech segment; obtaining a corrected text segment of each speech segment according to the initial text segment, the voice data, and an overlapping duration of each speech segment; and splicing the corrected text segments of each speech segment in the plurality of speech segments to obtain a recognized text of the speech to be recognized. The present application can achieve high-accuracy long-speech continuous recognition without a high-complexity model such as VAD.

Description

TECHNICAL FIELD

[0001] The present application relates to a speech recognition method and device, and a computer readable storage medium. BACKGROUND

[0002] With the development of research and engineering technology in the field of artificial intelligence, speech recognition (ASR) technology, as one of the technical means in human-computer interaction, has gradually entered people's daily life. Speech recognition capability has become one of the indispensable capabilities of terminal devices such as mobile phones, sound boxes, vehicle-mounted systems, and televisions.

[0003] In the practical application of speech recognition technology, according to the difference of speech technology application scene, speech recognition can be further divided into short speech recognition and long speech recognition. In long speech recognition, due to the limitation of the running memory and computing power of the computing device, too long audio cannot be directly sent to the speech recognition engine for speech recognition. In the face of such a problem, the traditional method is to deploy a speech recognition engine at the same time as deploying an endpoint detection engine, namely a voice activity detection (VAD) engine. However, due to the problems of harsh audio segmentation, limited recognition accuracy of speech endpoints, susceptibility to noise interference, and ambiguous segmentation semantics of the VAD model, the accuracy of long speech recognition results is low. SUMMARY

[0004] In view of the above problems of the related art, the present application provides a speech recognition method and device, and a computer readable storage medium, which can achieve high-accuracy long speech continuous recognition without high-complexity models such as VAD.

[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a speech recognition method, comprising:

[0006] obtaining a speech to be recognized;

[0007] segmenting the speech to be recognized to obtain a plurality of speech segments, the tail of a preceding speech segment and the head of a following speech segment in adjacent speech segments of the plurality of speech segments overlap;

[0008] obtaining attention data and an initial text segment of each speech segment in the plurality of speech segments by using a speech recognition model based on an attention mechanism;

[0009] extracting voice data of each speech segment from the attention data of each speech segment;

[0010] According to the initial text segment of each voice segment, the human voice data, and the overlap length, a corrected text segment of each voice segment is obtained, and the text corresponding to the tail part in the corrected text segment of the preceding voice segment in the adjacent voice segments is the same as the text corresponding to the head part in the corrected text segment of the following voice segment.

[0011] The corrected text segments of the voice segments are spliced to obtain the recognition text of the to-be-recognized voice.

[0012] In this way, a high-accuracy long-voice continuous recognition can be achieved without a high-complexity model such as VAD.

[0013] As a possible implementation manner of the first aspect, the splitting the to-be-recognized voice to obtain a plurality of voice segments specifically comprises: splitting the to-be-recognized voice according to a fixed window length and / or a fixed overlap length to obtain the plurality of voice segments with equal time length and / or overlap length.

[0014] In this way, the parallelism of voice segment processing can be improved, and the processing efficiency of the method of the embodiments of the present application can be improved, and the situation that the voice recognition model crashes due to the audio being too long after splitting can be eliminated.

[0015] As a possible implementation manner of the first aspect, the voice recognition model based on the attention mechanism is an encoder-decoder structure model, the encoder-decoder structure model comprises an encoder and a decoder, an attention module is arranged in a plurality of decoding layers of the decoder, and the attention data is obtained by an attention matrix output by the attention module in the last layer of the plurality of decoding layers.

[0016] In this way, better human voice features can be extracted with less computation.

[0017] As a possible implementation manner of the first aspect, the voice recognition model based on the attention mechanism is obtained by training a multi-target loss function, and the multi-target loss function comprises at least one loss function with frame alignment capability.

[0018] In this way, the voice recognition model can have higher recognition accuracy, and the hidden state output by the encoder can contain frame alignment information, and the attention data obtained by the decoder can contain clearer human voice interval information, so that high-precision human voice data can be obtained from the attention data.

[0019] As a possible implementation manner of the first aspect, the attention data has word dimension and frame dimension, and the human voice data is extracted from the attention data of each voice segment specifically by:

[0020] traversing the attention data according to the word dimension to extract an attention vector of the word dimension;

[0021] obtaining a vocal vector of the word dimension according to the attention vector of the word dimension and a preconfigured threshold value;

[0022] performing accumulation summation on values in the vocal vector of the word dimension corresponding to each of the speech segments to obtain a vocal sequence of each of the speech segments, the vocal sequence comprising vocal information of each audio frame in the speech segment, the vocal information being used to indicate whether the audio frame belongs to a vocal or not.

[0023] In this way, vocal data with an audio frame as a data granularity can be extracted without a high-complexity model such as a VAD model, which not only ensures continuity and accuracy of speech recognition, but also avoids misrecognition of the VAD model in a noisy scene and reduces development and maintenance costs of the VAD model.

[0024] As a possible implementation manner of the first aspect, the vocal vector of the word dimension is obtained by the following formula:

[0025]

[0026] wherein Thred a represents the threshold value, M s [l, t] represents a value of an audio frame t in the vocal vector of the word dimension l, M a [l, t] represents a value of an audio frame t in the attention vector of the word dimension l.

[0027] In this way, a vocal vector can be obtained from an attention vector through simple threshold value judgment.

[0028] As a possible implementation manner of the first aspect, the obtaining of the revised text segment of each of the speech segments according to the initial text segment of each of the speech segments, the vocal data and the overlap length specifically comprises:

[0029] The following steps are performed for each pair of adjacent speech segments in the plurality of speech segments:

[0030] extracting overlapping texts from the initial text segments of the adjacent speech segments, the overlapping texts comprising overlapping texts of a preceding speech segment and overlapping texts of a following speech segment in the adjacent speech segments, the overlapping texts of the preceding speech segment corresponding to a tail part of the vocal data of the preceding speech segment with a length of the overlap duration, and the overlapping texts of the following speech segment corresponding to a head part of the vocal data of the following speech segment with a length of the overlap duration;

[0031] aligning the overlapping text of the former speech segment with the overlapping text of the latter speech segment to obtain aligned text of the adjacent speech segments, the aligned text comprising aligned text of the former speech segment and aligned text of the latter speech segment;

[0032] corrected text of the adjacent speech segments according to the confidence of the word in the aligned text of the former speech segment and the confidence of the word in the aligned text of the latter speech segment, the corrected text of the former speech segment being the same as the corrected text of the latter speech segment;

[0033] obtaining the corrected post-text segment of the former speech segment and the corrected post-text segment of the latter speech segment in the adjacent speech segments using the corrected text of the adjacent speech segments.

[0034] Thus, by aligning and correcting the recognition result of the overlapping area of the speech segments, the recognition accuracy at the division of the speech segment segmentation can be improved, and the overall recognition accuracy of the speech to be recognized can be improved.

[0035] As a possible implementation manner of the first aspect, the confidence of the word at least comprises one of the following: frame alignment confidence of the word, attention confidence of the word, language confidence of the word, and position confidence of the word.

[0036] Thus, the performance of the speech recognition model, the language logic, and various factors such as the position of the word can be fused to correct the text.

[0037] As a possible implementation manner of the first aspect, the corrected text of the adjacent speech segments is obtained according to the confidence of the word in the aligned text of the former speech segment and the confidence of the word in the aligned text of the latter speech segment, and specifically comprises: obtaining the corrected text of the adjacent speech segments according to the comprehensive score of the word in the aligned text of the former speech segment and the comprehensive score of the word in the aligned text of the latter speech segment; wherein the comprehensive score of the word is determined by taking the position confidence of the word as a penalty term.

[0038] Thus, by introducing the position confidence of the word as a penalty term of the comprehensive score of the word, the influence of poor recognition effect at the short speech endpoint on the overall recognition accuracy of the long speech can be corrected.

[0039] As a possible implementation manner of the first aspect, the position confidence of the word is obtained by the following formula:

[0040] Posscore = -β | | - L / 2 |

[0041] Posscore = β × (l / L) where Posscore represents a position confidence value of the word, l represents a position of the word in the aligned text segment, L represents a number of words contained in the aligned text segment, and β represents a pre-set position weight.

[0042] Thus, the position confidence value is represented by a negative number, and the recognition result near the segmentation boundary can be punished to some extent, which can compensate for the poor speech recognition result at the segmentation boundary, and further effectively improve the overall recognition accuracy of the to-be-recognized speech recognition result.

[0043] As a possible implementation manner of the first aspect, the comprehensive score of the word is obtained by the following formula:

[0044] Jointscore = α × CTCscore + λ × Attscore + η × LMscore + Posscore

[0045] where Jointscore represents the comprehensive score of the word, CTCscore represents a frame alignment confidence of the word, Attscore represents an attention confidence of the word, LMscore represents a language confidence value of the word, Posscore represents a position confidence value of the word, α represents a weight of the frame alignment confidence of the word, λ represents a weight of the attention confidence of the word, and η represents a weight of the language confidence of the word.

[0046] Thus, the comprehensive score of the word can be determined by combining the frame alignment, the attention, the language, and the word position, and the like, by using the pre-set weights through addition and multiplication, which is low in hardware cost and easy to implement.

[0047] As a possible implementation manner of the first aspect, the modified text of the adjacent speech segments is obtained according to the comprehensive score of the word in the aligned text of the preceding speech segment and the comprehensive score of the word in the aligned text of the subsequent speech segment, and specifically includes:

[0048] Each word in the aligned text in the adjacent speech segments is adjusted by the following formula:

[0049]

[0050] Wherein, Uri[l](i=1,......N-1) represents a word at position 1 in the revised text of the speech segment Ai, N is the number of the speech segments, UPai[l] represents a word at position 1 in the alignment text of the speech segment Ai, UPai+1[l] represents a word at position 1 in the alignment text of the speech segment Ai+1, Jointscore(UPai[l]) represents a comprehensive score of the word at position 1 in the alignment text of the speech segment Ai, Jointscore(UPai+1[l]) represents a comprehensive score of the word at position 1 in the alignment text of the speech segment Ai+1, and * represents a placeholder.

[0051] Therefore, the text revision can be realized by only adjusting the texts of the overlapping parts in adjacent speech segments, which not only improves the recognition accuracy of the speech segment division boundary and the overall recognition accuracy of the speech to be recognized, but also reduces the computational complexity and is conducive to reducing the hardware cost and improving the processing efficiency.

[0052] As a possible implementation manner of the first aspect, the method further includes: obtaining a confidence of the recognized text of the speech to be recognized according to the confidence of the word in the revised text segment.

[0053] Therefore, the confidence of the recognized text of the speech to be recognized can be provided to the user at the same time as the recognized text of the speech to be recognized is provided to the user, which facilitates the user to generate the transcription text of the speech by referring to the confidence.

[0054] As a possible implementation manner of the first aspect, the method further includes: obtaining a vocal confidence of each speech segment in the plurality of speech segments by using the attention data.

[0055] Therefore, the corresponding vocal confidence can be provided to the user or the device thereof at the same time as the vocal data of the speech to be recognized is provided to the user or the device thereof, which facilitates the user or the device thereof to generate the transcription text of the speech by referring to the vocal confidence and the vocal data.

[0056] The second aspect of the present application provides a computing device, including: at least one processor; and at least one memory storing program instructions, which, when executed by the at least one processor, cause the at least one processor to execute the speech recognition method described above.

[0057] The third aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to execute the speech recognition method described above.

[0058] These and other aspects of the present application will become more fully understood from the following (multiple) embodiment descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0059] Various features and relationships between features of the present application will be further described below with reference to the drawings. The drawings are diagrammatic and some features can not be to scale, and some of the drawings can omit, for clarity, features related to the pertinent art or essential to the present application, or can add, for clarity, features not essential to the present application. The combination of features shown in the drawings is not intended to limit the present application. Also, like reference numerals are intended to refer to like features throughout the specification. The drawings are as follows:

[0060] Figure 1 is a flowchart of a speech recognition method according to an embodiment of the present application;

[0061] Figure 2 is a flowchart of an exemplary implementation process of a speech recognition method according to an embodiment of the present application;

[0062] Figure 3 is an exemplary diagram of speech window length and overlap segmentation according to an embodiment of the present application;

[0063] Figure 4 is an exemplary network structure of a speech recognition model according to an embodiment of the present application;

[0064] Figure 5 is an exemplary diagram of attention data according to an embodiment of the present application;

[0065] Figure 6 is an exemplary flowchart of extracting vocal data according to an embodiment of the present application;

[0066] Figure 7 is an exemplary flowchart of aligning and correcting an initial text segment according to an embodiment of the present application;

[0067] Figure 8 is a structural diagram of a speech recognition apparatus according to an embodiment of the present application;

[0068] Figure 9 is a structural diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The words "first", "second", "third", etc. or module A, module B, module C, etc. or similar words in the specification and claims are only used to distinguish similar objects, and can be understood that the specific order or sequence can be interchanged as allowed, so that the present application described herein can be implemented in an order other than that illustrated or described herein.

[0070] In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown by way of illustration specific embodiments in which reference numerals designate the same parts in the various figures thereof. It is to be understood that other embodiments can be utilized and that structural or logical changes can be made without departing from the scope of the present disclosure. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the present disclosure is defined by the appended claims.

[0071] The term "comprising" as used in the specification and in claims includes that the specified features, integers, steps or components are present, but not excluding the presence or addition of one or more other features, integers, steps, components or groups thereof. Therefore, the term "comprising" as used in the specification and claims should not be interpreted as being restricted to security devices consisting only of the recited features.

[0072] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to different embodiments. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The meaning and definition of the terms posted in the specification or derived from the context are prevail, in case of inconsistency. In addition, the terms used herein are for the purpose of describing the embodiments of the present application only and are not intended to limit the present application.

[0074] In order to accurately describe the technical content in the present application, and in order to accurately understand the present application, the following explanations or definitions of the terms used in the specification are given before the specific embodiments are described.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The meaning and definition of the terms posted in the specification or derived from the context are prevail, in case of inconsistency. In addition, the terms used herein are for the purpose of describing the embodiments of the present application only and are not intended to limit the present application.

[0076] In order to accurately describe the technical content in the present application, and in order to accurately understand the present application, the following explanations or definitions of the terms used in the specification are given before the specific embodiments are described.

[0077] Long speech, i.e. longer speech, also i.e. speech that cannot be directly recognized by the ASR model under the existing running memory and computing capacity of the computing device. For example, speech of 10 seconds or more.

[0078] Long speech recognition, i.e. recognition of long speech. For example, speech recognition of audio of tens of minutes or even hours, such as conference recording, telephone recording, or movie, television audio, etc.

[0079] Short speech, i.e. shorter speech, also i.e. speech that can be directly recognized by the ASR model under the existing running memory and computing capacity of the computing device. For example, speech of less than 10 seconds.

[0080] Short speech recognition, i.e. recognition of short speech, the speech text content is mainly simple to call, play programs, play music, navigation, etc. vertical field content. The voice assistant of a mobile phone, a sound box, and a television is a typical short speech recognition application.

[0081] ASR model, a machine learning model capable of recognizing short speech into text.

[0082] VAD model, a voice activity detection algorithm that recognizes the starting point and ending point of long speech, and then divides the long speech into multiple continuous short speech paragraphs, which can distinguish speech segments and non-speech segments, but ignores the semantic integrity of long speech. When a user's speech contains a pause due to thinking or hesitation, the VAD model will mistakenly recognize the user's pause as the end point of the speech, and then divide a complete speech into two short speech paragraphs with incomplete semantics, reducing the speech recognition effect of the short speech paragraph.

[0083] Voice activity detection algorithm, including a short-time energy and short-time zero-crossing rate analysis method or a method of classifying speech and non-speech based on a neural network model. Among them, the short-time energy and short-time zero-crossing rate method involves the selection of multiple threshold values, which requires more manpower and time to optimize to obtain better results, and these threshold values usually have greater differences in different use scenarios. The method of classifying speech and non-speech based on a neural network model, the detection accuracy largely depends on the training corpus of the model, when the actual deployment scenario of the model has a large difference with the training corpus, the voice activity detection algorithm is difficult to achieve effective accuracy.

[0084] Shuffle algorithm (Fisher), a dynamic programming algorithm.

[0085] Connectionist Temporal Classification (CTC) loss function, a loss function with frame alignment capability, can automatically align data without alignment, mainly used in training serialized data without prior alignment. For example, speech recognition, optical character recognition (OCR), etc.

[0086] Transducer loss function, a loss function with frame alignment capability, commonly used in training of sequence-to-sequence models.

[0087] KL divergence (Kullback-Leibler divergence) loss function, which calculates the KL divergence between the input and the target value. KL divergence is an indicator of the degree of matching between two probability distributions. The closer the two probability distributions, the smaller the KL divergence.

[0088] Encoder-decoder neural network model, including an encoder (also known as an encoding module or an encoding network) and a decoder (also known as a decoding module or a decoding network), a sequence-to-sequence model based on a recurrent neural network, whose input of the next layer depends on the output of the previous layer.

[0089] Attention mechanism, which determines the attention probability distribution (i.e., attention distribution) of the decoder output based on the matching degree between the current input and output of the model, such as the decoder. The higher the matching degree, the higher the score of the concentrated point of the attention distribution. Attention mechanism is more commonly used as a component of a neural network.

[0090] Multi-head attention, which uses multiple queries to select multiple groups of information from input information in parallel, with each attention focusing on a different part of the input information.

[0091] Transformer module, a neural network model based on attention mechanism, which can be an encoder-decoder structure based on multi-head attention mechanism, or only contain one of encoder or decoder.

[0092] Vocal data, data used to indicate which audio frames in a speech segment belong to vocals and which audio frames do not belong to vocals, including multiple pieces of vocal information, each piece of vocal information corresponding to an audio frame in the speech segment. The value of the vocal information can be used to indicate whether the corresponding audio frame belongs to vocals.

[0093] Related technology one: a continuous speech human-computer interaction method and system, the disclosed technical solution is: receiving the continuous speech signal input by the user; based on the long speech segmentation technology, the continuous speech signal is segmented into multiple short speeches; based on the dynamic language model, the multiple short speeches are recognized, and a corresponding recognition result is generated for each short speech; based on the context semantic analysis technology, the recognition result is semantically completed, and a semantic completion result is generated; and based on the dialogue management technology, a question and answer sentence corresponding to the semantic completion result is generated.

[0094] The defects of related technology one include: relying on the VAD model to segment long speeches, due to the limited accuracy of the VAD model and the harshness of audio segmentation, the overall accuracy of the speech recognition result is poor, and the long speech is often incorrectly segmented, resulting in poor recognition accuracy of each short speech.

[0095] Related technology two: a long speech continuous recognition and real-time feedback method and system, the disclosed technical solution is: adding an extension possibility from the end point of a sentence to the starting point of recognition in the recognition network; in the decoding process, the recognition path can generate a single sentence or multiple continuous sentences when expanding, and the overall optimization of acoustic and language probability is used to segment long speech signals; periodically detecting the common part of the optimal historical path of all active nodes; obtaining the recognition word sequence that has been fixed at the current time; feeding back the updated local recognition result to the user in real time; recycling the decoding space corresponding to the recognized part that has been determined.

[0096] The defects of related technology two include: based on the traditional speech recognition method based on the hidden Markov model, the decoding network is improved, the semantics, tone and silence duration of each frame are fused to recognize the speech end point, which is not applicable to the end-to-end deep learning speech recognition model (for example, ASR).

[0097] Related technology three: limited by the running memory and computing power of the computing device, long speech signals cannot be input into the ASR model for speech recognition. To solve this problem, an endpoint detection engine, i.e. a VAD engine, can be deployed at the same time as the speech recognition engine. The VAD engine can identify the speech starting point and ending point in the long speech, and then segment the long speech into multiple continuous short speech paragraphs, and the obtained short speech paragraphs are sent into the ASR engine one by one for speech recognition, and finally the recognized text of the ASR engine is spliced to obtain the recognition result of the long speech.

[0098] The defects of related technology three mainly include the following three:

[0099] 1) The speech endpoint recognition accuracy of the VAD engine is limited, and more manpower and time are needed to optimize to obtain better results.

[0100] 2) VAD engine implements the discrimination of speech segments and non-speech segments, ignoring the semantic integrity in speech. When a pause occurs in a long speech due to user thinking or hesitation, the VAD engine often mistakenly identifies the pause of the user as the end point of speech, and further divides a complete long speech into two semantically incomplete segments. Such semantically incomplete short speech input into the ASR engine usually has a poor recognition result.

[0101] 3) In addition to the problems of limited recognition accuracy and ambiguous segmentation semantics, the VAD model is also susceptible to noise interference, which also adversely affects the accuracy of long speech recognition results.

[0102] Therefore, the basic idea of the embodiments of the present application is to provide a speech recognition method and device, and a computer readable storage medium, which first performs overlapping segmentation on the speech to be recognized to obtain a plurality of speech segments overlapping at the segmentation positions, then obtains attention data and initial text segments of the plurality of speech segments through a speech recognition model based on an attention mechanism, extracts voice data of the speech segments through the attention data of the speech segments, and aligns and corrects the initial text segments based on the voice data and the overlapping duration to obtain corrected text segments, and finally obtains the recognition text of the speech to be recognized by using the corrected text segments. Thus, the embodiments of the present application can realize speech segmentation and obtain voice data without relying on a VAD model, and solve the problem of low recognition accuracy at the segmentation end point caused by segmentation through the combination of overlapping segmentation and text correction, effectively improving the accuracy of long speech continuous recognition. According to experiments, the word accuracy of the embodiments of the present application can reach 89%. It can be seen that the embodiments of the present application can realize long speech continuous recognition with high accuracy without a VAD model.

[0103] The embodiments of the present application can be applied to various scenarios requiring conversion of continuous speech to text. In particular, the embodiments of the present application are suitable for scenarios requiring automatic conversion of audio data of an ultra-long time to text, thereby replacing or reducing manual audio transcription. For example, the embodiments of the present application can be applied to transcription scenarios such as conference recording, court recording, and interview recording, and can also be applied to application scenarios such as audio and video subtitle generation, and can effectively improve the transcription efficiency of long speech to text in these scenarios.

[0104] Figure 1 An exemplary flow of a speech recognition method provided by the embodiments of the present application is shown. Referring to FIG. 1, Figure 1 As shown in FIG. 1, the exemplary speech recognition method of the embodiments of the present application can include the following steps:

[0105] Step S101, obtaining speech to be recognized;

[0106] In step S102, the speech to be recognized is segmented to obtain a plurality of speech segments, and a tail of a preceding speech segment and a head of a following speech segment in adjacent speech segments of the plurality of speech segments overlap.

[0107] In step S103, an attention data and an initial text segment of each speech segment in the plurality of speech segments are obtained by using a speech recognition model based on an attention mechanism.

[0108] In step S104, human voice data of each speech segment is extracted from the attention data of the speech segment.

[0109] In step S105, a corrected text segment of each speech segment is obtained according to the initial text segment, the human voice data and the overlap duration of the speech segment, and a text corresponding to the tail in the corrected text segment of the preceding speech segment in the adjacent speech segments of the plurality of speech segments is the same as a text corresponding to the head in the corrected text segment of the following speech segment.

[0110] In step S106, the corrected text segments of the plurality of speech segments are spliced to obtain a recognized text of the speech to be recognized.

[0111] The speech recognition method of the embodiment of the present application can realize long speech continuous recognition with high accuracy without high complexity models such as VAD models. Experiments show that the word accuracy of the method of the embodiment of the present application can reach 89%. The word accuracy of the traditional method is only 72%.

[0112] The embodiment of the present application can be applied not only to long speech transcription but also to short speech transcription. That is, the speech to be recognized herein can be long speech or short speech.

[0113] Various segmentation methods can be applied to the embodiment of the present application, as long as the plurality of speech segments obtained by segmentation overlap at the segmentation boundary. By using the overlapping segmentation method, the texts corresponding to the segmentation boundary in the initial text segments of adjacent speech segments overlap. In this way, the continuity of speech recognition can be ensured, and the initial text segments can be aligned and corrected by using the overlapping part of the initial text segments, thereby effectively improving the poor recognition accuracy at the segmentation boundary of the speech segments, and improving the accuracy of speech recognition while realizing continuous speech recognition.

[0114] In some embodiments, the speech to be recognized can be segmented according to a predetermined window length and a predetermined overlap length in step S102. The predetermined window length can be a fixed value or a variable value, which can be dynamically adjusted according to different application scenarios, characteristics of the speech to be recognized (e.g., length, language, type of speaker, etc.), hardware performance (e.g., memory size, processor performance, memory capacity and read-write performance, etc.), model performance (e.g., maximum length of speech that can be processed by a speech recognition model at a time), and / or any other factor related to speech recognition. Similarly, the predetermined overlap length can also be a fixed value or a variable value, which can be determined according to the predetermined window length and in combination with the above factors.

[0115] In an implementation, the predetermined window length and the predetermined overlap length can be determined by the performance of a speech recognition model (e.g., an ASR model) and the audio scenario. The values of the predetermined window length and the predetermined overlap length can be determined using various methods applicable to speech recognition scenarios. For example, they can be obtained by grid search experiments on a small sample. For another example, they can be directly set to empirical values.

[0116] In an implementation, the speech to be recognized can be segmented according to a fixed window length and / or a fixed overlap length in step S102 to obtain a plurality of speech segments with equal time lengths and / or equal overlap lengths. In this embodiment, the segmentation method using a fixed length or a fixed overlap length can obtain speech segments with equal lengths and / or equal overlap lengths, which can improve the parallelism of speech segment processing and thus improve the processing efficiency of the method of the present embodiment, while eliminating the situation that the speech recognition model crashes due to excessively long audio segments.

[0117] In step S103, the speech recognition model based on the attention mechanism can be any sequence-to-sequence model applicable to the embodiments of the present application. In some embodiments, the speech recognition model can be a neural network model based on the attention mechanism. In some implementations, the speech recognition model can be, but is not limited to, a recurrent neural network model based on the attention mechanism. In this implementation, the speech recognition model can be a model with an encoder-decoder structure, which includes an encoder and a decoder, and an attention module is arranged therein. The attention module can be a single-layer structure or a multi-layer structure, and can use a multi-head attention mechanism or a single-head attention mechanism. Alternatively, the speech recognition model can also use an encoder structure or any other network structure applicable to speech recognition. The specific implementation of the speech recognition model can be referred to the relevant embodiments below. It can be understood that the description of the speech recognition model herein and in the relevant specific implementations below is only by way of example and is not intended to limit the present application. In actual applications, the speech recognition model can be any model with speech recognition capability and is not limited to a neural network.

[0118] The speech recognition model based on the attention mechanism can be trained by a predefined loss function.

[0119] In some embodiments, the speech recognition model based on the attention mechanism can be trained by a multi-objective loss function (i.e., a hybrid loss function), which at least includes a loss function with frame alignment capability. In this way, not only the recognition accuracy of the speech recognition model can be improved, but also the frame alignment capability of the speech recognition model can be improved at the same time, so as to obtain more accurate vocal data (e.g., vocal data at the granularity of audio frames described below).

[0120] Taking an encoder-decoder model based on a multi-head attention mechanism as an example, the encoder and the decoder in the speech recognition model can adopt different loss functions. The loss function of the encoder can adopt a loss function with forced alignment characteristics. This not only enables the speech recognition model to have higher recognition accuracy, but also enables the hidden state output by the encoder to contain frame alignment information, so that the attention data obtained by the decoder contains clearer vocal interval information, so as to obtain high-precision vocal data from the attention data. In some examples, the decoder can adopt a KL divergence loss function, and the encoder can adopt a CTC loss function or a Transducer loss function, as described in detail in the relevant specific embodiments below.

[0121] In some embodiments, the speech recognition model can be trained by a single loss function. For example, the single loss function can be, but is not limited to, a Transducer loss function, a KL divergence loss function, a CTC loss function, or others.

[0122] In step S103, the initial text segment of each speech segment can be obtained by performing speech recognition on the speech segment by the speech recognition model. At the same time, the attention module in the speech recognition model obtains the attention data of each speech segment. The attention data can indicate the probability of each audio frame in the speech segment corresponding to a word in the preset word table. The higher the probability value, the greater the possibility that the audio frame corresponds to the corresponding word in the preset word table, and the higher the probability that the audio frame belongs to the vocal. That is, the attention data contains the vocal features of each audio frame in the speech segment.

[0123] The attention data can be a matrix with word dimension and frame dimension. The dimension number of the word dimension can be equal to the number of words in the preset word table, and the frame dimension can be equal to the number of audio frames in the speech segment. If a fixed window length is used for segmentation in step S102, the lengths of the speech segments are the same, and the dimension numbers of the attention data of the speech segments are also the same, which can improve the parallelism of the method of the embodiments of the present application.

[0124] When the speech recognition model adopts the single-head attention mechanism, the attention data can be an attention matrix obtained based on the single-head attention mechanism. In this embodiment, the attention data is obtained by using the single-head attention mechanism, the amount of data is small, the operation is less, the computational complexity is low, the requirement for hardware performance is low, and the overall processing efficiency of the method of the present application can be improved while reducing the hardware cost.

[0125] When the speech recognition model adopts the multi-head attention mechanism, the attention data can be a matrix obtained by performing an operation (for example, averaging, taking the mean square deviation, etc.) on a group of attention matrices obtained by the multi-head attention mechanism. The number of attention matrices in the group of attention matrices is equal to the number of heads of the multi-head attention mechanism, and each attention matrix can contain the vocal characteristics of the audio frames of the speech segment in different aspects. In this embodiment, the attention data is obtained by using the multi-head attention mechanism, and the vocal characteristics of the audio frames of the speech segment can be extracted in multiple aspects through multiple attention matrices. That is, the obtained attention data can more comprehensively and accurately represent the vocal characteristics of the speech segment, which helps to obtain more accurate vocal data, improves the accuracy of text correction, and further improves the accuracy of the recognized speech recognition result.

[0126] When the multiple layers in the speech recognition model are provided with attention modules, the attention data can be obtained by an attention matrix obtained by an attention module of a selected layer. The selected layer can be any intermediate layer or the last layer of the multiple layers in the speech recognition model. The inventors have found through experiments that in a multi-layer network structure, the features extracted in the last layer of the network have better vocal characteristics than those of other layers. Therefore, in an example, when the speech recognition model adopts an encoder-decoder structure and the multiple decoding layers in the decoder are provided with attention modules, the selected layer can be the last decoding layer, that is, the attention data can be obtained by an attention matrix output by the attention module of the last decoding layer. Here, if the attention module of the selected layer adopts a single-head attention mechanism, the attention data can be an attention matrix obtained by the attention module, and if the attention module of the selected layer adopts a multi-head attention mechanism, the attention data can be a matrix obtained by performing an operation (for example, averaging, taking the mean square deviation, etc.) on multiple attention matrices obtained by the attention module.

[0127] In step S104, the human voice data can be obtained by the following steps: step a1, traversing the attention data according to the word dimension to extract the attention vector of the word dimension; step a2, obtaining the human voice vector of the word dimension according to the attention vector of the word dimension and a pre-configured threshold; and step a3, accumulating and summing the values in the human voice vector of the word dimension corresponding to each speech segment to obtain the human voice sequence of each speech segment, the human voice sequence including human voice information of each audio frame in the speech segment, the human voice information being used to indicate whether the audio frame belongs to human voice or not. In this way, the human voice data with the data granularity of the audio frame can be extracted without a high-complexity model such as a VAD model, which not only ensures the continuity and accuracy of speech recognition, but also avoids the misrecognition problem of the VAD model in a noisy scene and reduces the development and maintenance cost of the VAD model.

[0128] The threshold in step a2 can be a hyperparameter, and the specific value thereof can be determined through experiments. The specific implementation of step a2 and the threshold can be referred to the related description of the embodiments below, which will not be described here again. It should be noted that the specific implementation of step S104 is not limited to this, and any human voice data extraction method applicable to the embodiments of the present application can be used to implement step S104.

[0129] The inventors have found through analysis and experiments that fixed-window-length segmentation may lose speech endpoint information, and the speech recognition result at the segmentation position may be relatively poor. Therefore, the method of the embodiments of the present application solves these problems by aligning and correcting the text of the overlapping part of the speech segment in step S105.

[0130] In actual applications, step S105 can be implemented by various methods applicable to the embodiments of the present application. In some embodiments, an exemplary implementation process of step S105 can include the following steps performed for each pair of adjacent speech segments in the plurality of speech segments: step b1, extracting overlapping texts from the initial text segments of the adjacent speech segments, the overlapping texts including overlapping texts of the preceding speech segment and the following speech segment in the adjacent speech segments, the overlapping texts of the preceding speech segment corresponding to the tail part of the length of the overlap duration in the voice data thereof, and the overlapping texts of the following speech segment corresponding to the head part of the length of the overlap duration in the voice data thereof; step b2, aligning the overlapping texts of the preceding speech segment and the overlapping texts of the following speech segment to obtain aligned texts of the adjacent speech segments, the aligned texts including aligned texts of the preceding speech segment and aligned texts of the following speech segment; step b3, obtaining corrected texts of the adjacent speech segments according to the confidence of the word in the aligned texts of the preceding speech segment and the confidence of the word in the aligned texts of the following speech segment, the corrected texts of the preceding speech segment being the same as the corrected texts of the following speech segment; and step b4, obtaining corrected post-text segments of the preceding speech segment and the following speech segment in the adjacent speech segments using the corrected texts of the adjacent speech segments. In this way, the voice data can be used to assist text alignment, ensuring the integrity of long speech recognition and facilitating the improvement of the efficiency of manual verification of long speech recognition text. Moreover, by aligning and correcting the recognition results of the overlapping regions of the speech segments, the recognition accuracy of the division boundaries of the speech segment division is improved, and thus the overall recognition accuracy of the speech to be recognized is improved.

[0131] The confidence of the word can include, but is not limited to, the speech recognition confidence of the word, the language confidence of the word, and / or the position confidence of the word, etc. In actual applications, the specific content of the confidence of the word can be pre-configured according to different application scenarios, characteristics of the speech to be recognized, user requirements, or other various factors. In this way, the correction of the initial text segments is implemented by the confidence accurate to the word dimension in the embodiments of the present application, which can significantly improve the word accuracy of the recognition text of the speech to be recognized.

[0132] The speech recognition confidence refers to a confidence related to a speech recognition model, which is related to a structure of the speech recognition model. Taking the speech recognition model with the encoder-decoder structure as an example, the speech recognition confidence can include one or both of a frame alignment confidence and an attention confidence. The frame alignment confidence can be obtained by a frame alignment evaluation module integrated in the speech recognition model. The frame alignment evaluation module can be a CTC network, a predefined function, a pre-trained mapping relationship, or other various forms. Similarly, the attention confidence can be obtained by a recognized text evaluation module integrated in the speech recognition model. The recognized text evaluation module can be a neural network module, a predefined function, a pre-trained mapping relationship, or other various forms. The frame alignment evaluation module and the recognized text evaluation module can be trained together with the speech recognition model.

[0133] The language confidence refers to a confidence related to natural language logic. The language confidence can be obtained by performing semantic analysis on the initial text segment by a language analysis module or a language model integrated in the language recognition module. The language model or the language analysis module can be, but is not limited to, a natural language understanding (NLU) model or other models with natural language understanding capability.

[0134] The position confidence refers to a confidence related to a position of a word in the initial text segment. The position confidence can be determined based on a length of the initial text segment after alignment (i.e., the initial text segment obtained by replacing the overlapping text in the initial text segment with the aligned text) and the position of the word in the initial text segment. The closer the position of the word to the segmentation boundary of the initial text segment, the lower the position confidence score of the word. The closer the position of the word to the middle position of the initial text segment, the higher the position confidence score of the word. The specific calculation method of the position confidence can be referred to in the specific embodiments below. In the specific embodiments below, the position confidence is represented by a negative number, which can penalize the recognition result close to the segmentation boundary, thereby compensating for the poor speech recognition result at the segmentation boundary and effectively improving the overall recognition accuracy of the to-be-recognized speech recognition result.

[0135] The speech recognition confidence of the word can be obtained synchronously in the speech recognition in step S103. The language confidence of the word can be obtained after step S103 and before step b3. After the alignment processing in step b2, the length of the initial text segment is effectively corrected, and therefore, the position confidence of the word calculated after the alignment in step b2 is more accurate.

[0136] In some embodiments, the voice recognition method can further include: obtaining a confidence of the recognized text of the voice to be recognized according to the confidence of the words in the modified text segment. In this way, the confidence of the recognized text can be provided to the user at the same time as the recognized text of the voice to be recognized, so as to facilitate the user to generate the transcription text of the voice by referring to the confidence. Here, the confidence of the recognized text can include the confidence of each word in the recognized text. In practical applications, the recognized text and its confidence can be determined synchronously in step S106.

[0137] In some embodiments, the voice recognition method can further include: determining a confidence of the voice in each voice segment by using the attention data. Similarly, the confidence of the voice can be provided to the user or the device thereof at the same time as the voice data of the voice to be recognized, so as to facilitate the user or the device thereof to generate the transcription text of the voice by referring to the confidence of the voice and the voice data. In practical applications, the confidence of the voice can be obtained synchronously with the voice data in step S104.

[0138] It should be noted that the "word" in this paper can be a Chinese character, an English word or a single word in other languages.

[0139] Based on the above introduction to the exemplary overall flow of the voice recognition method provided by the embodiments of the present application, next, the exemplary specific implementation of the voice recognition method provided by the embodiments of the present application will be described in detail.

[0140] Referring to Figure 2 The specific implementation flow of the voice recognition method in the embodiment can include the following steps:

[0141] Step S201, receiving a target audio from a sound pickup device, and performing window length fixed overlap segmentation on the target audio to obtain a voice segment set [A1,..., AN] in which adjacent voice segments overlap each other;

[0142] Here, the target audio can be a short audio (10 seconds) or a long audio (the time length can be tens of minutes or even hours).

[0143] Figure 3 An exemplary implementation process of window length fixed overlap segmentation is shown. Referring to Figure 3 As shown in FIG. 2, the target audio can be segmented according to a fixed time length wl to obtain a voice segment set [A1,..., AN] including N voice segments, and the two continuous voice segments need to ensure the overlap of wo time length. Wherein, N is an integer, representing the total number of voice segments obtained by segmentation. Referring to Figure 3As shown, the time length wl of the speech segment can be 16 seconds, and the overlap length wo can be 6 seconds. In actual applications, the overlap length wo is less than or equal to half of the time length wl of the speech segment. The specific value of the overlap length can be determined according to actual needs, application scenarios, characteristics of the target audio, performance of the speech recognition model, and the like. For example, in a case where high processing efficiency is required and the recognition accuracy is not high, the overlap length wo can be set to one third to one fourth of the time length wl of the speech segment. In a case where high recognition accuracy is required, the overlap length wo can be set to one half to one third of the time length wl of the speech segment.

[0144] In step S202, the speech recognition model is used to perform speech recognition on the set of speech segments [A1,..., AN] to obtain initial text segments Ui (i = 1,..., N) and attention data Mia[L, T] (i = 1,..., N) of each speech segment.

[0145] In an implementation, speech recognition can be performed on all speech segments in the set of speech segments [A1,..., AN] in parallel to improve processing efficiency.

[0146] In this step, the speech segment Ai (i = 1,..., N) is input into the trained ASR model to obtain the initial text segment Ui and the speech confidence Uci of the initial text segment Ui, as well as the multi-head attention matrix Mi[L, T x Nh] obtained by the selected layers of the attention module (assuming the number of heads is Nh) in the decoder of the ASR model. The attention data Mia[L, T] of the speech segment Ai is obtained from the multi-head attention matrix Mi[L, T x Nh].

[0147] In step S203, the vocal data Si[T] of the speech segment Ai is extracted from the attention data Mia[L, T] (i = 1,..., N).

[0148] In this step, the vocal data of each speech segment can be extracted in parallel. That is, the vocal data Si[T] of the corresponding speech segment Ai is extracted from the attention data Mia[L, T] (i = 1,..., N).

[0149] In step S204, the initial text segment Ui is aligned and corrected according to the vocal data Si[T] and the overlap length wo to obtain the corrected text segment Pi.

[0150] In an implementation, the text UPi in the corresponding overlap part of the initial text segment Ui can be aligned and corrected using the vocal data Si[T] and the overlap length wo. In this way, the split-end recognition accuracy can be improved while the computational complexity and the amount of operation data are reduced as much as possible, so that the purpose of improving the split-end recognition accuracy can be achieved at a low hardware cost.

[0151] In one implementation, the human voice data Si[T] and the overlapping duration wo can be used to perform alignment correction on the entire initial text segment Ui.

[0152] Step S205 , obtaining a recognition result P of the target audio according to the corrected text segment Pi (i=1, . . . , N) obtained by alignment correction.

[0153] Here, the recognition result of the target audio may include the recognition text, text confidence, and voice interval of the target audio.

[0154] In one implementation, the recognized text P[T] of the target audio can be obtained by concatenating the obtained corrected text segments Pi (i=1, . . . , N).

[0155] In one implementation, the text confidence Pc of the target audio may be obtained by concatenating the word confidences or word comprehensive scores of the corrected text segment Pi.

[0156] In one implementation, the vocal interval S[T] of the target audio may be obtained by concatenating the vocal data Si[T] of each speech segment.

[0157] Assume that the initial text segments Ui and Ui+1 of the speech segments Ai and Ai+1 obtained in step S202 are as follows:

[0158] Ui:

[0159] Ui+1:

[0160] The recognized text P obtained in step S205 is:

[0161] In the above example, "When there are no special road intersections with streetlights" in Ui and "When there are no road intersections with road turns" in Ui+1 correspond to the ASR recognition results obtained by performing the overlapping parts of the two consecutive speech segments Ai and Ai+1. As can be seen, the overlapping parts are close to the split point and lack context and semantic information, resulting in low ASR recognition accuracy for the speech segments near the split point. This leads to errors such as misidentifying "turn" as "special road intersection" and "road intersection" as "land intersection."

[0162] The method provided in the embodiments of the present application corrects and aligns Ui and Ui+1 through step S204, so that the recognition error of the overlapping part of the two continuous speech segments Ai and Ai+1, i.e., the part close to the speech segment division, is efficiently and accurately corrected and aligned, the recognition error "special item" is successfully corrected to "turn", "land port" is successfully corrected to "intersection", and the missing word "of" between "street lamp" and "intersection" is supplemented. The obtained recognition text after the correction and alignment not only conforms to the context and overall semantics of the long speech composed of the speech Ai and Ai+1, but also is fluent and coherent, and conforms to the natural language logic of Chinese. It can be seen that the overall recognition accuracy of the long speech recognition text is significantly improved.

[0163] As can be seen from the above examples, the embodiments of the present application significantly improve the accuracy of the recognition text while realizing continuous speech recognition.

[0164] The embodiments of the present application will be described in detail below.

[0165] In the embodiments of the present application, the speech recognition model is an ASR model, and the ASR model is a neural network model with an encoder-decoder structure based on a multi-head attention mechanism.

[0166] Figure 4 An exemplary network structure of the ASR model in the embodiments of the present application is shown. Referring to FIG. 2, the ASR model includes an input layer, an encoder layer, an attention layer, a decoder layer, and an output layer. Figure 4As shown, the ASR model includes an encoder and a decoder. The encoder performs feature extraction on the data X = {x1,..., xT} of the target audio (T represents the number of frames of the target audio) to obtain hidden feature data H = {h1,..., hL} (L represents the total number of words in the vocabulary supported by the ASR), and the decoder performs feature extraction on the previously obtained text data Y = {SOS, y1,..., yU} (U represents the length of the text, i.e., the number of words (which can include punctuation marks) of the text Y) to obtain the current text data Y, and the features of the text data (not shown in the figure) and the hidden features H of the audio data are fused together through a source attention mechanism. Among them, the encoder can include Ne encoding layers (Encoder Layer), Ne is an integer not less than 1, and each encoding layer can be connected to a multi-head attention module (Multi-head Attention). In an example, the encoding layer can be implemented through a Transformer module including a multi-head attention model. The decoder includes Nd decoding layers (Decoder Layer), Nd is an integer not less than 1, and each decoding layer is connected to a multi-head attention module (Multi-head Attention), and a feed forward layer (Feed Forward) can be arranged before the multi-head attention module (Multi-head Attention). In an example, the encoding layer can be implemented through a Transformer module including a multi-head attention model. Considering the model processing efficiency, accuracy, and computational complexity, etc., generally, the decoder in the ASR can include 6 decoding layers, and the number of heads of each multi-head attention module can be 8.

[0167] Referring to Figure 4 As shown, in addition to the encoding layer, the ASR encoder can also include a convolution activation module (Conv+ReLU) with a 2-layer structure, a dense layer (Dense), a positional encoding module (Positional Embedding), and a feature normalization layer (LayerNorm). In addition to the decoding layer, the ASR decoder can also include a character encoding module (Character Embedding), a positional encoding module (Positional Embedding), a feature normalization layer (LayerNorm), and a normalization layer (softmax). It should be noted that, Figure 4 The structure shown is only an example. Those skilled in the art should understand that the specific structure of the ASR model can be freely configured according to actual application requirements, application scenarios, characteristics of the target audio, etc.

[0168] In this embodiment, the ASR model can be trained by a multi-objective loss function. The ASR model obtained by the multi-objective loss function not only has high recognition accuracy, but also has good frame alignment capability, and can obtain higher-precision vocal data.

[0169] Referring to Figure 4 As shown in the figure, the encoder can use a CTC loss function, the decoder can use a KL divergence loss function (CE loss), and the multi-objective loss function value (i.e., the hybrid loss function value) of the ASR model can be determined by the following formula (1), and the training of the ASR model can be completed by minimizing the hybrid loss function value:

[0170] loss = λ x CTC_loss + (1-λ) x KL_loss (1)

[0171] In formula (1), loss represents the value of the hybrid loss function, CTC_loss represents the value of the CTC loss function, KL_loss represents the value of the KL divergence function, and λ is a pre-configured weight, which can be a fixed value greater than 0 and less than 1, determined by experiment or taken as an empirical value. For example, λ can be taken as 0.7, 0.6 or other values.

[0172] In the above example, the CTC loss function has frame alignment capability. Thanks to the forced alignment feature of the CTC algorithm, training the encoder using the CTC loss function can make the hidden state H of the encoder have frame-by-frame alignment information, and further make the attention matrix output by the multi-head attention model in the decoder contain clearer vocal interval information, that is, the attention data obtained by the ASR model can contain clearer vocal information. At the same time, experiments have verified that training the ASR model using the multi-objective loss function can help improve the recognition accuracy of the ASR model.

[0173] Figure 5 A visualization diagram of attention data obtained by the ASR model is shown. Figure 5 The diagram shown is obtained by visualizing the attention data obtained by the ASR model, and is used to intuitively present that the attention data obtained by the ASR model contains clear vocal interval information. In the diagram, the horizontal axis represents the frame number of the audio frame, the vertical axis represents the word dimension number of the attention data, and the gray level of each point can be used to indicate the probability that the audio frame corresponding to the point belongs to the vocal, and the deeper the gray level of the point, the higher the probability that the audio frame corresponding to the point belongs to the vocal. Figure 5 In the diagram, the corresponding area of the non-vocal segment (i.e., the audio segment without vocal) has the lightest gray level (almost zero), indicating that the non-vocal segment basically does not contain audio frames belonging to the vocal, and the vocal segment (i.e., the speech segment containing the vocal) has a darker gray level, indicating that the vocal segment contains audio frames belonging to the vocal. Figure 5The corresponding area gray scale of the area between the non-vocal segments is clear, indicating that the audio frames belonging to the vocal segments and the audio frames not belonging to the vocal segments are clear, the area gray scale of the audio frames corresponding to the vocal segments in the vocal segments is the deepest, and the area gray scale of the audio frames corresponding to the non-vocal segments (for example, the pause and hesitation) in the vocal segments is relatively shallow, which is consistent with the actual situation of the vocal segments in the voice, that is, the attention data obtained based on the ASR model contains clear vocal segment information, which fully shows that the accurate vocal data can be extracted from the attention data.

[0174] In this embodiment, the effective vocal sequence S corresponding to the attention matrix M of each short voice segment obtained from the ASR model is extracted, that is, the vocal sequence. At the same time, the corresponding vocal confidence Sc can also be extracted.

[0175] Figure 6 An exemplary implementation process of vocal data extraction in this embodiment is shown. Referring to FIG. 6, the exemplary specific process of extracting the vocal sequence in this embodiment can include the following steps: Figure 6

[0176] In step S601, the Nh attention matrices Mi[L,T×Nh] obtained by the multi-head attention module of the last layer of the decoder of the ASR model (assuming the number of heads is Nh) are extracted, and the mean of these attention matrices is calculated to obtain the attention data Mia[L,T] (i=1,..., N).

[0177] The attention data Mia[L,T] represents the corresponding relationship between each word in the initial text segment Ui of the short voice and each audio segment in the short voice, and the two dimensions represent the feature information of the short voice in the word dimension and the time dimension. L represents the number of words contained in the initial text segment Ui obtained by the ASR model for recognizing the short voice, and T represents the number of audio frames of the voice segment Ai, which can be calculated according to the length of the short voice.

[0178] Through experiments, it is found that in a multi-layer network structure, the features extracted in the last layer of the network have better vocal characteristics than those in other layers, so in this step, the attention data is determined directly through the attention matrices Mi[L,T×Nh] of the multi-head attention module of the last layer of the multiple decoding layers.

[0179] In step S602, the vocal vector Mis[1,T] is extracted from the attention data Mia[L,T] to form the vocal matrix Mis[L,T].

[0180] ​Among them, the human voice matrix Mis[L, T] contains the human voice feature information of the speech segment Ai in the word dimension and frame dimension, L represents the number of words contained in the initial text segment Ui obtained by the ASR model to recognize the speech segment Ai, T represents the number of frames of the speech segment Ai, and each element Mis[1, t] in the human voice matrix Mis[L, T] indicates whether the audio frame t on the word dimension 1 belongs to the human voice. When its value is 0, it means it does not belong to the human voice, and when the value is not 0, it means it belongs to the human voice.

[0181] After repeated experiments and data analysis, the inventors found that when an audio frame belongs to a human voice, its value in the attention vector will be higher than a certain threshold. Therefore, in one implementation, the process of obtaining the human voice matrix Mis[L, T] may include: traversing the matrix Mia[L, T] by word dimension, extracting the attention vector Mia[1, T] corresponding to the word dimension l, calculating the human voice vector Mis[1, T] corresponding to each attention vector Mia[1, T] according to the following formula (2), and splicing these human voice vectors Mis[l, T] according to the word dimension to obtain the human voice matrix Mis[L, T].

[0182]

[0183] Among them, Thred a Represents a pre-set threshold, Mis[l, t] represents the value of audio frame t in the voice vector of word dimension l, and Mia[l, t] represents the value of audio frame t in the attention vector of word dimension l. a It is a hyperparameter, usually with values ​​of 0.09, 0.1, 0.16, etc. a It can be determined through experiments or empirical values.

[0184] Therefore, a human voice vector can be extracted from the attention data through a hyperparameter, and then a human voice sequence can be obtained. The computational complexity is low and the requirements for hardware performance are low, which is conducive to reducing hardware costs and improving processing efficiency.

[0185] In this step, the vector Mia[l, T] can be used to calculate the voice confidence. Specifically, the maximum value of Mia[1, T] can be selected in each word dimension and recorded as the effective voice confidence Sic[L]. The effective voice confidence Sic[L] is the voice confidence.

[0186] Step S603: traverse the vocal matrix Mis[L, T] according to the word dimension, accumulate and sum the vocal vectors therein to obtain the result Si[t], and regularize the accumulated and summed result Si[t] based on the following formula (3) to obtain the effective vocal sequence Si[T].

[0187]

[0188] wherein Si[T] can indicate whether each audio frame in the speech segment Ai belongs to human voice, if a certain element Si[t] in Si[T] takes a value greater than 0, it means that the corresponding audio frame t belongs to human voice, if Si[t] takes a value of 0, it means that the audio frame t does not belong to human voice.

[0189] In this embodiment, for the multi-layer attention matrix of the speech recognition model, the attention matrix of the last layer is selected, the mean of the multi-head attention matrix is calculated along the head dimension to obtain attention data, the human voice matrix is obtained by threshold judgment on each attention vector in the attention data in the word dimension, and the human voice confidence is obtained by maximum value calculation; the human voice matrix obtained after threshold judgment is accumulated and summed, and is normalized according to the size of the value obtained by the accumulation and summation, and finally the effective human voice sequence is obtained. Thus, the embodiment solves the problem of poor endpoint recognition under fixed-length segmentation, and improves the accuracy of endpoint recognition in a noisy environment by using the noise resistance of the speech recognition model.

[0190] The embodiment provides an exemplary implementation of initial text segment alignment correction.

[0191] Figure 7 An exemplary implementation process of the text correction of the embodiment is shown. Referring to Figure 7 As shown, the exemplary implementation process of the text correction of the embodiment can include the following steps:

[0192] Step S701, for each pair of adjacent speech segments Ai&Ai+1(i=1,..., N-1) in the speech segment set [A1,..., AN], the overlapping text UPi of each speech segment is extracted from its initial text segment Ui according to the effective human voice sequence Si[T] and the overlap duration wo.

[0193] Suppose the adjacent speech segments Ai and Ai+1 are identified by the ASR model to obtain their initial text segments Ui, Ui+1. Suppose the overlapping time length of the speech segment Ai and the speech segment Ai+1 is wo, the text corresponding to the tail time length wo of the human voice sequence Si[T] in the initial text segment Ui is extracted as the overlapping text UPi of the speech segment Ai. The text corresponding to the head time length wo of the human voice sequence Si+1[T] of the speech segment Ai+1 in the initial text segment Ui+1 is extracted as the overlapping text UPi+1 of the speech segment Ai+1.

[0194] For example, Ui and Ui+1 are shown in italics as follows:

[0195] Ui:

[0196] Ui+1:

[0197] The extracted overlapping texts UPi and UPi+1 are as follows:

[0198] UPi:

[0199] UPi+1:

[0200] In step S702, the dynamic programming algorithm is used to align the overlapping text UPi of the preceding speech segment Ai and the overlapping text UPi+1 of the following speech segment Ai+1 in the adjacent speech segments to obtain the aligned texts UPai and UPai+1 of the adjacent speech segments.

[0201] In an implementation, the dynamic programming algorithm such as the Fisher algorithm or the like can be used to length-align the overlapping texts of the adjacent speech segments, and the missing words are filled with a pre-set placeholder (for example, *) to obtain the aligned texts of each piece of short speech.

[0202] Still taking the adjacent speech segments Ai and Ai+1 as an example, the aligned texts UPai and UPai+1 as follows can be obtained from the overlapping texts UPi and UPi+1 of the adjacent speech segments Ai and Ai+1 through this step:

[0203] UPai:

[0204] UPai+1:

[0205] As can be seen, the overlapping text UPi contains 11 characters, and the overlapping text UPi+1 contains 12 characters. Therefore, the missing word in UPi is filled with a pre-set placeholder “*” to obtain the aligned texts UPai and UPai+1. In this way, the situation of missing or fewer words at the position corresponding to the segmentation of the speech segment in the recognized text can be effectively eliminated.

[0206] In other implementations, the overlapping texts of the adjacent speech segments can be matched based on the confidence of each word in the initial text segment obtained by the ASR model to obtain the aligned texts of the overlapping parts of the adjacent speech segments.

[0207] In step S703, the corrected texts UPxi of the adjacent speech segments Ai and Ai+1 are obtained according to the comprehensive scores of the words in the aligned text UPai of the preceding speech segment Ai and the comprehensive scores of the words in the aligned text UPai+1 of the following speech segment Ai+1, and the aligned text segments Pi and Pi+1 (i = 1,..., N-1) of the adjacent speech segments Ai and Ai+1 are further obtained from the corrected texts UPxi.

[0208] As described above, the confidence of each word can include a speech recognition confidence, a language confidence and a position confidence. Figure 4 As an example of the ASR model shown in the foregoing, the speech recognition confidence of each word can include a frame alignment confidence and an attention confidence, and details of the frame alignment confidence and the attention confidence can be referred to the foregoing description, and will not be described herein. In the embodiment, the position confidence is taken as a penalty term to determine the comprehensive score of each word in the aligned text UPai, UPai+1, respectively.

[0209] In an implementation manner, the comprehensive score of each word can be obtained by the following formula (4):

[0210] Jointscore = a x CTCscore + l x Attscore + h x LMscore + Posscore (4)

[0211] Wherein, Jointscore represents the comprehensive score of the word, CTCscore represents the frame alignment confidence of the word, Attscore represents the attention confidence of the word, LMscore represents the language confidence value of the word, Posscore represents the position confidence value of the word, a represents the weight of the frame alignment confidence of the word, l represents the weight of the attention confidence of the word, h represents the weight of the language confidence of the word, and l, h, a are all hyperparameters, and the values thereof can be determined by experiments or taken as empirical values in advance.

[0212] Wherein, Posscore can be calculated by the following formula (5):

[0213] Posscore = -b |l - L / 2| (5)

[0214] Wherein, Posscore represents the position confidence value of the word, l represents the position of the word in the aligned text segment, L represents the number of words contained in the aligned text segment, the aligned text segment is a text segment obtained by replacing the overlapping text in the initial text segment with the aligned text, and b represents a position weight set in advance, b is a hyperparameter, and the value of b can be determined by experiments or taken as an empirical value.

[0215] It is found by experiments that for the speech segment obtained by segmentation, the recognition result at the beginning and the end of the speech segment often shows poor recognition effect due to chaotic semantic information. In the embodiment, the position score of the word is introduced, and it can be seen from formula (5) that the position confidence value of the word closer to the beginning or the end of the initial text segment of the speech segment is smaller, and the position confidence value of the word closer to the middle position of the initial text segment of the speech segment is larger. Therefore, the position confidence of the word is introduced as a penalty term of the comprehensive score of the word, so as to correct the influence of poor recognition effect at the short speech endpoint on the overall recognition accuracy of the long speech.

[0216] In an implementation, step S703 can be performed according to the following formula (6), i.e., adjusting each word in the aligned text of the adjacent speech segment according to the following formula (6):

[0217]

[0218] wherein Uri[l] (i = 1,..., N-1) represents a word at position l in the revised text of the speech segment Ai, N is the number of speech segments, UPai[l] represents a word at position l in the aligned text of the speech segment Ai, UPai+1[l] represents a word at position l in the aligned text of the speech segment Ai+1, Jointscore(UPai[l]) represents the comprehensive score of the word at position 1 in the aligned text of the speech segment Ai, Jointscore(UPai+1[l]) represents the comprehensive score of the word at position 1 in the aligned text of the speech segment Ai+1, and * represents a placeholder.

[0219] Taking the speech segments Ai and Ai+1 above as an example, suppose the confidence of “lamp” in the aligned text UPai is represented as (-0.235, -0.525, -0.216, 0), i.e., the frame alignment confidence of “lamp” is -0.235, the attention confidence is -0.525, the language confidence is -0.216, and the position confidence is 0, and the confidence of “wait” in the aligned text UPai+1 is represented as (-0.287, -0.672, -0.228, -0.871). Suppose α, λ, η, and β are all 1, then the comprehensive score of “lamp” is (-0.235) + (-0.525) + (-0.216) + 0 = -0.976, and the comprehensive score of “wait” is (-0.287) + (-0.672) + (-0.228) + (-0.871) = -2.058. Comparing the comprehensive scores of the two, it is obvious that the comprehensive score of “lamp” is greater than that of “wait”, so “wait” in UPi+1 is revised to “lamp”, and the word at this position in UPxi is selected as “lamp”. Similarly, the words at each recognition difference position in the aligned text are revised, and thus the revised text UPri&i+1 of the adjacent speech segments Ai and Ai+1 is obtained. The revised text UPri of the speech segment Ai is the same as the revised text UPri+1 of the speech segment Ai+1, and both are UPri&i+1.

[0220] UPri&i+1:

[0221] After obtaining the revised text of each speech segment, the corresponding text in the initial text segment of the speech segment can be replaced with the revised text, and thus the revised text segment of the speech segment is obtained.

[0222] Taking the speech segments Ai and Ai+1 above as an example, the revised text segments Pi and Pi+1 thereof are respectively:

[0223] Pi:

[0224] Pi+1:

[0225] In this embodiment, a dynamic programming algorithm is used to align the initial text segments of adjacent speech segments. Each character in the aligned text is scored by integrating a speech recognition model, a language model, and the character's position in the text. The aligned text is then corrected based on the character's score. This significantly improves the accuracy of characters at the endpoints of speech segment segmentation, resolves the problem of poor speech endpoint recognition results in fixed-window long segmentation, improves recognition accuracy at speech segment boundaries, and ultimately improves the overall recognition accuracy of long speech.

[0226] Figure 8 FIG shows an exemplary structure of a speech recognition device 800 provided in an embodiment of the present application. Figure 8 As shown, the exemplary speech recognition device 800 of the embodiment of the present application may include

[0227] The speech acquisition unit 810 is configured to acquire the speech to be recognized;

[0228] a speech segmentation unit 820 configured to segment the speech to be recognized to obtain a plurality of speech segments, wherein the tail of the preceding speech segment overlaps with the head of the succeeding speech segment in adjacent speech segments of the plurality of speech segments;

[0229] The speech recognition unit 830 is configured to obtain attention data and an initial text segment of each of the plurality of speech segments using a speech recognition model based on an attention mechanism;

[0230] A human voice data extraction unit 840 is configured to extract human voice data from the attention data of each speech segment;

[0231] an alignment correction unit 850 configured to obtain a corrected text segment for each of the speech segments based on the initial text segment of each speech segment, the vocal data, and the overlap duration, wherein the text corresponding to the tail portion in the corrected text segment of the preceding speech segment and the text corresponding to the head portion in the corrected text segment of the succeeding speech segment among adjacent speech segments of the plurality of speech segments are identical;

[0232] The text concatenation unit 860 is configured to concatenate the corrected text segments of each of the multiple speech segments to obtain the recognized text of the speech to be recognized.

[0233] In some embodiments, the speech segmentation unit 820 is configured to segment the speech to be recognized according to a fixed window length and / or a fixed overlapping time length to obtain the multiple speech segments with equal time length and / or overlapping part length.

[0234] In some embodiments, the attention mechanism based speech recognition model is an encoder-decoder structure model, which comprises an encoder and a decoder, and an attention module is arranged in each decoding layer of the decoder, and the attention data is obtained by the attention matrix output by the attention module of the last decoding layer.

[0235] In some embodiments, the attention mechanism based speech recognition model is trained by a multi-target loss function, and the multi-target loss function comprises at least one loss function with frame alignment capability.

[0236] In some embodiments, the attention data has a word dimension and a frame dimension, the human voice data extraction unit 840 is configured to obtain the human voice data by the following steps: traversing the attention data according to the word dimension to extract an attention vector of the word dimension; obtaining a human voice vector of the word dimension according to the attention vector of the word dimension and a pre-configured threshold; and accumulating and summing the values in the human voice vector of the word dimension corresponding to each speech segment to obtain a human voice sequence of each speech segment, wherein the human voice sequence comprises human voice information of each audio frame in the speech segment, and the human voice information is used to indicate whether the audio frame belongs to human voice or not.

[0237] In some embodiments, the human voice vector of the word dimension is obtained by formula (2).

[0238] In some embodiments, the alignment correction unit 850 is configured to obtain the corrected text segment of each speech segment by the following steps:

[0239] For each pair of adjacent speech segments in the plurality of speech segments, the following steps are performed:

[0240] extracting overlapping texts from the initial text segments of the adjacent speech segments, wherein the overlapping texts comprise overlapping texts of the preceding speech segment and the following speech segment in the adjacent speech segments, the overlapping texts of the preceding speech segment correspond to the tail part of the human voice data with a length of the overlap duration, and the overlapping texts of the following speech segment correspond to the head part of the human voice data with a length of the overlap duration;

[0241] aligning the overlapping texts of the preceding speech segment and the following speech segment to obtain the aligned texts of the adjacent speech segments, wherein the aligned texts comprise aligned texts of the preceding speech segment and the following speech segment;

[0242] obtaining the corrected texts of the adjacent speech segments according to the confidence of the word in the aligned texts of the preceding speech segment and the confidence of the word in the aligned texts of the following speech segment, wherein the corrected texts of the preceding speech segment are the same as the corrected texts of the following speech segment;

[0243] The modified text of the adjacent speech segment is obtained using the modified text of the adjacent speech segment to obtain a modified post-text segment of the preceding speech segment and a modified post-text segment of the following speech segment.

[0244] In some embodiments, the confidence of a word comprises at least one of: a frame alignment confidence of the word, an attention confidence, a language confidence of the word, and a position confidence of the word.

[0245] In some embodiments, the alignment modification unit 850 is configured to obtain the modified text of the adjacent speech segment according to a joint score of a word in the alignment text of the preceding speech segment and a joint score of the word in the alignment text of the following speech segment, wherein the joint score of the word is determined by taking the position confidence of the word as a penalty term.

[0246] In some embodiments, the position confidence of the word is obtained by formula (5).

[0247] In some embodiments, the joint score of the word is obtained by formula (4).

[0248] In some embodiments, the alignment modification unit 850 is configured to obtain the modified text of the adjacent speech segment according to a joint score of a word in the alignment text of the preceding speech segment and a joint score of the word in the alignment text of the following speech segment, and specifically comprises:

[0249] Each word in the alignment text of the adjacent speech segment is adjusted by the following formula:

[0250]

[0251] Wherein, Uri[l] (i = 1,..., N-1) represents a word at position l in the modified text of speech segment Ai, N is the number of the speech segments, UPai[l] represents a word at position l in the alignment text of speech segment Ai, UPai+1[l] represents a word at position l in the alignment text of speech segment Ai+1, Jointscore(UPai[l]) represents a joint score of a word at position l in the alignment text of speech segment Ai, Jointscore(UPai+1[l]) represents a joint score of a word at position l in the alignment text of speech segment Ai+1, and * represents a placeholder.

[0252] In some embodiments, the text splicing unit 860 is further configured to obtain a confidence of the recognized text of the speech to be recognized according to the confidence of the word in the modified post-text segment.

[0253] In some embodiments, the vocal data extraction unit 840 is further configured to obtain a vocal confidence of each speech segment in the plurality of speech segments using the attention data.

[0254] The speech recognition apparatus 800 of the embodiments of the present application can be implemented by software, hardware, or a combination of both. In some examples, the speech recognition apparatus 800 can be implemented by the computing device 900 described below.

[0255] The conventional method records important information during a meeting by manual recording, for subsequent archiving and retrieval. Manual recording is time-consuming and laborious, and is prone to recording errors. In order to improve the efficiency of meeting recording, the speech recognition method of the embodiments of the present application can automatically convert the recording of the meeting into text, while outputting the confidence of the text and the vocal interval, to help the verification personnel quickly complete the transcription and verification of the meeting recording.

[0256] With the rapid development of streaming media, short videos, live broadcasts, movies, and TV series occupy a large part of people's entertainment time in daily life. Subtitles are an indispensable part of video resources. The traditional method adds subtitles to video resources by manual dictation.

[0257] For the current situation of rapid replacement of video resources in today's society, the manual dictation method is too inefficient, especially in the field of video live broadcast, which is impractical. Using the speech recognition method of the embodiments of the present application, subtitles of the audio stream of the video resource can be automatically and efficiently generated, and the subtitles can be automatically added to the video according to the vocal interval.

[0258] 1. The super-long audio is not divided depending on the VAD model, avoiding the misidentification problem of the VAD model in a noisy environment, and reducing the development and maintenance cost of the VAD model.

[0259] 2. The speech endpoint recognition is based on attention data, which can effectively improve the long speech recognition accuracy in a noisy environment, and use the noise resistance of the speech recognition engine to improve the accuracy of endpoint recognition in a noisy environment.

[0260] 3. The problems of low recognition accuracy, high deletion error, and model collapse in super-long speech recognition requirements are solved. The word accuracy rate is improved from 72% to 89%.

[0261] Figure 9 is a structural schematic diagram of a computing device 900 provided by the embodiments of the present application. The computing device 900 includes a processor 910 and a memory 920. In addition, it can also include a communication interface 930 and a bus 940.

[0262] It should be understood that Figure 9 The communication interface 930 in the computing device 900 shown can be used for communication with other devices.

[0263] The processor 910 can be connected with the memory 920. The memory 920 can be used to store the program code and data. Therefore, the memory 920 can be a storage unit inside the processor 910, can be an external storage unit independent of the processor 910, or can be a component including the storage unit inside the processor 910 and the external storage unit independent of the processor 910.

[0264] Optionally, the computing device 900 can further include a bus 940. The memory 920 and the communication interface 930 can be connected with the processor 910 through the bus 940. The bus 940 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 940 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 9 In the embodiment of the present application, only one line is used to represent the bus 940, but it does not mean that there is only one bus or only one type of bus.

[0265] It should be understood that, in the embodiment of the present application, the processor 910 can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. Alternatively, the processor 910 uses one or more integrated circuits to execute related programs to implement the technical solutions provided by the embodiment of the present application.

[0266] The memory 920 can include read-only memories and random access memories, and provide instructions and data for the processor 910. A part of the processor 910 can also include a non-volatile random access memory. For example, the processor 910 can also store device type information.

[0267] When the computing device 900 is running, the processor 910 executes the computer execution instructions in the memory 920 to perform the operation steps of the above method.

[0268] It should be understood that the computing device 900 according to the embodiment of the present application can correspond to the corresponding subject in executing the method according to each embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the computing device 900 are respectively for implementing the corresponding processes of each method of the present embodiment. For the sake of brevity, they will not be repeated here.

[0269] The computing device 900 in the embodiment of the present application can be, but is not limited to, a mobile phone, a laptop computer, a voice transcription device, or various other types. Of course, the computing device 900 in the embodiment of the present application can also be embodied as a server or other device.

[0270] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0271] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0272] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0273] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0274] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0275] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the related art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program code storage media.

[0276] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The program is executed by a processor to perform a speech recognition method. The method includes at least one of the schemes described in the above embodiments.

[0277] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0278] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus.

[0279] The computer readable media on which the program code can be carried can be any appropriate media including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0280] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0281] Note that, the above only is the preferred embodiment of the present application and the technology principle used. The person skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and the person skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and all belong to the protection scope of the present application.

Claims

1. A speech recognition method, characterized in that: The method comprises: Get the speech to be recognized; Segmenting the speech to be recognized to obtain a plurality of speech segments, wherein the tail of the preceding speech segment overlaps with the head of the succeeding speech segment in adjacent speech segments of the plurality of speech segments; Obtaining attention data and an initial text segment for each of the plurality of speech segments using a speech recognition model based on an attention mechanism, wherein the attention data is obtained by an attention matrix obtained by an attention module of a selected layer; Extracting the human voice data from the attention data of each speech segment; wherein the human voice data is data for indicating which audio frames in a speech segment belong to human voice and which audio frames do not belong to human voice; wherein the attention data has a word dimension and a frame dimension, and extracting the human voice data from the attention data of each speech segment specifically includes: traversing the attention data according to the word dimension to extract an attention vector of the word dimension, and obtaining a human voice vector of the word dimension according to the attention vector of the word dimension; obtaining a human voice sequence for each speech segment according to the human voice vector of the word dimension corresponding to each speech segment, the human voice sequence including human voice information of each audio frame in the speech segment, and the human voice information indicating whether the audio frame belongs to human voice or not; Obtaining a revised text segment for each of the speech segments based on the initial text segment of each speech segment, the vocal data, and the overlapping duration, wherein the text corresponding to the tail portion in the revised text segment of the preceding speech segment and the text corresponding to the head portion in the revised text segment of the succeeding speech segment in adjacent speech segments of the plurality of speech segments are the same; The corrected text segments of each speech segment in the multiple speech segments are spliced ​​together to obtain the recognized text of the speech to be recognized.

2. The speech recognition method according to claim 1, wherein: The segmenting of the speech to be recognized to obtain multiple speech segments specifically includes: segmenting the speech to be recognized according to a fixed window length and / or a fixed overlapping time length to obtain the multiple speech segments with equal time length and / or overlapping part length.

3. The speech recognition method according to claim 1 or 2, characterized in that: The speech recognition model based on the attention mechanism is a model of an encoder-decoder structure, which includes an encoder and a decoder. An attention module is provided in the multiple decoding layers of the decoder, and the attention data is obtained by the attention matrix output by the attention module of the last layer of the multiple decoding layers.

4. The speech recognition method according to claim 3, wherein: The speech recognition model based on the attention mechanism is obtained by training a multi-objective loss function, and the multi-objective loss function includes at least one loss function with frame alignment capability.

5. The speech recognition method according to claim 1, wherein: Obtaining a human voice vector of the word dimension according to the attention vector of the word dimension, comprising: obtaining a human voice vector of the word dimension according to the attention vector of the word dimension and a pre-configured threshold; The method of obtaining the vocal sequence of each speech segment based on the vocal vector of the word dimension corresponding to each speech segment includes: accumulating and summing the values ​​in the vocal vector of the word dimension corresponding to each speech segment to obtain the vocal sequence of each speech segment.

6. The speech recognition method according to claim 5, characterized in that The word-dimensional vocal vector is obtained by the following formula: ； in, represents the threshold value, represents the value of the audio frame t in the human voice vector of word dimension l, Represents the value of the attention vector of word dimension l for audio frame t.

7. The speech recognition method according to claim 1, wherein: The step of obtaining the corrected text segments of the speech segments according to the initial text segments of the speech segments, the human voice data, and the overlap length specifically includes: For each pair of adjacent speech segments in the plurality of speech segments, perform the following steps: Extracting overlapping text from initial text segments of adjacent speech segments, the overlapping text including the overlapping text of the preceding speech segment and the overlapping text of the following speech segment in the adjacent speech segments, the overlapping text of the preceding speech segment corresponding to the tail portion of the human voice data thereof having a length equal to the overlapping duration, and the overlapping text of the following speech segment corresponding to the head portion of the human voice data thereof having a length equal to the overlapping duration; Aligning the overlapping text of the preceding speech segment with the overlapping text of the following speech segment to obtain aligned text of the adjacent speech segments, wherein the aligned text includes the aligned text of the preceding speech segment and the aligned text of the following speech segment; Obtaining, based on the confidence levels of the characters in the aligned text of the preceding speech segment and the confidence levels of the characters in the aligned text of the following speech segment, a revised text of the adjacent speech segment, wherein the revised text of the preceding speech segment is identical to the revised text of the following speech segment; The corrected texts of the adjacent speech segments are used to obtain a corrected text segment of a preceding speech segment and a corrected text segment of a succeeding speech segment in the adjacent speech segments.

8. The speech recognition method according to claim 7, characterized in that: The confidence of the word includes at least one of the following: frame alignment confidence of the word, attention confidence of the word, language confidence of the word and position confidence of the word.

9. The speech recognition method according to claim 8, characterized in that The step of obtaining the corrected text of the adjacent speech segments according to the confidence levels of the characters in the aligned text of the preceding speech segment and the confidence levels of the characters in the aligned text of the succeeding speech segment specifically includes: According to the comprehensive scores of the characters in the aligned text of the preceding speech segment and the comprehensive scores of the characters in the aligned text of the following speech segment, a revised text of the adjacent speech segment is obtained; wherein the comprehensive scores of the characters are determined with the position confidence of the characters as a penalty item.

10. The speech recognition method according to claim 9, characterized in that: The position confidence of the word is calculated by the following formula: ； in, Indicates the confidence value of the word position, l indicates the position of the word in the aligned text segment, and L indicates the number of words contained in the aligned text segment. represents a preset position weight, and the aligned text segment is a text segment obtained by replacing the overlapping text in the initial text segment with the aligned text.

11. The speech recognition method according to claim 10, characterized in that: The comprehensive score of the word is calculated by the following formula: ； in, Indicates the comprehensive score of the word, represents the frame alignment confidence of the word, represents the attention confidence of the word, represents the language confidence value of the word, represents the position confidence value of the word, represents the weight of the frame alignment confidence of the word, represents the weight of the attention confidence of the word, A weight representing the language confidence of the word.

12. The speech recognition method according to claim 9, characterized in that: The step of obtaining the corrected text of the adjacent speech segments according to the comprehensive scores of the characters in the aligned text of the preceding speech segment and the comprehensive scores of the characters in the aligned text of the following speech segment specifically includes: Each word in the aligned text of adjacent speech segments is adjusted by the following formula: ； in, , i=1, ... N-1, represents the word at position l in the corrected text of the speech segment Ai, N is the number of the speech segments, represents the word at position l in the aligned text of speech segment Ai, represents the word at position l in the aligned text of speech segment Ai+1, represents the comprehensive score of the word at position l in the aligned text of speech segment Ai, represents the comprehensive score of the word in the aligned text position l of the speech segment Ai+1, Represents a placeholder.

13. The speech recognition method according to claim 1, 8, 10 or 11, characterized in that: The method further includes: obtaining the confidence of the recognized text of the speech to be recognized based on the confidence of the characters in the corrected text segment.

14. The speech recognition method according to claim 1, wherein: The method further includes: using the attention data to obtain the voice confidence of each of the multiple speech segments.

15. A computing device, characterized in that include: at least one processor; as well as At least one memory stores program instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 14.

16. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 14.

Images