Speech recognition method, device and equipment based on sentence end, medium and product

Abstract

The invention discloses a speech recognition method and device based on a sentence end, equipment, a medium and a product. The speech recognition method based on the sentence end comprises the following steps: acquiring speech stream data in real time; and inputting the acquired voice stream data into a trained stream type voice recognition model in real time, recognizing a character corresponding to a current frame of the input voice stream data by adopting the current stream type voice recognition model, determining a sentence end attribute corresponding to the character, and outputting a real-time voice recognition result according to the character and the sentence end attribute. According to the technical scheme, the streaming speech recognition model is adopted to recognize the current character in real time and determine whether the current character is the sentence end or not, and the streaming speech recognition model is obtained through training based on the training data containing the text content label and the sentence end label, so that the capabilities of recognizing the character in real time and accurately recognizing the sentence end are achieved; the semantic integrity of the recognized statement can be ensured, so that the accuracy of speech recognition is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, device, medium and product based on sentence ending. Background Technology

[0002] Voice recognition technology is widely used in smart homes, in-vehicle services, and artificial intelligence. For example, smart speakers can receive users' voice commands, such as "play the next track" or "stop playing," and then execute the corresponding operations. Clearly, accurately and completely recognizing users' voice commands is crucial for realizing these functions.

[0003] Existing speech recognition typically utilizes Voice Activity Detection (VAD) models, which determine the end of a voice command based on the duration of silence detected at the voice input device. For example, if no new voice signal appears within one second after a user's voice command is detected, the voice command is considered finished. Because this model is non-streaming, it is highly susceptible to noise, leading to premature or delayed judgments of command termination. For instance, the user may not have finished speaking the command, but the device has already executed an incorrect instruction; conversely, the user may have finished speaking the command, but the device fails to promptly confirm its completion or execute it, resulting in excessive delays. Therefore, real-time speech recognition and timely sentence termination are urgent challenges that need to be addressed. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for speech recognition based on sentence endings, so as to achieve timely and accurate recognition of sentence endings, thereby improving the accuracy of speech recognition.

[0005] In a first aspect, embodiments of this application provide a speech recognition method based on the end of a sentence, including:

[0006] Real-time acquisition of voice stream data;

[0007] The acquired speech stream data is input into a trained streaming speech recognition model in real time, so that the streaming speech recognition model can identify the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute.

[0008] The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

[0009] Secondly, embodiments of this application also provide a speech recognition device based on sentence endings, comprising:

[0010] The acquisition module is used to acquire voice stream data in real time;

[0011] The recognition module is used to input the acquired speech stream data into the trained streaming speech recognition model in real time, so as to use the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute.

[0012] The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

[0013] Thirdly, embodiments of this application provide an electronic device, including:

[0014] One or more processors;

[0015] Storage device for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the sentence-end-based speech recognition method as described in the first aspect.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sentence-end-based speech recognition method as described in the first aspect.

[0018] Fifthly, embodiments of this application also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the sentence-end-based speech recognition method as described in any of the above embodiments.

[0019] This application provides a method, apparatus, device, medium, and product for sentence-end-based speech recognition. The sentence-end-based speech recognition method includes: real-time acquisition of speech stream data; real-time input of the acquired speech stream data into a trained streaming speech recognition model, using the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determine the sentence-end attribute corresponding to the text, and output a real-time speech recognition result based on the text and the sentence-end attribute. The above technical solution uses a streaming speech recognition model to recognize the current text in real time and determine whether the current text is the end of a sentence. This streaming speech recognition model is trained based on training data containing text content tags and sentence-end tags, thus possessing the ability to recognize text in real time and determine whether the currently recognized text is the end of a sentence. When the last character of a sentence is recognized, the sentence-end can be determined in a timely manner, ending the recognition of the current sentence. This avoids situations where the sentence-end is mistakenly recognized while the user is issuing a voice command, thus preventing interruption of recognition, and also avoids situations where recognition continues after the user's voice command has ended because the sentence-end was not recognized. This ensures the semantic integrity of the recognized sentence, thereby improving the accuracy of speech recognition. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 A flowchart illustrating a sentence-end-based speech recognition method provided in this application embodiment;

[0022] Figure 2 A schematic diagram illustrating a text content tag and a sentence ending tag provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram illustrating a speech recognition method combining streaming and non-streaming speech recognition models, provided as an embodiment of this application;

[0024] Figure 4 This is a schematic diagram illustrating the interaction between a user terminal and an electronic device, provided as an embodiment of this application.

[0025] Figure 5 A schematic diagram of the structure of a speech recognition device based on the end of a sentence, provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0028] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0029] It should be noted that the concepts of "first" and "second" mentioned in the embodiments of this application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order of functions performed by these devices, modules, units or other objects or their interdependencies.

[0030] Furthermore, the embodiments and features described in this application may be combined with each other, unless otherwise specified.

[0031] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0032] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the relevant content of the solution.

[0033] Existing speech recognition usually uses a VAD model, which uses the detected silence duration as the basis for determining the end of a speech command. For example, if the first 2 seconds of the user's speech signal has speech consistency and no new speech signal appears within the next 1 second, it is determined that the speech command ends. This method does not consider the semantic integrity. Since the model is non-streaming, it is very susceptible to noise, resulting in premature or delayed judgment of the end of the speech command and recognition of incorrect speech commands. Taking a smart speaker as an example, for the speech command "***, play the next, the next", the existing speech recognition will repeatedly recognize "the next" in it, not only prolonging the time for speech command recognition but also causing waste of computing and processing resources. In fact, it can be judged as the end of the sentence after the first 'the next'; another example is that for the speech command "***, tell a story", if the silence duration between 'ge' and 'gu' is relatively long, the existing speech recognition may only recognize "***, tell a", thus unable to understand the correct semantics and perform the correct operation.

[0034] To solve the above problems, in the embodiments of the present application, a sentence-ending based speech recognition method is provided, which is a streaming speech recognition method. By using a streaming speech recognition model, while recognizing text in real time, the logic of sentence ending judgment is also added to correctly identify the sentence ending in a timely manner and ensure the semantic integrity of the recognition result.

[0035] It should be noted that for ease of understanding, in the embodiments of the present application, Chinese is used as an example to illustrate speech commands, sentences, and text, etc. In actual applications, other languages can be used. For example, for an English speech command, the text can refer to a single word.

[0036] Figure 1 FIG. is a flowchart of a sentence-ending based speech recognition method provided in the embodiments of the present application. This embodiment is applicable to the situation of real-time speech recognition. Specifically, the sentence-ending based speech recognition method can be executed by a sentence-ending based speech recognition device, which can be implemented in software and / or hardware and integrated in an electronic device. The electronic device can refer to a client device or a server device, including but not limited to vehicle-mounted computers, electronic control units (ECUs), in-vehicle intelligent terminals, smartphones, or cloud servers, etc.

[0037] As Figure 1 shown, the method specifically includes the following steps:

[0038] S110. Obtain speech stream data in real time.

[0039] In this embodiment, the electronic device can acquire voice stream data in real time. For example, when the user issues a voice command, the voice stream data of the voice command can be acquired in real time through the microphone on the electronic device. If the electronic device is a server device, the voice stream data can also be acquired in real time by the user device and transmitted to the server device in real time via wired or wireless means.

[0040] S120. The acquired speech stream data is input into the trained streaming speech recognition model in real time, so as to use the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute.

[0041] The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

[0042] The acquired audio stream data is input into a streaming speech recognition model in real time. This model can be an automatic speech recognition (ASR) model that supports real-time return of speech recognition results during the processing of the audio stream data. This means that the speech recognition results are returned simultaneously with the user's speech. This functionality can be used in scenarios such as live streaming captions, real-time meeting recording, and voice wake-up. For example, for the user's voice command "play next," the streaming speech recognition model can process the audio stream data in real time. If the user's voice is recognized as "play" in the first X1 frames (X1 is a positive integer), it can output "play." If the user's voice is recognized as "release" in the next X2 frames (X2 is a positive integer), it can output "release." This continues until the user's voice is recognized as "each," at which point it can output "each." It can also determine that a complete voice command has been recognized (i.e., the end of a sentence or a complete sentence), and thus execute the operation corresponding to "play next." It is understandable that the audio stream data input to the streaming speech recognition model typically consists of multiple frames, and one character usually corresponds to multiple frames.

[0043] Streaming speech recognition models can be neural network models, such as Transformer models, Gaussian Mixture Model-Hidden Markov Models (GMM-HMM), or Deep Neural Network-Hidden Markov Models (DNN-HMM). Streaming speech recognition models are trained on training data. The training data mainly consists of a large amount of sample speech stream data of sample sentences. Each character in each sample sentence has a corresponding text content label, and a sentence end label is added to the end of each sample sentence. After training the streaming speech recognition model using this training data, the streaming speech recognition model has the ability to recognize text in real time and recognize sentence ends in a timely manner. In actual inference, in addition to recognizing the text corresponding to the current frame, the streaming speech recognition model also performs sentence end judgment on the text corresponding to the current frame (i.e., determines the sentence end attribute of the text corresponding to the current frame). Specifically, it can classify the text corresponding to the current frame as sentence end (sentence end or not sentence end) based on the probability that it is sentence end. If the probability of it being sentence end is high enough, the classification result (i.e., sentence end attribute) is judged as sentence end, thus judging that a complete sentence has been recognized, and the speech recognition result is output based on this.

[0044] The speech recognition results can be output in real time. The speech recognition results can include the text recognized for the current frame. If the text recognized for the current frame is the end of a sentence, the speech recognition results can also include information to indicate the end of the sentence, such as a period, a command to start executing a speech command, and / or a command to stop recognizing the current sentence.

[0045] This application provides a sentence-end-based speech recognition method. It employs a streaming speech recognition model to recognize the current text in real time and determine whether the current text is the end of a sentence. This streaming speech recognition model is trained on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence-end tags corresponding to the sample speech stream data. Therefore, it has the ability to recognize text in real time and determine whether the currently recognized text is the end of a sentence. When the last character of a sentence is recognized, the sentence end can be determined in a timely manner, and the recognition of the current sentence can be terminated. This method can meet the actual needs of various scenarios and achieve the effect of jointly judging the sentence end by comprehensively considering factors such as semantic integrity and silence duration. It can ensure the semantic integrity of the recognized sentence, thereby improving the accuracy of speech recognition.

[0046] In one embodiment, the process by which a streaming speech recognition model identifies the text corresponding to the current frame of the input speech stream data, determines the sentence-end attribute corresponding to the text, and outputs real-time speech recognition results based on the sentence-end attribute includes:

[0047] For the current frame of the input speech stream data, the streaming speech recognition model determines the classification of the word candidates corresponding to the current frame and the scores on each classification;

[0048] The word candidate corresponding to the classification with the highest score is taken as the word corresponding to the current frame;

[0049] Determine whether the word corresponding to the current frame is the end of a sentence;

[0050] If it is not the end of a sentence, output the word corresponding to the current frame, take the next frame as the current frame, and return to perform the operations of determining the classification of the word candidates corresponding to the current frame and the scores on each classification, taking the word candidate corresponding to the classification with the highest score as the word corresponding to the current frame, and determining whether the word corresponding to the current frame is the end of a sentence;

[0051] If it is the end of a sentence, output the word corresponding to the current frame, and the recognition of the entire sentence corresponding to the current frame ends.

[0052] Specifically, the process of recognizing the word corresponding to the current frame can be implemented using the idea of the maximum path (which can also be understood as greedy decoding). That is, at each moment, for the current frame, all possible word recognition results (i.e., word candidates) are calculated. Each word candidate serves as a classification, and each classification corresponds to a score, which represents the probability that the corresponding classification is the correct word recognition result. Then, based on the probability distribution of each classification, the classification with the highest score is selected as the word finally recognized for the current frame. It should be noted that the scores (i.e., probabilities) of each classification may be related to the previous moment or the previous frame, so as to make more accurate predictions and recognitions in combination with the context and semantics.

[0053] Exemplarily, the word candidates corresponding to the t1 moment include "want", "play", and "stop", corresponding to three classifications, and the corresponding scores are S1, S2, and S3 respectively. Assuming that the score of S2 is the highest, the recognition result at the t1 moment is "play"; in addition, from a semantic perspective, "play" is usually not the end of a voice command. After training, the streaming speech recognition model has the ability to recognize the end of a sentence and can determine that the word corresponding to the current frame is not the end of a sentence, so as to continue processing the next frame of the input; the word candidates corresponding to the t2 moment include "put" and "listen", corresponding to two classifications, and the corresponding scores are S3 and S4 respectively. Considering the context semantic relevance with the t1 moment, usually the score of S3 is higher, so the recognition result at the t2 moment is "put"; in addition, the streaming speech recognition model will also determine whether the word corresponding to the current frame is the end of a sentence. If it is not the end of a sentence, continue to process the next frame of the input, otherwise judge that the recognition of the current sentence ends.

[0054] In some embodiments, the process of identifying the text corresponding to the current frame can also be implemented by means of Beam Search decoding or Connectionist Temporal Classification (CTC) decoding, etc.

[0055] Taking Beam Search decoding as an example, at each moment, for the current frame, calculate the classifications of all possible text candidates. According to the probability distribution of each classification, multiple classifications with the highest scores can be selected as the text recognized in the current frame. For example, the text candidates corresponding to the t1 moment include "want", "play", and "stop", corresponding to three classifications, and the corresponding scores are S1, S2, and S3 respectively. Assuming that the scores of S1 and S2 are relatively high, then the two classifications "want" and "play" can be retained at the t1 moment; the text candidates corresponding to the t2 moment include "play" and "listen", corresponding to two classifications, and the corresponding scores are S3 and S4 respectively. Assuming that the score of S3 is relatively high, then the recognition result at the t2 moment is "play". The two classifications retained at the t1 moment can be used to generate the sequences "want to play" and "play" respectively with the recognition result at the t2 moment, and so on until the recognition of the current sentence is completed.

[0056] CTC decoding can be understood as an optimization of greedy decoding. First, select a suitable decoding path (such as selecting the maximum path based on the highest score), and then obtain the final recognition result by merging continuously occurring Tokens (Tokens can be understood as basic units used to represent a word or phrase in text or speech) and removing virtual characters (blank). This embodiment will not elaborate on this.

[0057] On this basis, by using a streaming speech recognition model, the current text can be recognized in real time and accurately while considering semantics, and it can be determined in real time whether the current text is the end of a sentence, thereby improving the accuracy of speech recognition.

[0058] In one embodiment, the training process of the streaming speech recognition model includes:

[0059] Real-time input the sample speech stream data into the current streaming speech recognition model to use the current streaming speech recognition model to recognize the sample text corresponding to the current frame of the input sample speech stream data, determine the sample end-of-sentence attribute corresponding to the sample text, and output the real-time sample speech recognition result according to the sample text and the sample end-of-sentence attribute;

[0060] Calculate the value of the objective function according to the sample speech recognition result and the label corresponding to the sample speech stream data;

[0061] If the value of the objective function does not meet the training termination condition, the parameters of the current streaming speech recognition model are adjusted, and the process returns to the steps of inputting sample speech stream data into the current streaming speech recognition model in real time and calculating the value of the objective function, until the value of the objective function meets the training termination condition, thus obtaining the trained streaming speech recognition model.

[0062] The tags corresponding to the sample speech stream data include the text content tags corresponding to each frame of the sample speech stream data, as well as the sentence end tags of the sample sentences; each character in the sample sentences corresponds to at least one frame; the sentence end tags correspond to the frame following the last frame corresponding to the last character of the sample sentences.

[0063] In this embodiment, each sample statement has a corresponding text content (query) tag. Additionally, a sentence-end tag is added to each sample statement to obtain complete training data. Specifically, a sentence-end tag is added to the frame following the last frame (i.e., the right boundary) corresponding to the last character in each sample statement. The text content tags allow the streaming speech recognition model to learn the relationship between the features of the current frame and the corresponding text, thereby enabling it to recognize the text corresponding to the current frame and all frames corresponding to each character. The sentence-end tag allows the streaming speech recognition model to learn the relationship between the features of the current frame and the sentence-end attribute, thereby enabling it to identify whether the text corresponding to the current frame is the end of a sentence.

[0064] Figure 2 This is a schematic diagram illustrating a text content tag and a sentence ending tag provided in an embodiment of this application. For example... Figure 2 As shown, assuming the sample sentence is "abcdefg", each character represents a word in the user's voice command; the text content labels include a, b, c, d, e, f, g, and Sil, where Sil can indicate that the recognition result of the corresponding frame is blank; the sentence ending label is... <eos>The text is added to the frame following the last frame (the fourth frame of g) of the last text (g). A streaming speech recognition model is trained using training data with text content labels and sentence end labels, enabling it to recognize text in real time and determine whether it is the end of a sentence, thus improving the reliability of streaming speech recognition.

[0065] In each iteration of the training process, the sample text and sentence-end attributes recognized by the current streaming speech recognition model can be compared with the corresponding query labels and sentence-end labels. The process of calculating the value of the objective function can be understood as analyzing the gap or loss between the result recognized by the current streaming speech recognition model and the corresponding label. Based on this, it is determined whether to continue to the next iteration and to adjust the parameters of the current streaming speech recognition model so that the gap or loss can be minimized in the next iteration, until the training termination condition is met. The training termination condition may refer to the value of the objective function being within a preset range, the gap or loss being less than a preset threshold, the number of iterations reaching a specified number, or the value of the objective function changing little in multiple consecutive iterations. In this case, the parameters of the streaming speech recognition model have reached a relatively optimal state, and the streaming speech recognition model can meet the requirements of practical applications.

[0066] Based on the above, the objective function is calculated according to the difference between the sample speech recognition results and the text content labels and sentence end labels. The parameters of the streaming speech recognition model are adjusted accordingly so that the streaming speech recognition model can learn the correct results for each character and sentence end, and can learn the position of the character output, thereby improving accuracy and real-time performance.

[0067] In one embodiment, calculating the value of the objective function based on the sample speech recognition result and the labels corresponding to the sample speech stream data includes:

[0068] The objective function is calculated based on the maximum and minimum scores of the text in each category in the sample speech recognition results, as well as the penalty term.

[0069] The penalty includes the amount of time that the first recognition position is earlier or later than the second recognition position in the sample speech recognition result; the first recognition position is the recognition position of the text or the recognition position at the end of the sentence, and correspondingly, the second recognition position is the position of the text content label corresponding to the text at the first recognition position or the position of the sentence end label.

[0070] Specifically, the maximum and minimum scores (maximum and minimum tensors) of text in the sample speech recognition results across various categories can be used to measure the accuracy of the streaming speech recognition model in recognizing text. A large difference between the maximum and minimum scores indicates that the streaming speech recognition model can clearly and effectively classify the text correctly. The objective function can be optimized through maximization or minimization. That is, the objective function value can be set to be positively correlated with the maximum score, negatively correlated with the minimum score, and / or positively correlated with the difference between the maximum and minimum scores. In this case, a larger objective function value indicates a better training effect of the streaming speech recognition model. Conversely, the objective function value can be set to be negatively correlated with the maximum score, positively correlated with the minimum score, and / or negatively correlated with the difference between the maximum and minimum scores. In this case, a smaller objective function value indicates a better training effect of the streaming speech recognition model.

[0071] The penalty term is used to measure the accuracy of a streaming speech recognition model in recognizing the end of a sentence. If the model recognizes the end of the sentence earlier than the label (i.e., prematurely) or later than the label (i.e., not in time), the penalty term value increases, and the greater the time difference between recognizing and recognizing the end, the larger the penalty term value. Understandably, a smaller penalty term value indicates better training performance of the streaming speech recognition model. Based on this, we can set the objective function value to be positively correlated with the penalty term value, in which case a smaller objective function value indicates better training performance; or, we can set the objective function value to be negatively correlated with the penalty term value, in which case a larger objective function value indicates better training performance.

[0072] By setting a penalty term for incorrect sentence ending detection in the objective function, streaming speech recognition models can better learn the position of the sentence ending. Optionally, an early penalty term can be added if the sentence ending is detected early, and a late penalty term can be added if the sentence ending is detected late.

[0073] For example, the objective function without added penalty terms can refer to the optimization objective settings of CTC decoding, and can be expressed as:

[0074] Lctc = -log(p(y∣x)), where p(y∣x) represents the posterior probability of the streaming speech recognition model for the input training data x and the output vector y.

[0075] After adding the penalty term, the objective function can be modified to: Lc′tc=-log(p′(y∣x));

[0076] Wherein, softmax is the normalized exponential function, p′(y∣x)=softmax(l′(y∣x)); l′(y∣x) represents the derivative of l(y∣x), and l(y∣x) represents the posterior probability of the streaming speech recognition model for the input training data x and the output vector y.

[0077] For each character and the end of a sentence, the loss between the recognition position and the corresponding label can be calculated. Taking the loss between the recognition position and the label of the end of a sentence as an example, the objective function can be expressed as:

[0078] l(y <eos>∣x t)=(αearly*max(t <eos>-t,0)+αlate*max(t-t <eos>-t buffer,0)))*(max(l(y∣xt)-min(l(y∣xt)));

[0079] Where, l(y) <eos>|xt) represents the output vector y of the streaming speech recognition model for the training data x input at time t, where y is the sentence ending ( <eos>The posterior probability of ), max(l(y|xt) and min(l(y|xt)) represent the maximum and minimum scores of the streaming speech recognition model for each category of the training data x input at time t, respectively; αearly and αlate are the coefficients (or weights) controlling the size of the first and second penalty terms, respectively; t buffer represents t <eos>After that <eos>The output provides a grace period, that is, the output is within the t buffer. <eos>It will not increase the second penalty item.

[0080] Understandably, the coefficients corresponding to the first and second penalty items, as well as the grace period, can be set according to actual needs, or they can be left unset. That is, the coefficients corresponding to the first and second penalty items can both be 1, and the grace period can be 0.

[0081] It's important to note that the response speed and latency of speech recognition directly impact the execution time of subsequent operations or functions. In practical applications, user-side computing resources are relatively limited, unlike server-side resources which are sufficient for deploying large machine learning or neural network models. Furthermore, user-side processors often need to run multiple applications simultaneously, leading to frequent lag due to concurrent computation. Issues such as slow word output, significant discrepancies between the decoded word position and the user's spoken words, or simultaneous output of multiple words can also cause display problems, negatively affecting the user experience. Therefore, reducing speech recognition latency and improving its fluency are crucial objectives of speech recognition. Users require a faster response time, meaning they receive real-time recognition results and can quickly execute corresponding operations as soon as the speech ends.

[0082] The method in this application embodiment designs an objective function based on the maximum and minimum scores in each category, enabling the streaming speech recognition model to learn how to correctly classify and thus correctly recognize each character. On the other hand, the objective function is designed based on a penalty term, that is, by setting a penalty term for each decoded spike in the query that is earlier or later than the speech boundary token of the corresponding character (i.e., the last frame of the corresponding character), optimizing the interval between the spike and the speech boundary of the corresponding character, improving the accuracy of the spike, so that the streaming speech recognition model can recognize each character and the end of the sentence in a timely manner, without recognizing them prematurely or delayed. Thus, it can recognize and output each character in real time while the user issues a voice command. That is, for every character the user speaks, the streaming speech recognition model can recognize and output a character, thereby improving the character output speed and efficiency, effectively reducing the latency of speech recognition, and improving the fluency of speech recognition.

[0083] In one embodiment, the method further includes:

[0084] S130. When the streaming speech recognition model recognizes that the text corresponding to the current frame is the end of a sentence, the speech stream data of the current sentence is input into the non-streaming speech recognition model as a whole sentence, and the non-streaming speech recognition model outputs the recognition result of the whole sentence corresponding to the input speech stream data.

[0085] Specifically, in practical applications, based on real-time recognition of user voice commands using a streaming speech recognition model, a non-streaming speech recognition model can be triggered for re-recognition when the streaming speech recognition model reaches the end of a sentence. This involves inputting the entire speech stream data of the current complete sentence into the non-streaming speech recognition model, which then re-recognizes the entire sentence and outputs the recognition result. This dual recognition mode, combining real-time recognition by a streaming speech recognition model with sentence-by-sentence recognition by a non-streaming speech recognition model, satisfies the need for real-time recognition and output of results during the user's voice command process, while also ensuring the accuracy of text and semantic recognition by performing sentence-by-sentence recognition after reaching the end of the sentence.

[0086] Optionally, the recognition result for the whole sentence can be used to correct the recognition result output by the streaming speech recognition model in real time. For example, the user's screen displays the result of the streaming speech recognition model in real time. When the streaming speech recognition model recognizes and displays the last word, the non-streaming speech recognition model can output the recognition result of the whole sentence and refresh the user's screen. The recognition result of the whole sentence replaces the complete sentence output by the streaming speech recognition model when it recognizes the end of the sentence. This allows the user to not only conveniently see the real-time word output while issuing voice commands, but also to intuitively see the final more accurate recognition result of the whole sentence after each complete sentence is spoken.

[0087] Figure 3 This is a schematic diagram illustrating a speech recognition method combining streaming and non-streaming speech recognition models, provided as an embodiment of this application. Figure 3 As shown, beamforming refers to the extraction of features from user-issued voice commands. The voice stream data may be acquired simultaneously by multiple microphones, potentially containing directional information. The sentence-end judgment logic is introduced into the streaming decoding process, i.e., a sentence-end identifier is added to the end of the sample sentence. <eos>In the process of streaming speech recognition, the streaming speech recognition model can use greedy decoding to recognize text in real time, and even when it reaches the end of the sentence. <eos>When it is, it can trigger the re-identification of the non-streaming speech recognition model.

[0088] Specifically, the speech stream data is input into the streaming speech recognition model in real time. The streaming speech recognition model can include two parts: a pre-trained streaming encoder and greedy decoding. The streaming encoder can be used to perform streaming encoding on the input speech stream data to convert the speech signal into a digital code stream, and greedy decoding can use methods such as the maximum path to identify text and end-of-sentence attributes in real time. For example, for the speech command "Play the next one", when the corresponding frame of the speech signal of "Play" is input to the streaming encoder, the streaming encoder can process the extracted features to obtain the encoding result corresponding to "Play", and greedy decoding can obtain the recognition result of "Play" (obtained at time t1, that is, the recognition position or the word output position of "Play" corresponds to time t1); when the corresponding speech signal of "the next one" is input to the streaming encoder, the streaming encoder can process the extracted features to obtain the encoding result corresponding to "the next one", and greedy decoding can obtain the recognition result of "the next one" (obtained at time t2, that is, the recognition position or the word output position of "the next one" corresponds to time t2), and so on. After each word of the speech command is input, finally, at time t5, the recognition result of "Play the next one" can be obtained and when the streaming speech recognition model recognizes "one" as the end of the sentence ( <eos>When it corresponds to the next frame of the last frame of "ge", the voice stream data of the whole sentence can be input into the non-streaming speech recognition model for encoding and decoding in the form of the whole sentence (at one time), and the recognition result of the whole sentence can be output at one time at time t6. On this basis, re-recognition is performed using the non-streaming speech recognition model, and some homophones with different characters, homophonic words or synonyms, etc. can be further corrected, thereby improving the reliability of speech recognition.

[0089] In one embodiment, the voice stream data is uploaded in real time by the user terminal;

[0090] The method further includes:

[0091] S140. When the streaming speech recognition model recognizes that the text corresponding to the current frame is the end of the sentence, send an instruction to the user terminal to stop receiving the voice stream data.

[0092] Specifically, in this embodiment, the streaming speech recognition model can be deployed on the server side, and the streaming speech recognition model is trained using the online-annotated training data on the server side; the user terminal can refer to the user terminal, and the user terminal is responsible for collecting and uploading the voice stream data in real time, but does not need to deploy and run the streaming speech recognition model, thereby saving the storage space and computing resources of the user terminal.

[0093] Figure 4 This is a schematic diagram of the interaction between the user terminal and the electronic device provided by the embodiment of the present application. As Figure 4 shown, when the user terminal detects a voice signal, it can collect and upload the voice stream data in real time, and the server can send the speech recognition result to the user terminal in real time and display it to the user; when the streaming speech recognition model recognizes that the text corresponding to the current frame is the end of the sentence, the server can send an instruction to the user terminal to stop receiving the voice stream data, indicating that the recognition of the current sentence is over, and the collection of the voice stream data for the current sentence can be stopped until a new voice signal of a voice command is detected.

[0094] It should be noted that, in some embodiments, the streaming speech recognition model can also be deployed on the user terminal, that is, the user terminal locally collects the voice stream data in real time and performs recognition without interacting with the server. This situation is applicable when the user terminal has sufficient computing resources, storage space and good processing performance. <This application provides a sentence-end-based speech recognition method that introduces sentence-end judgment logic into the streaming decoding process. By adding sentence-end tags to construct a multi-task optimization objective, the streaming speech recognition model learns the sentence end based on speech energy and semantics. After training, the streaming speech recognition model can not only correctly recognize text, but also ensure timely recognition of each text and sentence end, reducing premature and delayed microphone muting, improving the accuracy and real-time performance of speech recognition, and achieving low-latency real-time word output.

[0096] Figure 5 This is a schematic diagram of a sentence-end-based speech recognition device provided in an embodiment of this application. The sentence-end-based speech recognition device provided in this embodiment includes:

[0097] The acquisition module 210 is used to acquire voice stream data in real time;

[0098] The recognition module 220 is used to input the acquired speech stream data into the trained streaming speech recognition model in real time, so as to use the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute.

[0099] The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

[0100] This device uses a streaming speech recognition model to recognize the current text in real time and determine whether the current text is the end of a sentence. The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data. Therefore, it has the ability to recognize text in real time and accurately identify sentence ends, which can ensure the semantic integrity of the recognized sentences and thus improve the accuracy of speech recognition.

[0101] Optionally, the process by which the streaming speech recognition model identifies the text corresponding to the current frame of the input speech stream data, determines the sentence-end attribute corresponding to the text, and outputs real-time speech recognition results based on the text and the sentence-end attribute includes:

[0102] For the current frame of the input speech stream data, the streaming speech recognition model determines the classification of the text candidates corresponding to the current frame and the score on each classification;

[0103] The text candidate corresponding to the category with the highest score is taken as the text corresponding to the current frame.

[0104] Determine whether the text corresponding to the current frame is the end of a sentence;

[0105] If it is not the end of a sentence, output the text corresponding to the current frame, take the next frame as the current frame, and return to perform the operations of determining the category of the text candidate corresponding to the current frame and the score in each category, taking the text candidate corresponding to the category with the highest score as the text corresponding to the current frame, and determining whether the text corresponding to the current frame is the end of a sentence.

[0106] If it is the end of a sentence, then the text corresponding to the current frame is output, and the recognition of the entire sentence corresponding to the current frame ends.

[0107] Optionally, the device also includes a training module for:

[0108] The sample speech stream data is input into the current streaming speech recognition model in real time, so that the current streaming speech recognition model can identify the sample text corresponding to the current frame of the input sample speech stream data, determine the sample sentence end attribute corresponding to the sample text, and output the real-time sample speech recognition result based on the sample text and the sample sentence end attribute.

[0109] The objective function is calculated based on the sample speech recognition results and the labels corresponding to the sample speech stream data.

[0110] If the value of the objective function does not meet the training termination condition, the parameters of the current streaming speech recognition model are adjusted, and the process returns to the steps of inputting sample speech stream data into the current streaming speech recognition model in real time and calculating the value of the objective function, until the value of the objective function meets the training termination condition, thus obtaining the trained streaming speech recognition model.

[0111] Optional,

[0112] The tags corresponding to the sample speech stream data include the text content tags corresponding to each frame of the sample speech stream data, and the sentence end tags of the sample sentences.

[0113] Each character in the sample statement corresponds to at least one frame; the sentence end tag corresponds to the frame following the last frame corresponding to the last character of the sample statement.

[0114] Optionally, calculating the value of the objective function based on the sample speech recognition result and the labels corresponding to the sample speech stream data includes:

[0115] The objective function is calculated based on the maximum and minimum scores of the text in each category in the sample speech recognition results, as well as the penalty term.

[0116] The penalty item includes: the amount of time that the first recognition position is earlier or later than the second recognition position in the sample speech recognition result; the first recognition position is the recognition position of the text or the recognition position at the end of the sentence, and correspondingly, the second recognition position is the position of the text content tag corresponding to the text at the first recognition position or the position of the sentence end tag.

[0117] Optionally, the voice stream data is uploaded in real time by the user terminal;

[0118] The device also includes:

[0119] The sending module is used to send a command to the user terminal to stop receiving voice stream data when the streaming speech recognition model recognizes that the text corresponding to the current frame is the end of a sentence.

[0120] The sentence-end-based speech recognition device provided in this application embodiment can be used to execute the sentence-end-based speech recognition method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0121] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, user equipment, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0122] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks and wireless networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above.

[0125] In some embodiments, the methods described above can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the methods of any of the embodiments described above by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device 10, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device 10. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0132] This application also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the sentence-end-based speech recognition method as described in any of the above embodiments.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0134] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.< / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos> < / eos>

Claims

1. A speech recognition method based on sentence ending, characterized in that, include: Real-time acquisition of voice stream data; The acquired speech stream data is input into a trained streaming speech recognition model in real time, so that the streaming speech recognition model can identify the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute. The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

2. The method according to claim 1, characterized in that, The process of using the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determining the sentence ending attribute corresponding to the text, and outputting the real-time speech recognition result of the text and the sentence ending attribute based on the text and the sentence ending attribute includes: For the current frame of the input speech stream data, the streaming speech recognition model is used to determine the classification of the text candidates corresponding to the current frame and the score on each classification. The text candidate corresponding to the category with the highest score is taken as the text corresponding to the current frame. Determine whether the text corresponding to the current frame is the end of a sentence; If it is not the end of a sentence, output the text corresponding to the current frame, take the next frame as the current frame, and return to perform the operations of determining the category of the text candidate corresponding to the current frame and the score in each category, taking the text candidate corresponding to the category with the highest score as the text corresponding to the current frame, and determining whether the text corresponding to the current frame is the end of a sentence. If it is the end of a sentence, then the text corresponding to the current frame is output, and the recognition of the entire sentence corresponding to the current frame ends.

3. The method according to claim 1, characterized in that, The training process of the streaming speech recognition model includes: The sample speech stream data is input into the current streaming speech recognition model in real time, so that the current streaming speech recognition model can identify the sample text corresponding to the current frame of the input sample speech stream data, determine the sample sentence end attribute corresponding to the sample text, and output the real-time sample speech recognition result based on the sample text and the sample sentence end attribute. The objective function is calculated based on the sample speech recognition results and the labels corresponding to the sample speech stream data. If the value of the objective function does not meet the training termination condition, the parameters of the current streaming speech recognition model are adjusted, and the process returns to the steps of inputting sample speech stream data into the current streaming speech recognition model in real time and calculating the value of the objective function, until the value of the objective function meets the training termination condition, and a trained streaming speech recognition model is obtained. The tags corresponding to the sample speech stream data include the text content tags corresponding to each frame of the sample speech stream data, and the sentence end tags of the sample sentences; each character in the sample sentences corresponds to at least one frame; the sentence end tags correspond to the frame following the last frame corresponding to the last character of the sample sentences.

4. The method according to claim 3, characterized in that, The step of calculating the value of the objective function based on the sample speech recognition result and the labels corresponding to the sample speech stream data includes: The objective function is calculated based on the maximum and minimum scores of the text in each category in the sample speech recognition results, as well as the penalty term. The penalty item includes: the amount of time that the first recognition position is earlier or later than the second recognition position in the sample speech recognition result; the first recognition position is the recognition position of the text or the recognition position at the end of the sentence, and correspondingly, the second recognition position is the position of the text content tag corresponding to the text at the first recognition position or the position of the sentence end tag.

5. The method according to claim 1, characterized in that, Also includes: When the streaming speech recognition model recognizes that the text corresponding to the current frame is the end of a sentence, the speech stream data of the current sentence is input to the non-streaming speech recognition model as a whole sentence, and the non-streaming speech recognition model outputs the recognition result of the whole sentence corresponding to the input speech stream data.

6. The method according to any one of claims 1-5, characterized in that, The voice stream data is uploaded in real time by the user terminal; The method further includes: When the streaming speech recognition model identifies the text corresponding to the current frame as the end of a sentence, it sends a command to the user terminal to stop receiving audio stream data.

7. A speech recognition device based on sentence ending, characterized in that, include: The acquisition module is used to acquire voice stream data in real time; The recognition module is used to input the acquired speech stream data into the trained streaming speech recognition model in real time, so as to use the streaming speech recognition model to recognize the text corresponding to the current frame of the input speech stream data, determine the sentence end attribute corresponding to the text, and output the real-time speech recognition result based on the text and the sentence end attribute. The streaming speech recognition model is trained based on training data, which includes sample speech stream data of sample sentences, as well as text content tags and sentence end tags corresponding to the sample speech stream data; the sentence end attribute is either sentence end or not sentence end.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the sentence-end-based speech recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the sentence-end-based speech recognition method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the sentence-end-based speech recognition method as described in any one of claims 1-6.

Images