Audio processing method for achieving streaming TTS

Abstract

The present invention relates to a method for realizing streaming TTS (Text-to-Speech) speech text. The method includes the steps of: a user starting to speak and a voice wake-up of the user terminal; the user terminal sending streaming voice packets to an identification engine for voice identification; the identification engine continuously responding to the user terminal with uncertain identification text; the user terminal detecting and identifying the uncertain identification text and obtaining definitive identification text in advance; and the user terminal sending all of the pre-determined definitive identification texts to a translation engine for translation and, after completing speech synthesis, playing back the synthesized speech. The method of the present invention reduces the waiting time for the user to hear the translated speech and achieves the effect of artificial simultaneous interpretation by partially determining the identification result in advance and translating and synthesizing the determined identification result into a stream.

Description

Technical Field

[0001] The present invention relates to the field of speech translation technology, and specifically to a speech processing method for realizing streaming TTS.

Background Art

[0002] With the progress of internationalization, communication between different countries is very frequent, but the inability to communicate in the same language has become an obstacle to people's communication. By acquiring a translator, smooth communication can be realized, but this method is costly and lacks convenience. Currently, many of the translation equipment and software on the market perform translation after the speaker has completed a sentence and provide feedback after the translation result has been heard by the other party. Such a communication method that waits for translation is very inefficient and affects work efficiency.

[0003] Also, some so-called "simultaneous interpretation devices" first continuously send voice packets to a recognition and translation engine, and then the recognition and translation engine responds to the simultaneous interpretation device with unstable recognition results. Only when the speaker's speech stops (i.e., voice stops) for a certain period of time, the recognition and translation engine recognizes the judgment result for the previously received voice packets, and then translates the determined recognition result and synthesizes it into voice. Such a "simultaneous interpretation" device can achieve consistent recognition and feedback of translation results. However, when the user speaks continuously, due to the continuous variation of the recognized content, when the user tries to listen to the translated voice output, the voice cannot be converted from the completely recognized text by synthesizing it until waiting for the result finally determined by the recognition engine, and the effect like artificial simultaneous interpretation cannot be achieved. That is, before the user finishes speaking, the other party cannot hear the translated voice. As a result, the waiting time for the user to hear the translated voice becomes longer, and this problem becomes more prominent especially when the user expresses continuously in a long text.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an audio processing method that realizes streaming TTS applied in the translation field. This method partially determines the identification results in advance, translates the determined identification results into a stream and synthesizes speech, reduces the waiting time when listening to the translated speech, and achieves the effect of artificial simultaneous interpretation. [Means for solving the problem]

[0005] To achieve the above objective, the present invention provides an audio processing method for realizing streaming TTS, and the method is The process includes the steps of: the user starting to speak and the user terminal performing a voice wake-up; the user terminal sending a voice recognition request to the identification engine; if the voice recognition request is successful, the user terminal sending a streaming voice packet to the identification engine to perform voice recognition; and the identification engine continuously responding to the user terminal with indeterminate identification text. When the user terminal performs detection and identification of the uncertain identification text, specifically, the user terminal performs character detection on the uncertain identification text that the identification engine has responded to each time, and when it is detected that the number of characters is n or more for the first time, N takes the value (20, 50), and the uncertain identification text at this time is denoted as B1, and the user terminal triggers the first punctuation break detection on the uncertain identification text B1, and thereafter, in the latest responding uncertain identification text, a punctuation break detection is triggered each time n characters are added based on the length of the uncertain identification text B1. Each time punctuation break detection is successfully triggered, the user terminal performs one punctuation break detection operation. If the user terminal's punctuation break detection meets the punctuation break condition, it is determined that the break was successful, and the uncertain identified text prior to the punctuation break is recognized as the definitive identified text. Each time an identification text is determined, the user terminal sends the determined identification text to the translation engine for translation and completion of speech synthesis, and finally the user terminal plays the synthesized speech.

[0006] According to one embodiment of the present invention, each time punctuation break detection is successfully triggered, first, it is determined whether the most recent indeterminate identified text received from the identification engine contains one or more definitive identified texts obtained through previous break detection successes. If the latest uncertain identified text does not contain any definitive identified text obtained through successful fragmentation so far, then it is directly determined whether the latest uncertain identified text meets the punctuation fragmentation condition. If the latest uncertain identified text contains one or more definitive identified texts obtained through successful fragmentation so far, first, a similarity comparison is performed between the latest uncertain identified text and the one or more definitive identified texts obtained through successful fragmentation so far. The one or more definitive identified texts obtained so far are ignored, the remaining uncertain identified text is cut out, and punctuation fragmentation detection is performed on the cut-out uncertain identified text to determine whether the cut-out uncertain identified text meets the punctuation fragmentation condition.

[0007] According to one embodiment of the present invention, whether the punctuation break condition has been met is determined by performing detection on an uncertain identification text that is to be subjected to punctuation break detection, and if the uncertain identification text that is to be subjected to punctuation break detection contains a first class punctuation mark for break detection, then it is determined that the break has been successful.

[0008] According to one embodiment of the present invention, if the uncertain identification text to be detected for punctuation breaks contains one first-class punctuation mark for breaks, then that first-class punctuation mark for breaks becomes the punctuation break point, and the portion of the uncertain identification text prior to this punctuation break point is determined as the definitive identification text.

[0009] According to one embodiment of the present invention, if the uncertain identification text to be detected for punctuation breaks contains two or more first-class punctuation symbols for breaks, the last first-class punctuation symbol for breaks is designated as the punctuation break point at that time, and the portion of the uncertain identification text prior to this punctuation break point is determined as the definitive identification text at that time.

[0010] According to one embodiment of the present invention, if the uncertain identification text to be detected for punctuation breaks does not contain a first class of punctuation marks for breaks, it is detected whether it contains a second class of punctuation marks for breaks. If the uncertain identification text that is to be detected as a punctuation break contains two or more Class 2 punctuation marks for breaks, the break is determined to be successful, and the last Class 2 punctuation mark for breaks is designated as the punctuation break point at that time, and the portion of the uncertain identification text preceding this punctuation mark is determined to be the definitive identification text at that time; otherwise, the break is determined to be unsuccessful.

[0011] According to one embodiment of the present invention, the first class of punctuation marks for breaks includes periods, exclamation marks, question marks, and semicolons.

[0012] According to one embodiment of the present invention, the second class of punctuation marks for breaks includes a comma.

[0013] According to one embodiment of the present invention, the definitively identified text obtained by punctuation-breaking sentences is transmitted to the translation engine each time for translation, and the translated text is synthesized into speech. If the length of the text string that has already been synthesized but has not yet been played back by the user terminal is 40 or more, the user terminal adjusts the playback speed of that text to 1.5 times the default value. If the length of a text string that has already been synthesized but has not yet been played back by the user terminal is less than 40 characters, the user terminal adjusts the playback speed to the default value.

[0014] Another aspect of the present invention provides an audio processing method for realizing streaming TTS, comprising the steps of: the user starting to speak and performing an audio wake-up on the user terminal; the user terminal sending a speech recognition request to an identification engine; and, upon success of the speech recognition request, the user terminal sending a streaming audio packet to the identification engine to perform speech recognition; and the identification engine continuing to respond to the user terminal with uncertain identification text. The user terminal performs detection and identification of the uncertain identification text by, specifically, continuously receiving the uncertain identification text that has been responded to multiple times by the identification engine, and then the user terminal makes a judgment by comprehensively comparing it with the most recently responded N uncertain identification texts, where N takes a value in the range of (2,6). If the identification content of the preceding M clauses remains unchanged for all N uncertain identification texts, the range of M is (1, 5), and in this case, the identification content of the preceding M clauses is recognized as the current definitive identification text. The user terminal transmits the definitive identification text determined each time to the identification engine for translation and completion of speech synthesis, and finally plays the synthesized speech on the user terminal.

[0015] According to one embodiment of the present invention, if the most recently responded N uncertain identification texts by the identification engine each contain one or more previously determined identification texts, the user terminal compares the most recently responded N uncertain identification texts with the one or more previously determined identification texts for similarity, ignores the one or more previously determined identification texts, cuts out the remaining portion of the N uncertain identification texts, and then makes a judgment by comprehensively comparing the remaining portion of the N uncertain identification texts. If the identification content of the preceding M clauses for the remaining parts of the N uncertain identification texts remains unchanged, then the identification content of the preceding M clauses is recognized as the current definitive identification text.

[0016] According to one embodiment of the present invention, the user terminal transmits the identification text determined each time to the translation engine for translation, and performs speech synthesis on the translated text. If speech synthesis has already been completed but the length of the text string that has not been played back via the user terminal is 40 or more, the user terminal adjusts the speech playback speed to 1.5 times the default value. If speech synthesis has already been completed but the length of the text string that has not been played back via the user terminal is less than 40 characters, the user terminal adjusts the speech playback speed to the default value. [Effects of the Invention]

[0017] The present invention offers the following beneficial effects compared to the prior art. (1) The present invention performs punctuation detection on uncertain identification texts that the identification engine has recently responded to by the user terminal, determines some definite identification results in advance, and transmits the already determined identification results to the translation engine in a stream for translation and speech synthesis. In such a method, before the identification engine responds with definite identification text or final identification text to the user terminal, the user terminal can obtain or determine the identification text in advance. Compared with the conventional method, the waiting time until the user listens to the translated voice is reduced, achieving the effect of artificial simultaneous interpretation. (2) The present invention also comprehensively compares and judges a plurality of uncertain identification texts that the identification engine has recently responded to by the user terminal, collates some definite identification results in advance, and transmits the already determined identification results to the translation engine in a stream for translation and speech synthesis, thereby similarly reducing the waiting time until the user listens to the translated voice and achieving the effect of artificial simultaneous interpretation. (3) The present invention determines the length of the text string of the text for which speech synthesis has been completed but the voice playback by the user terminal has not been performed, dynamically adjusts the playback speed of the TTS voice text, and reduces the time required for the playback of the translated voice.

Brief Description of the Drawings

[0018] [Figure 1] It is a flowchart of a voice processing method for realizing streaming TTS according to the present invention. [Figure 2] It is a schematic diagram of an embodiment of the present invention. [Figure 3] It is another flowchart of a voice processing method for realizing streaming TTS according to the present invention.

Modes for Carrying Out the Invention

[0019] Hereinafter, based on the embodiments and the drawings, the present invention will be described in more detail, but the embodiments of the present invention are not limited thereto.

[0020] As shown in Figure 1, the audio processing method for realizing streaming TTS according to the present invention is specifically as follows: The user begins speaking and performs an audio wake-up on the user terminal. Once the user terminal is woken up, the user terminal sends an audio identification request to the identification engine. If the audio identification request is successful, the user terminal sends a streaming audio packet to the identification engine to perform audio identification. The identification engine continuously responds to the user terminal with an uncertain identification text based on the streaming audio packet. The uncertain identification text in the embodiment of the present invention is the intermediate state identification result actually sent by the identification engine. In the prior art, the intermediate state identification result is not considered the final result sent by the identification engine. That is, this final result is not considered a specific result sent by the identification engine, and the identification engine only sends the final identification result to the user terminal when it detects that the user's utterance has reached a certain period of audio silence. At that time, the identification result that the identification engine sends to the user terminal is the determined identification result. Furthermore, in the prior art, the uncertain identification text that the identification engine has responded to the user terminal was often divided into different short sentences by punctuation marks. These punctuation marks include periods, question marks, commas, exclamation marks, and semicolons.

[0021] When the translation engine responds to the user terminal with uncertain identification text, the result cannot be sent to the translation engine for TTS synthesis, resulting in a long waiting time for the user to finally receive the translated and synthesized speech. In particular, when the user speaks continuously, communication efficiency is low. Therefore, the improvement of the present invention is that the user terminal performs detection and identification on the uncertain identification text responded by the identification engine before the user terminal receives the final result sent from the identification engine. Specifically, the user terminal performs character detection on the uncertain identification text responded by the identification engine each time. If it is detected that the number of characters is always less than n, the user terminal does not trigger punctuation break detection, and N takes a value of (20, 50) until it is detected that the number of characters is n or greater for the first time. In the embodiment of the present invention, the magnitude of the value of n is determined based on the actual situation. Preferably, the value of n is 30. For convenience of explanation, the uncertain identification text at this time is denoted as B1. The user terminal triggers the first punctuation break detection for the uncertain identified text B1. Subsequently, for the most recently responded uncertain identified text, one punctuation break detection is triggered each time n characters are added, based on the length of the uncertain identified text B1. Each time a punctuation break detection is successfully triggered, the user terminal performs one punctuation break detection operation. If the user terminal's punctuation break detection meets the punctuation break condition, it is determined that the break was successful, and the uncertain identified text prior to the current punctuation break is recognized as the current definitive identified text.

[0022] Each time an identification text is determined, the user terminal sends the determined identification text to the translation engine for translation and completion of speech synthesis, and finally the user terminal plays the synthesized speech.

[0023] In embodiments of the present invention, each time punctuation break detection is successfully triggered, it is determined whether the latest indeterminate identified text responded by the identification engine contains one or more definitive identified texts obtained through successful break detections so far. If the latest indeterminate identified text does not contain any definitive identified texts obtained through successful break detections so far, it is directly determined whether the latest indeterminate identified text meets the punctuation break condition. If the latest indeterminate identified text contains one or more definitive identified texts obtained through successful break detections so far, first a similarity comparison is performed between the latest indeterminate identified text and the one or more definitive identified texts obtained through successful break detections so far. The one or more definitive identified texts obtained so far are ignored, the remaining indeterminate identified text is cut off, punctuation break detection is performed on the cut-off indeterminate identified text, and it is determined whether the cut-off indeterminate identified text meets the punctuation break condition.

[0024] In the embodiments of the present invention, whether the punctuation break condition has been met is determined by performing detection on the uncertain identified text that is to be subjected to punctuation break detection. Specifically, if the uncertain identified text that is to be subjected to punctuation break detection contains a first-class punctuation mark, it is determined that the break has been successful. In the embodiments of the present invention, there may be one or more successful breaks containing the first-class punctuation mark, but in all cases, the break is determined to have been successful. If the uncertain identified text that is to be subjected to punctuation break detection contains one first-class punctuation mark, that first-class punctuation mark becomes the punctuation break point, and the portion of the uncertain identified text prior to this punctuation break point is determined as the definitive identified text. In the uncertain identification text to be detected for punctuation breaks, if it contains two or more Class 1 punctuation marks for breaks, the last Class 1 punctuation mark is designated as the punctuation break point, and the portion of the uncertain identification text prior to this punctuation break point is determined as the definitive identification text. In the embodiment of the present invention, if the uncertain identification text to be detected for punctuation breaks does not contain any Class 1 punctuation marks for breaks, it is detected whether it contains Class 2 punctuation marks for breaks. If the uncertain identification text to be detected for punctuation breaks contains two or more Class 2 punctuation marks for breaks, it is determined that the break was successful, and the last Class 2 punctuation mark is designated as the punctuation break point, and the portion of the uncertain identification text prior to this punctuation mark is determined as the definitive identification text. Otherwise, it is determined that the break was unsuccessful. In other words, if the uncertain identification text that is to be subjected to punctuation fragment detection does not contain a first-class fragmentation punctuation mark, nor does it contain two or more second-class fragmentation punctuation marks, then the fragmentation is determined to be a failure. If the fragmentation fails, no new definitive identification text is generated.

[0025] In embodiments of the present invention, the first class of punctuation marks for breaks includes periods, exclamation marks, question marks, and semicolons. The second class of punctuation marks for breaks includes commas.

[0026] In embodiments of the present invention, definitively identifiable text obtained from punctuated sentences is sent to a translation engine for translation, and the translated text is then synthesized into speech. To prevent the user from having to wait too long for speech translation due to an excessive accumulation of already translated and synthesized TTS speech, the method of the present invention further adjusts the playback speed of the TTS speech by the user terminal. Specifically, if the length of a text string that has already been synthesized into speech but has not yet been played back by the user terminal is 40 or more, the user terminal adjusts the playback speed to 1.5 times the default value. If the length of a text string that has already been synthesized into speech but has not yet been played back by the user terminal is less than 40, the user terminal adjusts the playback speed to the default value. This default value is determined based on the normal speaking speed.

[0027] To explain the spirit of the present invention in more detail, Figure 2 is an example in which ""isLast":false, "rText":" is an uncertain identification text representation, and the Arabic numerals "123456789" (including punctuation marks following it) are represented as the identification text. In Figure 2, each line is an uncertain identification result in response from the identification engine to the user terminal. After the user terminal receives the uncertain identification text, it first performs character count detection on the uncertain identification text. This will be explained using the example where n is 20. When the user terminal performs character count detection on the characters in the previous four lines of uncertain identification text, it does not trigger punctuation break detection because the number of characters in each line is less than 20. However, when it detects that the number of characters in the fifth line exceeds 20, it triggers the first punctuation break detection on the uncertain identification text "123456789,123456789,123" in the fifth line, and the user terminal also performs punctuation detection on the uncertain identification text in the fifth line. During the detection process, it notices that the uncertain identification text in the fifth line does not have a period, that is, it discovers that it does not have first-class break punctuation. In this case, break detection cannot be performed successfully. Subsequently, it re-examines whether the uncertain identification text in the fifth line contains two or more commas, that is, two or more second-class break punctuation symbols. If the uncertain identification text on line 5 contains two commas, the second comma is designated as the current punctuation break point, and the identification text before the second comma, "123456789,123456789," is designated as the current definitive identification text. Since the uncertain identification text on lines 6 through 9 does not increase by 20 characters compared to the uncertain identification text on line 5, no punctuation break detection trigger is performed for each line from line 6 to line 9. If the uncertain identification text on line 10 increases by 20 characters compared to the uncertain identification text on line 5, punctuation break detection is triggered for the uncertain identification text on line 10.When the user terminal performs punctuation break detection on line 10, the uncertain identified text on line 10 contains the definitive identified text "123456789,123456789," which was successfully identified earlier by the punctuation break on line 5. Therefore, a similarity contrast is first performed between the uncertain identified text on line 10 and the definitive identified text "1234567889,123456789,". Based on similarity contrast, the definitively identified text "123456789,123456789,123456789.123456789,123" in the 10th line is used to ignore the previously obtained definitively identified text "123456789,123456789," and the remaining uncertainly identified text "123456789.123456789,123" is extracted. Punctuation break detection is then performed on the extracted uncertainly identified text "123456789.123456789,123". In this case, since "123456789.123456789,123" contains one period, it is determined that this truncated, uncertain identification text meets the punctuation break condition, and "123456789." before the period is used as the definitive identification text. In the embodiment of the present invention, the above definitive identification text "123456789,123456789," or "123456789." is either sent to the translation engine for translation, speech synthesis is completed, and finally the synthesized speech is played back by the user terminal.

[0028] As shown in Figure 3, the present invention provides another audio processing method for realizing streaming TTS. The method includes the following steps.

[0029] In step S1, the user begins speaking and the user's device is woken up by voice.

[0030] In step S2, the user terminal sends a speech recognition request to the identification engine. If the speech recognition request is successful, the user terminal sends a streaming audio packet to the identification engine and performs speech recognition. The identification engine continues to respond to the user terminal with uncertain identification text. The user terminal detects and identifies the uncertain identification text. Specifically, the user terminal continuously receives uncertain identification text that has been responded to multiple times by the identification engine. Subsequently, the user terminal makes a judgment by comprehensively comparing the most recently responded N uncertain identification texts. N takes a value in the range of (2,6). If the identification content of the preceding M clauses for the N uncertain identification texts has not changed at all, the range of M is (1,5). In this case, the identification content of the preceding M clause is recognized as the current definitive identification text.

[0031] In step S3, the user terminal sends the previously determined definitive identification text to the aforementioned identification engine for translation and completes the speech synthesis. Finally, the user terminal plays the synthesized speech.

[0032] In an embodiment of the present invention, if the N most recent indeterminate identification texts received by the identification engine each contain one or more previously determined identification texts, the user terminal compares the N most recent indeterminate identification texts with the one or more previously determined identification texts for similarity, ignores the one or more previously determined identification texts, cuts out the remaining portion of the N indeterminate identification texts, and then makes a judgment by comprehensively comparing the remaining portion of the N indeterminate identification texts. If the identification content of the preceding M phrase for the remaining portion of the N indeterminate identification texts has not changed in any case, the identification content of the preceding M phrase is recognized as the current definitive identification text. Similarly, in this method, the user terminal sends the identification text determined each time to the translation engine for translation and performs speech synthesis on the translated text. If the length of the text string for which speech synthesis has already been completed but has not been played back via the user terminal is 40 or more, the user terminal adjusts the speech playback speed to 1.5 times the default value. If text strings whose speech synthesis has already been completed but have not yet been played back via the user's terminal are less than 40 characters long, the user's terminal will adjust the speech playback speed to the default value.

[0033] Similarly, using Figure 2 as an example, N is 5 and M is 2. Since there are no two sentences in the four lines of uncertain identified text, we ignore them. From the fifth line to the tenth line of uncertain identified text, there are five lines of uncertain identified text in between. Each of these lines contains the same two preceding sentences, "123456789,123456789," so we identify "123456789,123456789," as the definite identified text. From the lines six through nine of uncertain identified text, we ignored the previously determined identified text "123456789,123456789," so there are no two sentences in the remaining uncertain identified text, and therefore no overall comparison is performed between them. However, if there are still two or more sentences remaining after ignoring "123456789,123456789," from the uncertain identified text from lines 10 to 14, a similarity comparison is performed between the uncertain identified text from lines 10 to 14 and "123456789,123456789,". Then, all of the definitive identified texts "123456789,123456789," are ignored, and a judgment is made by comprehensively comparing the remaining uncertain identified texts. If it is found that all of them contain the same part "123456789.123456789," then the identified text at that time is designated as the definitive identified text "123456789.123456789".

[0034] In the embodiment of the present invention, the definitive identification texts "123456789,123456789," and "123456789.123456789" obtained by this method are both sent to a translation engine for translation, and speech synthesis is completed. Finally, the synthesized speech is played back by the user terminal.

[0035] Based on the above, the present invention has the following advantages. (1) The present invention performs punctuation and sentence break detection on uncertain identification text recently responded to by the identification engine via the user terminal, predetermines some definitive identification results, and sends the already determined identification results to the translation engine in a stream for translation and speech synthesis. In this method, the user terminal can obtain or determine the identification text in advance before the identification engine responds to the user terminal with definitive or final identification text. Compared to conventional methods, the waiting time for the user to hear the translated speech is reduced, and the effect of artificial simultaneous interpretation is achieved. (2) The present invention also reduces the waiting time until the user hears the translated audio and achieves the effect of simultaneous artificial interpretation by having the identification engine at the user terminal comprehensively compare and judge multiple uncertain identification texts that have recently responded, thereby predetermining some definitive identification results, and sending the already determined identification results to the translation engine in a stream for translation and speech synthesis. (3) The present invention dynamically adjusts the playback speed of TTS speech text by determining the length of the text string for which speech synthesis has been completed but which has not been played back by the user terminal, thereby reducing the time required to play back the translated speech.

[0036] The embodiments described above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to those described above. Any other modifications, alterations, substitutions, combinations, or simplifications that do not violate the spirit and principles of the present invention are all within the scope of protection of the present invention.

Claims

1. A speech processing method for implementing a streaming TTS, comprising the steps of: a user starting to speak and a voice wake-up being performed on the user terminal; the user terminal sending a voice recognition request to an identification engine, and if the voice recognition request is successful, the user terminal sending a streaming audio packet to the identification engine to perform voice recognition; and the identification engine continuously responding to the user terminal with uncertain identification text, wherein When the user terminal performs detection and identification of the uncertain identification text, specifically, the user terminal performs character detection on the uncertain identification text that the identification engine has responded to each time, and when it is detected that the number of characters is n or more for the first time, N takes the value (20, 50), and the uncertain identification text at this time is denoted as B1, and the user terminal triggers the first punctuation break detection on the uncertain identification text B1, and thereafter, in the latest responding uncertain identification text, a punctuation break detection is triggered each time n characters are added based on the length of the uncertain identification text B1. Each time punctuation break detection is successfully triggered, the user terminal performs one punctuation break detection operation. If the user terminal's punctuation break detection meets the punctuation break condition, it is determined that the break was successful, and the uncertain identified text prior to the punctuation break is recognized as the definitive identified text. A speech processing method for realizing streaming TTS, characterized in that each time an identification text is determined, the user terminal transmits the newly determined identification text to a translation engine for translation and completion of speech synthesis, and finally the user terminal plays the synthesized speech.

2. Each time punctuation break detection is successfully triggered, first, it is determined whether the most recent indeterminate identified text returned from the identification engine contains one or more definitive identified texts obtained through previous break detection successes. If the latest uncertain identified text does not contain any definitive identified text obtained through successful fragmentation so far, then it is directly determined whether the latest uncertain identified text meets the punctuation fragmentation condition. If the latest uncertain identified text contains one or more definitive identified texts obtained through successful fragmentation so far, first, a similarity comparison is performed between the latest uncertain identified text and the one or more definitive identified texts obtained through successful fragmentation so far. Then, the one or more definitive identified texts obtained so far are ignored, the remaining uncertain identified text is cut out, and punctuation fragmentation detection is performed on the cut-out uncertain identified text to determine whether the cut-out uncertain identified text meets the punctuation fragmentation condition. The audio processing method for realizing the streaming TTS described in feature 1.

3. The audio processing method for realizing a streaming TTS according to claim 2 is characterized in that, specifically, when performing detection on an uncertain identified text that is to be subjected to punctuation break detection, if the uncertain identified text that is to be subjected to punctuation break detection contains a first class punctuation mark for break detection, it is determined that the break was successful.

4. The audio processing method for realizing a streaming TTS according to claim 3, characterized in that if the uncertain identification text to be detected for punctuation breaks contains one first-class punctuation mark for breaks, that first-class punctuation mark for breaks becomes the current punctuation break point, and the portion of the uncertain identification text prior to this punctuation break point is determined as the current definitive identification text.

5. The audio processing method for realizing a streaming TTS according to claim 3, characterized in that, if the uncertain identification text to be detected for punctuation breaks contains two or more first-class punctuation symbols for breaks, the last first-class punctuation symbol for breaks is designated as the punctuation break point at that time, and the portion of the uncertain identification text prior to this punctuation break point is designated as the definitive identification text at that time.

6. If the uncertain identification text to be detected for punctuation breaks does not contain a first-class break punctuation mark, then it is detected whether it contains a second-class break punctuation mark. The audio processing method for realizing a streaming TTS according to claim 3, characterized in that, if the uncertain identification text to be detected for punctuation breaks contains two or more second-class punctuation symbols for breaks, it is determined that the break was successful, and the last second-class punctuation symbol for breaks is designated as the punctuation break point at that time, and the portion of the uncertain identification text before this punctuation mark is determined as the definitive identification text at that time, otherwise it is determined that the break was unsuccessful.

7. The audio processing method for realizing a streaming TTS according to any one of claims 3 to 6, characterized in that the first class of punctuation marks for fragments includes periods, exclamation marks, question marks, and semicolons.

8. The audio processing method for realizing a streaming TTS according to claim 6, characterized in that the second class of punctuation marks for fragments includes a comma.

9. Each time, the definitively identified text obtained from punctuation-based sentences is sent to the translation engine for translation, and the translated text is then synthesized into speech. If the length of the text string that has already been synthesized but has not yet been played back by the user terminal is 40 or more, the user terminal adjusts the playback speed of that text string to 1.5 times the default value. The audio processing method for realizing streaming TTS according to claim 1, characterized in that, if the length of a text string that has already been synthesized but has not been played back by the user terminal is less than 40 characters, the user terminal adjusts the speed of the audio playback to a default value.

10. The steps include: the user begins speaking and the user's device is voice-waked; The user terminal sends a speech recognition request to the identification engine, and if the speech recognition request is successful, the user terminal sends a streaming audio packet to the identification engine to perform speech recognition. The identification engine is an audio processing method for realizing a streaming TTS, which includes the step of continuously responding to the user terminal with uncertain identification text, The user terminal performs detection and identification of the uncertain identification text by, specifically, continuously receiving the uncertain identification text that has been responded to multiple times by the identification engine, and then the user terminal makes a judgment by comprehensively comparing it with the most recently responded N uncertain identification texts, where N takes a value in the range of (2, 6). If the identification content of the preceding M clauses of the N uncertain identification texts remains unchanged, the range of M is (1, 5), and in this case, the identification content of the preceding M clauses is recognized as the current definitive identification text. A speech processing method for realizing streaming TTS, characterized in that the user terminal transmits the definitive identification text determined each time to the identification engine for translation and completion of speech synthesis, and finally plays the synthesized speech on the user terminal.

11. The method, in which, if the most recently responded N uncertain identification texts by the identification engine each contain one or more previously determined identification texts, the user terminal compares the similarity of the most recently responded N uncertain identification texts with the one or more previously determined identification texts, ignores the one or more previously determined identification texts, cuts out the remaining portion of the N uncertain identification texts, and then makes a judgment by comprehensively comparing the remaining portion of the N uncertain identification texts. If the identification content of the preceding M clauses for the remaining parts of the N uncertain identification texts remains unchanged, then the identification content of the preceding M clauses is recognized as the current definitive identification text. The audio processing method for realizing the streaming TTS described in feature 10.

12. The user terminal sends the determined identification text to the translation engine each time for translation, and then performs speech synthesis on the translated text. If speech synthesis has already been completed but the length of the text string that has not been played back via the user terminal is 40 or more, the user terminal adjusts the speech playback speed to 1.5 times the default value. The audio processing method for realizing a streaming TTS according to claim 10, characterized in that, if speech synthesis has already been completed but the length of the text string that has not been played back via the user terminal is less than 40, the user terminal adjusts the audio playback speed to a default value.

Images