Speech translation method, related device, equipment and storage medium

Abstract

The invention discloses a speech translation method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining a recognition result of a to-be-translated speech; wherein the recognition result comprises recognition characters and pronunciation units which are respectively aligned with the audio features of each voice frame in the to-be-translated voice; analyzing based on a recognition result, and selecting a voice frame as a semantic segmentation point; the method comprises the following steps: detecting a sentence end segmentation point after a historical segmentation point in response to the historical segmentation point, determining a target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and combining audio features as a feature set by taking a semantic segmentation point not later than the historical segmentation point as an initial position of the segmentation and taking the target segmentation point as an end position of the segmentation; based on the feature set, translating to obtain a translated text; wherein the historical segmentation points are semantic segmentation points serving as end positions during last segmentation. According to the scheme, the real-time performance and accuracy of speech translation can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech translation technology, and in particular to a speech translation method and related devices, equipment and storage media. Background Art

[0002] With the development of globalization, the demand for cross-language communication is increasing, and the demand for faster and more accurate voice translation is also growing.

[0003] However, existing speech translation technology still has a large delay problem, or although the delay problem has been improved, the translation results are prone to deviation. In view of this, how to improve the real-time and accuracy of speech translation has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech translation method and related devices, equipment and storage media, which can improve the real-time and accuracy of speech translation.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech translation method, including: obtaining a recognition result of a speech to be translated; wherein the recognition result includes: recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated; analyzing based on the recognition result, selecting the speech frame as a semantic segmentation point; wherein the semantic segmentation point belongs to any one of the categories of a sentence end segmentation point and a sentence intra-sentence segmentation point; in response to detecting a sentence end segmentation point after a historical segmentation point, determining a target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and taking a semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combining audio features as a feature set, and translating to obtain a translation text based on the feature set; wherein the historical segmentation point is the semantic segmentation point that was the ending position at the time of the last segmentation.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech translation device, including: an acquisition module, an analysis module and a translation module, the acquisition module is used to obtain the recognition result of the speech to be translated; wherein the recognition result includes: recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated; the analysis module is used to analyze based on the recognition result and select the speech frame as a semantic segmentation point; wherein the semantic segmentation point belongs to any one of the categories of sentence end segmentation point and sentence intra-sentence segmentation point; the translation module is used to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and use the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as a feature set, and translate to obtain a translation text based on the feature set; wherein the historical segmentation point is the semantic segmentation point used as the ending position at the time of the last segmentation.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech translation method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the speech translation method of the first aspect.

[0009] The above scheme obtains the recognition result of the speech to be translated, and the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated, and then analyzes based on the recognition result, selects the speech frame as the semantic segmentation point, and the semantic segmentation point belongs to any category of the sentence end segmentation point and the sentence intra-sentence segmentation point, so as to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and take the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as a feature set, and based on the feature set , the translation is done to get the translated text, and the historical segmentation point is the semantic segmentation point that was the end position at the last segmentation. Therefore, on the one hand, the latest end-of-sentence segmentation point is used as the segmentation basis for each segmentation to obtain the feature set and translate accordingly, which can reduce the number of calculation rounds of speech translation and reduce data accumulation as much as possible, which helps to improve the real-time performance of speech translation. On the other hand, since the end-of-sentence segmentation point with the most complete semantic information is used as the segmentation basis for each segmentation as much as possible, rather than the intra-sentence segmentation point with relatively complete semantic information or other segmentation points, the semantic integrity of each segmentation can be improved as much as possible, the possibility of translation deviation can be reduced as much as possible, and the accuracy of speech translation can be improved. Therefore, the real-time performance and accuracy of speech translation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the speech translation method of the present application; Figure 2a This is a process diagram of an embodiment of the speech translation method of the present application; Figure 2b It is a process diagram of another embodiment of the speech translation method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the speech translation device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech translation method of the present application. Specifically, it may include the following steps: Step S11: Obtain the recognition result of the speech to be translated.

[0015] In the disclosed embodiment, the recognition result includes: recognized characters and pronunciation units that are respectively aligned with the audio features of each speech frame in the speech to be translated. For ease of understanding, taking the speech to be translated "Good morning! Have you eaten?" as an example, Table 1 shows a possible recognition result of the speech to be translated. Of course, what is shown in Table 1 is only one of the possible recognition results of the translated speech. Due to factors such as the accent and speaking speed of the speaker of the speech to be translated, the recognition result may also be other situations than those shown in Table 1, and examples will not be given one by one here.

[0016] Table 1 Schematic table of recognition results of an embodiment

[0017]

[0018] It should be noted that the recognition result can be obtained by processing the speech to be translated by a speech recognition model. The speech recognition model may include but is not limited to a neural network model such as an encoder-decoder architecture, and the network structure of the speech recognition model is not limited here.

[0019] Step S12: Analyze based on the recognition result and select the speech frame as the semantic segmentation point.

[0020] In the embodiments of the present disclosure, the semantic segmentation point belongs to any one of the end-of-sentence segmentation points and in-sentence segmentation points. It should be noted that the end-of-sentence segmentation point has more complete semantics than the in-sentence segmentation point. Taking "Ah, the weather is really nice today!" as an example, among them, the speech frame aligned with the punctuation mark "," can be used as an in-sentence segmentation point, while the speech frame aligned with the punctuation mark "!" can be used as an end-of-sentence segmentation point. Obviously, the semantic information of the former is relatively scarce (only able to reflect the tone), while the semantic information of the latter is more complete (able to reflect the expression of the nice weather). Of course, the above example is only a possible example of the end-of-sentence segmentation point and the in-sentence segmentation point, and other possible situations will not be exemplified one by one here.

[0021] In one implementation scenario, the recognized characters may include text characters, punctuation characters, etc. As shown in Table 1 above, "zao", "shang", "hao", "chi", "le", "ma" are all text characters, while "!" and "?" are all punctuation characters. Of course, the text characters and punctuation characters are not limited to this. For example, "," is also a punctuation character. Other text characters and punctuation characters can refer to the specific meanings of the text and punctuation, which will not be elaborated here. On this basis, as a possible example, the speech frame aligned with the punctuation character can be directly selected as the semantic segmentation point; or, as another possible example, the speech frame aligned with the punctuation characters indicating the end of the sentence, such as "!", "?", "." etc., can also be directly selected as the semantic segmentation point. Of course, the above examples are only several possible examples for determining the semantic segmentation point in the actual application process, and other possible situations will not be exemplified one by one here.

[0022] In another implementation scenario, different from the above implementation manner of directly determining the semantic segmentation point by the punctuation character, as another possible implementation manner, the text character adjacent to the punctuation character can be first selected as the target character, and then the pronunciation information mapped by the target character from the preset dictionary is obtained as the first pronunciation, and the pronunciation information of the pronunciation unit corresponding to the target character is obtained as the second pronunciation. Thus, in response to the successful matching of the first pronunciation and the second pronunciation, based on the relative position between the target character and the punctuation character, and based on the speech frame aligned with the target unit, the semantic segmentation point is determined, and the target unit is the pronunciation unit corresponding to the target character. The above method combines the punctuation character and the pronunciation information determined from different channels by the adjacent text character for matching to determine the semantic segmentation point. Compared with simply directly determining the semantic segmentation point by the punctuation character, it can maximize the avoidance of missing some pronunciations or including extra pronunciations when segmenting with the semantic segmentation point subsequently.

[0023] In a specific implementation scenario, the text characters adjacent to punctuation characters can be the text characters before the punctuation character or the text characters after the punctuation character. Still taking the recognition result shown in Table 1 as an example, when selecting the punctuation character "!", the text character "good" before it can be selected as the target character, or the text character "eat" after it can be selected as the target character, which is not limited here. Of course, the above example is only a possible example of selecting the target character based on the recognition result shown in Table 1, and other possible situations are not listed one by one here.

[0024] In a specific implementation scenario, the preset dictionary can be determined according to the language to which the speech to be translated belongs. For example, when the language to which the speech to be translated belongs is Chinese, a Chinese dictionary such as "Modern Chinese Words" can be selected as the preset dictionary, and when the language to which the speech to be translated belongs is English, an English dictionary such as "Oxford English Dictionary" can be selected as the preset dictionary. The above examples are only several possible situations of the preset dictionary, and are not listed one by one here either.

[0025] In a specific implementation scenario, still taking the recognition result shown in Table 1 as an example, when selecting the punctuation character "!" and selecting the text character "good" before it as the target character, the target character "good" can obtain its pronunciation information "hao3" through the preset dictionary, which can be used as the first pronunciation, and the pronunciation information "hao3" of the pronunciation unit corresponding to the target character "good" can be used as the second pronunciation; or, when selecting the punctuation character "!" and selecting the text character "eat" after it as the target character, the target character "eat" can obtain its pronunciation information "chi1" through the preset dictionary, which can be used as the first pronunciation, and the pronunciation information "chi1" of the pronunciation unit corresponding to the target character "eat" can be used as the second pronunciation. Of course, the above example is only several possible examples of the first pronunciation and the second pronunciation based on the recognition result shown in Table 1, and other possible situations are not listed one by one here.

[0026] In a specific implementation scenario, the first pronunciation and the second pronunciation can be matched by means of string matching to determine whether the two match successfully or fail. The specific matching method can refer to the relevant technology of string matching, which will not be elaborated here.

[0027] In a specific implementation scenario, when the first pronunciation matches the second pronunciation successfully, if the target character is a text character before a punctuation character, the speech frame aligned with the last target unit can be selected as the semantic segmentation point. That is to say, the pronunciation unit corresponding to the target character can be determined first as the target unit, and then the last one among the target units and the speech frame aligned with it can be selected as the semantic segmentation point. Still taking the recognition result shown in Table 1 above as an example, when the punctuation character "!" is selected and the text character "good" before it is selected as the target character, if the match is successful, the pronunciation unit corresponding to the target character "good" (i.e., frame numbers 13 to 17) can be selected as the target unit, and the speech frame aligned with the last one (i.e., frame number 17) among them can be selected as the semantic segmentation point. Of course, the above example is only a possible example of the semantic segmentation point taking the recognition result described in Table 1 as an example, and no more examples will be given here one by one.

[0028] In a specific implementation scenario, when the first pronunciation matches the second pronunciation successfully, different from the aforementioned possible situations, if the target character is a text character after a punctuation character, the speech frame before the speech frame aligned with the first target unit can be selected as the semantic segmentation point. That is to say, the pronunciation unit corresponding to the target character can be determined first as the target unit, and then the first one among the target units and the speech frame before the speech frame aligned with it can be selected as the semantic segmentation point. Still taking the recognition result shown in Table 1 above as an example, when the punctuation character "!" is selected and the text character "eat" after it is selected as the target character, if the match is successful, the pronunciation unit corresponding to the target character "eat" (i.e., frame numbers 19 to 21) can be selected as the target unit, and the speech frame before the speech frame aligned with the first one (i.e., frame number 19) among them (i.e., the speech frame of frame number 18) can be selected as the semantic segmentation point. Of course, the above example is only a possible example of the semantic segmentation point taking the recognition result described in Table 1 as an example, and no more examples will be given here one by one.

[0029] In a specific implementation scenario, during actual application, it is also possible that the first pronunciation fails to match the second pronunciation. In such a case, it is possible to continue waiting for a new recognition result to add a target character to the punctuation character. Exemplarily, still taking the recognition result shown in Table 1 above as an example, when selecting the punctuation character "!", if the previous text character "good" is selected as the target character but the match fails, it is possible to continue waiting for a new recognition result, such as "eat" in Table 1, and continue to add the new target character "eat" to the punctuation character "!", and return to the step of obtaining the pronunciation information mapped by the target character from the preset dictionary as the first pronunciation. If the match is successful, the semantic segmentation point can be determined according to the relevant description above; if the match still fails, the punctuation character can be skipped (i.e., the punctuation character "!" can be skipped in this example) and the next punctuation character can be selected (i.e., when a new recognition result appears and a new punctuation character "?" appears in this example, the punctuation character "?" can be selected), and continue to execute the step of selecting the text character adjacent to the punctuation character as the target character to determine the semantic segmentation point according to the next punctuation character. In the above manner, when matching in combination with pronunciation information based on the punctuation character, if the match fails, continue to wait for a new recognition result to add a target character to the punctuation character and continue the pronunciation match. In the case where the match still fails, skip the punctuation character and select the next punctuation character to continue the above process, which can maximize the avoidance of missing some pronunciations or including extra pronunciations when performing segmentation based on the semantic segmentation point subsequently.

[0030] It should be noted that, as a possible example, whether the semantic segmentation point is an in-sentence segmentation point or an end-of-sentence segmentation point can be determined by the punctuation character. For example, if the semantic segmentation point itself or the recognition character corresponding to the adjacent speech frame is a punctuation character indicating the end of a sentence, such as ".", "!", "?", etc., it can be considered that the semantic segmentation point is an end-of-sentence segmentation point; conversely, if the semantic segmentation point itself or the recognition character corresponding to the adjacent speech frame is a punctuation character indicating that the sentence has not ended, such as ",", ";", etc., it can be considered that the semantic segmentation point is an in-sentence segmentation point. Of course, the above examples are only a possible example in the actual application process, and other possible situations are not limited here. For example, after obtaining the semantic segmentation point, the semantic segmentation point can also be classified by a neural network model (such as a large language model, etc.) to determine whether the semantic segmentation point is an end-of-sentence segmentation point or an in-sentence segmentation point. That is, the semantic segmentation point can no longer be classified solely based on the punctuation character, but can be classified by the neural network model based on the context semantics.

[0031] Step S13: In response to detecting a sentence end segmentation point after a historical segmentation point, a target segmentation point is determined based on the latest sentence end segmentation point after the historical segmentation point, and a semantic segmentation point no later than the historical segmentation point is used as the starting position of this segmentation and the target segmentation point is used as the ending position of this segmentation, audio features are combined as a feature set, and based on the feature set, a translation is performed to obtain a translation text.

[0032] In the disclosed embodiment, the historical segmentation point is the semantic segmentation point used as the end position at the last segmentation. Still taking the recognition result shown in Table 1 as an example, the speech frame with frame number 18 (i.e., the speech frame corresponding to "!") can be used as the semantic segmentation point. If the speech frame with frame number 18 is used as the end position at the last segmentation, the historical segmentation point can be the speech frame with frame number 18. Of course, the above example is only a possible example of the historical segmentation point using the recognition result shown in Table 1 as an example, and other possible situations will not be given examples one by one here.

[0033] In an implementation scenario, as a possible example, when a sentence-end segmentation point is detected after a historical segmentation point, the latest sentence-end segmentation point after the historical segmentation point can be directly selected as the target segmentation point. Exemplarily, still taking the recognition result shown in Table 1 as an example, when the historical segmentation point is speech frame No. 18, if the latest recognition result is speech frame No. 28 in Table 1, and speech frame No. 27 is determined as the sentence-end segmentation point through the aforementioned process steps, then since the latest sentence-end segmentation point after the historical segmentation point is speech frame No. 27, speech frame No. 27 can be used as the target segmentation point; if the latest recognition result further adds several speech frames based on the recognition result shown in Table 1, such as the speech frame corresponding to "Just going to eat!" and the speech frame corresponding to the punctuation character "!" in "Just going to eat!" (recorded as speech frame No. N) is determined as the sentence-end segmentation point through the aforementioned process steps, then since the latest sentence-end segmentation point after the historical segmentation point is speech frame No. N, speech frame No. N can be used as the target segmentation point. Of course, the above examples are only several possible examples of directly selecting the latest sentence end segmentation point after the historical segmentation point as the target segmentation point, and other possible situations are not given one by one here. The above method directly selects the latest sentence end segmentation point after the historical segmentation point as the target segmentation point, which can minimize the number of calculation rounds of speech translation and reduce data accumulation while ensuring the accuracy of speech translation as much as possible.

[0034] In another implementation scenario, as another possible example, when a sentence-end segmentation point is detected after a historical segmentation point, a judgment result about the sentence-end segmentation point can also be obtained first, and the judgment result includes: whether there is at least one sentence-end segmentation point between the historical segmentation point and the latest sentence-end segmentation point after the historical segmentation point. In response to the judgment result being yes, the sentence-end segmentation point after the historical segmentation point and closest to the historical segmentation point can be selected as the target segmentation point, and in response to the judgment result being no, the latest sentence-end segmentation point after the historical segmentation point can be selected as the target segmentation point. Exemplarily, still taking the above example as an example, if the historical segmentation point is the 18th speech frame and the latest sentence-end segmentation point is the aforementioned Nth speech frame, then since the judgment result about the sentence-end segmentation point includes that there is another sentence-end segmentation point (i.e., the 27th speech frame) between the sentence-end segmentation point and the latest sentence-end segmentation point after the historical segmentation point, then the sentence-end segmentation point (i.e., the 27th speech frame) after the historical segmentation point (i.e., the 18th speech frame) and closest to the historical segmentation point (i.e., the 18th speech frame) can be selected as the target segmentation point; if the historical segmentation point is the 18th speech frame and the latest sentence-end segmentation point is the 27th speech frame, then since the judgment result about the sentence-end segmentation point includes that there is no sentence-end segmentation point between the sentence-end segmentation point and the latest sentence-end segmentation point after the historical segmentation point, then the latest sentence-end segmentation point after the historical segmentation point (i.e., the 27th speech frame) can be selected as the target segmentation point. Of course, the above examples are only several possible examples when the latest sentence-end segmentation point after the historical segmentation point is not directly used as the target segmentation point, and other possible situations are not given examples one by one here. In the above method, a judgment result about the sentence-end segmentation point is first obtained, and the judgment result includes: whether there is at least one sentence-end segmentation point between the historical segmentation point and the latest sentence-end segmentation point after the historical segmentation point, and then in response to the judgment result being yes, a sentence-end segmentation point after the historical segmentation point and closest to the historical segmentation point is selected as the target segmentation point, and in response to the judgment result being no, the latest sentence-end segmentation point after the historical segmentation point is selected as the target segmentation point. This method can select the latest sentence-end segmentation point as much as possible without skipping the sentence-end segmentation point when selecting the target segmentation point, which helps to avoid as much as possible the following unfavorable situations after the segmentation: translation errors due to excessively long data, loss of cached data due to limiting the input length, and so on, so that speech translation is suitable for scenarios with limited inference calculation length.

[0035] In an implementation scenario, in the actual application process, in addition to the possibility of detecting the end-of-sentence segmentation point after the historical segmentation point, it is also possible that the end-of-sentence segmentation point is not detected after the historical segmentation point, but the intra-sentence segmentation point is detected. In this case, as a possible implementation method, it is possible to wait from the intra-sentence segmentation point detected after the historical segmentation point until the end-of-sentence segmentation point is detected, and then perform the step of determining the target segmentation point based on the latest end-of-sentence segmentation point after the historical segmentation point. In other words, in the case where the sentence segmentation point is not detected but the intra-sentence segmentation point is detected, the intra-sentence segmentation point can be delayed calculated, that is, the intra-sentence segmentation point is not used as the target segmentation point to perform subsequent translation calculations, but waits for a period of time in the expectation that the end-of-sentence segmentation point will appear later. As another possible implementation method, the speech to be translated can be a streaming speech, that is, the speech to be translated can be continuously input (or collected), and then the speech to be translated can be streamed recognized to obtain a recognition result, and the recognition result can also be continuously updated over time (that is, new content is added, as in the above example). In this case, in order to avoid waiting too long and affecting the real-time performance of speech translation, it is also possible to wait for a first time period (such as 3 seconds, 4 seconds, etc.) starting from the intra-sentence segmentation point detected after the historical segmentation point. In response to detecting the sentence-end segmentation point within the first time period, the aforementioned step of determining the target segmentation point based on the latest sentence-end segmentation point after the historical segmentation point can be performed; and in response to not detecting the sentence-end segmentation point within the first time period, the target segmentation point can be determined based on the intra-sentence segmentation point after the historical segmentation point. That is to say, if the sentence-end segmentation point is still not detected after waiting for the first time period starting from the intra-sentence segmentation point, the intra-sentence segmentation point after the historical segmentation point can be directly selected as the target segmentation point.

[0036] In an implementation scenario, in the actual application process, in addition to the possibility that the sentence-end segmentation point may be detected after the historical segmentation point, or the sentence-end segmentation point may not be detected but the sentence-inside segmentation point may be detected, it is also possible that the semantic segmentation point may not be detected within the second time length after the historical segmentation point (that is, neither the sentence-end segmentation point nor the sentence-inside segmentation point can be detected), then the sentence-end segmentation point can be forcibly determined after the historical segmentation point based on the recognition result. Exemplarily, a logical judgment can be made on the recognition result so that the sentence-end segmentation point that is forcibly appended is more reasonable in semantics. For example, the logical judgment can be that the sentence-end segmentation point that is forcibly appended is not in the alignment of the speech frame with the pronunciation unit, etc. In the above manner, in response to the semantic segmentation point not being detected within the second time length after the historical segmentation point, the sentence-end segmentation point is forcibly determined after the historical segmentation point based on the recognition result, which can constrain the calculation amount of subsequent speech translation as much as possible.

[0037] In an implementation scenario, no matter which of the above situations (i.e., a sentence-end segmentation point is detected after a historical segmentation point, a sentence-end segmentation point is not detected after a historical segmentation point but an intra-sentence segmentation point is detected, or a semantic segmentation point is not detected after a historical segmentation point), when determining a target segmentation point (the specific process can refer to the aforementioned related description), a semantic segmentation point that is no later than the historical segmentation point can be used as the starting position of this segmentation and the target segmentation point can be used as the ending position of this segmentation, audio features can be combined as a feature set, and based on the feature set, a translation is performed to obtain a translation text. Exemplarily, still taking the recognition result shown in the aforementioned Table 1 as an example, when the target segmentation point is a speech frame with frame number 27, it can be used as the ending position of this segmentation, and a semantic segmentation point that is no later than the historical segmentation point (i.e., a speech frame with frame number 18), such as the historical segmentation point itself (i.e., a speech frame with frame number 18), or an end-of-sentence segmentation point before the historical segmentation point, can be used as the starting position of this segmentation. On this basis, the audio features of the speech frames from the starting position to the end position can be selected and combined to obtain a feature set, such as the audio features of the speech frames from frame number 18 to frame number 27, and the feature set can be combined, and then translated accordingly (for example, a large language model can be used for translation), and a translated text is obtained, such as the translated text "Have you had breakfast?" translated into the target language English. Of course, the above example is only a possible example in the actual application process, and other possible situations are not given one by one. In addition, as a possible example, after obtaining the translated text, speech synthesis can also be performed based on the translated text to obtain synthesized speech, and the synthesized speech can be transmitted to relevant personnel using the target language, or the translated text can be directly transmitted to relevant personnel using the target language to achieve simultaneous speech interpretation.

[0038] In an implementation scenario, please refer to Figure 2a and Figure 2b , Figure 2a is a process diagram of an embodiment of the speech translation method of the present application, Figure 2b FIG. 1 is a schematic diagram of another embodiment of the speech translation method of the present application. Figure 2a and Figure 2bAs shown, in the process of inputting the speech to be translated, speech recognition and data caching can be continuously performed. Based on this, it can be determined whether the distance to the previous end-of-sentence segmentation point is greater than a certain time length D1. If so, it is continued to determine whether an intra-sentence segmentation point appears, and the intra-sentence segmentation point is used as the sentence end segmentation point to obtain a new sentence end segmentation point. If no intra-sentence segmentation point appears, it can be determined whether the distance to the previous end-of-sentence segmentation point is greater than another time length D2. If so, a sentence end segmentation point can be forcibly added to obtain a new sentence end segmentation point. In this way, the above-mentioned data caching step can be returned to loop and iterate until the speech ends. In addition, if a new segmentation point (i.e., the above-mentioned semantic segmentation point) is added during the data caching process, it is determined whether the segmentation point is a sentence end segmentation point. If so, translation calculation can be allowed. If not, it can be determined whether the distance to the previous end-of-sentence segmentation point is delayed by more than a time length T1. If not, it is continued to return to the above-mentioned data caching step for iteration. On the contrary, if it is, translation calculation can be allowed. On this basis, if the translation calculation is not completed, you can return to the above data cache step for iteration. If the translation calculation is not completed, you can select the segmentation point from the position of the last translation calculation (i.e. the historical segmentation point) for translation calculation. If there is a sentence-end segmentation point after it, you can select the nearest sentence-end segmentation point for translation calculation. If there is no sentence-end segmentation point after it, you can select the latest intra-sentence segmentation point for translation calculation. Of course, Figure 2a and Figure 2b What is shown is only a possible example in actual application. Other possible situations of speech translation are as described above, and other possible related situations will not be given examples one by one here.

[0039] The above scheme obtains the recognition result of the speech to be translated, and the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated, and then analyzes based on the recognition result, selects the speech frame as the semantic segmentation point, and the semantic segmentation point belongs to any category of the sentence end segmentation point and the sentence intra-sentence segmentation point, so as to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and take the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as a feature set, and based on the feature set , the translation is done to get the translated text, and the historical segmentation point is the semantic segmentation point that was the end position at the last segmentation. Therefore, on the one hand, the latest end-of-sentence segmentation point is used as the segmentation basis for each segmentation to obtain the feature set and translate accordingly, which can reduce the number of calculation rounds of speech translation and reduce data accumulation as much as possible, which helps to improve the real-time performance of speech translation. On the other hand, since the end-of-sentence segmentation point with the most complete semantic information is used as the segmentation basis for each segmentation as much as possible, rather than the intra-sentence segmentation point with relatively complete semantic information or other segmentation points, the semantic integrity of each segmentation can be improved as much as possible, the possibility of translation deviation can be reduced as much as possible, and the accuracy of speech translation can be improved. Therefore, the real-time performance and accuracy of speech translation can be improved.

[0040] See also Figure 3 , Figure 3 It is a schematic diagram of a framework of an embodiment of a speech translation device of the present application. The speech translation device 30 includes: an acquisition module 31, an analysis module 32 and a translation module 33, wherein the acquisition module 31 is used to obtain the recognition result of the speech to be translated; wherein the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated; the analysis module 32 is used to analyze based on the recognition result and select the speech frame as a semantic segmentation point; wherein the semantic segmentation point belongs to any category of a sentence end segmentation point and a sentence intra-sentence segmentation point; the translation module 33 is used to respond to the detection of a sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and use the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as a feature set, and translate to obtain a translation text based on the feature set; wherein the historical segmentation point is the semantic segmentation point used as the ending position at the last segmentation.

[0041] In the above scheme, the speech translation device 30 obtains the recognition result of the speech to be translated, and the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated, and then analyzes based on the recognition result, selects the speech frame as the semantic segmentation point, and the semantic segmentation point belongs to any category of the sentence end segmentation point and the sentence intra-sentence segmentation point, so as to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and take the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as the feature set, and based on The feature set is used to translate the translated text, and the historical segmentation point is the semantic segmentation point that was the end position at the last segmentation. Therefore, on the one hand, the latest end-of-sentence segmentation point is used as the segmentation basis for each segmentation to obtain the feature set and translate accordingly, which can reduce the number of calculation rounds of speech translation and reduce data accumulation as much as possible, which helps to improve the real-time performance of speech translation. On the other hand, since the end-of-sentence segmentation point with the most complete semantic information is used as the segmentation basis for each segmentation as much as possible, rather than the intra-sentence segmentation point with relatively complete semantic information or other segmentation points, the semantic integrity of each segmentation can be improved as much as possible, the possibility of translation deviation can be reduced as much as possible, and the accuracy of speech translation can be improved. Therefore, the real-time performance and accuracy of speech translation can be improved.

[0042] In some disclosed embodiments, the translation module 33 includes a direct selection submodule for directly selecting the latest sentence-end segmentation point after the historical segmentation point as the target segmentation point.

[0043] In some disclosed embodiments, the translation module 33 includes an identification judgment submodule for obtaining a judgment result about a sentence-end segmentation point; wherein the judgment result includes: whether there is at least one sentence-end segmentation point between a historical segmentation point and the latest sentence-end segmentation point after the historical segmentation point; the translation module 33 includes a first selection submodule for selecting, in response to a yes judgment result, a sentence-end segmentation point after the historical segmentation point and closest to the historical segmentation point as a target segmentation point; the translation module 33 includes a second selection submodule for selecting, in response to a no judgment result, the latest sentence-end segmentation point after the historical segmentation point as a target segmentation point.

[0044] In some disclosed embodiments, the speech translation device 30 further includes a first loop module for responding to the fact that no sentence-end segmentation point is detected after a historical segmentation point but an intra-sentence segmentation point is detected, waiting from the intra-sentence segmentation point detected after the historical segmentation point until a sentence-end segmentation point is detected, and then executing the step of determining a target segmentation point based on the latest sentence-end segmentation point after the historical segmentation point.

[0045] In some disclosed embodiments, the first loop module includes a first waiting submodule, which is used to wait for a first time period starting from an intra-sentence segmentation point detected after a historical segmentation point; the first loop module includes a first response submodule, which is used to, in response to detecting an end-of-sentence segmentation point within the first time period, execute the step of determining a target segmentation point based on the latest end-of-sentence segmentation point after the historical segmentation point; the first loop module includes a second response submodule, which is used to determine the target segmentation point based on the intra-sentence segmentation point after the historical segmentation point in response to not detecting an end-of-sentence segmentation point within the first time period.

[0046] In some disclosed embodiments, the speech translation device 30 further includes a forced appending module for forcibly determining a sentence end segmentation point after the historical segmentation point based on the recognition result in response to not detecting a semantic segmentation point within a second time period after the historical segmentation point.

[0047] In some disclosed embodiments, the recognized characters include at least two types, namely, text characters and punctuation characters. The analysis module 32 includes a character selection submodule for selecting text characters adjacent to punctuation characters as target characters. The analysis module 32 includes a pronunciation acquisition submodule for acquiring pronunciation information of the target character mapped by a preset dictionary as a first pronunciation, and acquiring pronunciation information of a pronunciation unit corresponding to the target character as a second pronunciation. The analysis module 32 includes a point position determination submodule for determining a semantic segmentation point in response to a successful match between the first pronunciation and the second pronunciation, based on the relative position between the target character and the punctuation character and based on a speech frame aligned with the target unit. The target unit is a pronunciation unit corresponding to the target character.

[0048] In some disclosed embodiments, the point position determination submodule includes a first determination unit for selecting a speech frame aligned with the last target unit as a semantic segmentation point in response to the target character being a text character before a punctuation character; the point position determination submodule includes a second determination unit for selecting a speech frame before a speech frame aligned with the first target unit as a semantic segmentation point in response to the target character being a text character after a punctuation character.

[0049] In some disclosed embodiments, the analysis module 32 includes a second waiting submodule for, in response to a failure in matching the first pronunciation with the second pronunciation, continuing to wait for a new recognition result to be generated to add a target character for the punctuation character, and returning to the step of obtaining the pronunciation information of the target character mapped by a preset dictionary as the first pronunciation, and in the case of still failing to match, skipping the punctuation character and selecting the next punctuation character, and continuing to execute the step of selecting a text character adjacent to the punctuation character as the target character.

[0050] In some disclosed embodiments, the speech to be translated is streaming speech, and the speech translation device 30 further includes a second loop module for returning to the step of obtaining the recognition result of the speech to be translated to iterate after translating based on the feature set to obtain the translated text until the streaming speech ends.

[0051] See also Figure 4 , Figure 4 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned speech translation method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a server, a camera, a smart phone, a tablet computer, etc., and the specific type of the electronic device 40 is not limited here.

[0052] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned speech translation method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0053] In the above scheme, the electronic device 40 obtains the recognition result of the speech to be translated, and the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated, and then analyzes based on the recognition result, selects the speech frame as the semantic segmentation point, and the semantic segmentation point belongs to any category of the sentence end segmentation point and the sentence intra-sentence segmentation point, so as to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and take the semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as the feature set, and based on the feature The collection is collected, and the translation is obtained by translation. The historical segmentation point is the semantic segmentation point that was the end position of the last segmentation. Therefore, on the one hand, the latest sentence-end segmentation point is used as the segmentation basis for each segmentation to obtain the feature set and translate accordingly, which can reduce the number of calculation rounds of speech translation and reduce data accumulation as much as possible, which helps to improve the real-time performance of speech translation. On the other hand, since the sentence-end segmentation point with the most complete semantic information is used as the segmentation basis for each segmentation as much as possible, rather than the intra-sentence segmentation point with relatively complete semantic information or other segmentation points, the semantic integrity of each segmentation can be improved as much as possible, and the possibility of translation deviation can be reduced as much as possible, which helps to improve the accuracy of speech translation. Therefore, the real-time performance and accuracy of speech translation can be improved.

[0054] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned speech translation method embodiments.

[0055] In the above scheme, the computer-readable storage medium 50 obtains the recognition result of the speech to be translated, and the recognition result includes: the recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated, and then analyzes based on the recognition result, selects the speech frame as the semantic segmentation point, and the semantic segmentation point belongs to any category of the sentence end segmentation point and the sentence intra-sentence segmentation point, so that in response to the detection of the sentence end segmentation point after the historical segmentation point, based on the latest sentence end segmentation point after the historical segmentation point, the target segmentation point is determined, and the semantic segmentation point no later than the historical segmentation point is used as the starting position of this segmentation and the target segmentation point is used as the ending position of this segmentation, the audio features are combined as the feature set, and based on Based on the feature set, the translation gets the translated text, and the historical segmentation point is the semantic segmentation point that was the end position at the last segmentation. Therefore, on the one hand, the latest end-of-sentence segmentation point is used as the segmentation basis for each segmentation to obtain the feature set and translate accordingly, which can reduce the number of calculation rounds of speech translation and reduce data accumulation as much as possible, which helps to improve the real-time performance of speech translation. On the other hand, since the end-of-sentence segmentation point with the most complete semantic information is used as the segmentation basis for each segmentation as much as possible, rather than the intra-sentence segmentation point with relatively complete semantic information or other segmentation points, the semantic integrity of each segmentation can be improved as much as possible, the possibility of translation deviation can be reduced as much as possible, and the accuracy of speech translation can be improved. Therefore, the real-time performance and accuracy of speech translation can be improved.

[0056] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0057] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0058] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0059] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0060] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0061] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0062] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech translation method, characterized in that: include: Acquire the recognition result of the speech to be translated; wherein the recognition result includes: recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated; Analyzing the recognition result, selecting the speech frame as a semantic segmentation point; wherein the semantic segmentation point belongs to any one of the categories of a sentence end segmentation point and a sentence intra-sentence segmentation point; In response to detecting the sentence end segmentation point after the historical segmentation point, a target segmentation point is determined based on the latest sentence end segmentation point after the historical segmentation point, and a semantic segmentation point no later than the historical segmentation point is used as the starting position of this segmentation and the target segmentation point is used as the ending position of this segmentation, the audio features are combined as a feature set, and based on the feature set, a translation is performed to obtain a translation text; wherein the historical segmentation point is the semantic segmentation point used as the ending position at the time of the last segmentation.

2. The method according to claim 1, characterized in that The determining of the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point includes: The latest sentence end segmentation point after the historical segmentation point is directly selected as the target segmentation point.

3. The method according to claim 1, characterized in that The determining of the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point includes: Obtaining a judgment result on the sentence end segmentation point; wherein the judgment result includes: whether there is at least one sentence end segmentation point between the historical segmentation point and the latest sentence end segmentation point after the historical segmentation point; In response to the judgment result being yes, selecting a sentence-end segmentation point after the historical segmentation point and closest to the historical segmentation point as the target segmentation point; In response to the judgment result being no, the latest sentence end segmentation point after the historical segmentation point is selected as the target segmentation point.

4. The method according to claim 1, characterized in that The method further comprises: In response to the fact that the sentence-end segmentation point has not been detected after the historical segmentation point but the intra-sentence segmentation point has been detected, waiting starts from the intra-sentence segmentation point detected after the historical segmentation point until the sentence-end segmentation point is detected, and then the step of determining the target segmentation point based on the latest sentence-end segmentation point after the historical segmentation point is performed.

5. The method according to claim 4, characterized in that The method further comprises: Waiting for a first duration starting from an intra-sentence segmentation point detected after the historical segmentation point; In response to detecting the sentence end segmentation point within the first duration, performing the step of determining the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point; In response to the sentence end segmentation point still not being detected within the first duration, the target segmentation point is determined based on the sentence intra-sentence segmentation point after the historical segmentation point.

6. The method according to claim 1, characterized in that The method further comprises: In response to the semantic segmentation point not being detected within a second time period after the historical segmentation point, the sentence end segmentation point is forcibly determined after the historical segmentation point based on the recognition result.

7. The method according to claim 1, characterized in that The recognized characters include at least two types: text characters and punctuation characters, and the analyzing based on the recognition result and selecting the speech frame as a semantic segmentation point includes: Selecting a text character adjacent to the punctuation character as a target character; Acquire the pronunciation information of the target character mapped by the preset dictionary as the first pronunciation, and acquire the pronunciation information of the pronunciation unit corresponding to the target character pair as the second pronunciation; In response to the first pronunciation successfully matching the second pronunciation, the semantic segmentation point is determined based on the relative position between the target character and the punctuation character and based on the speech frame aligned with the target unit; wherein the target unit is the pronunciation unit corresponding to the target character.

8. The method according to claim 7, characterized in that The determining of the semantic segmentation point based on the speech frame aligned with the target unit comprises at least one of the following: In response to the target character being the text character before the punctuation character, selecting a speech frame aligned with the last target unit as the semantic segmentation point; In response to the target character being the text character after the punctuation character, a speech frame preceding the speech frame aligned with the first target unit is selected as the semantic segmentation point.

9. The method according to claim 7, characterized in that: The method further comprises: In response to the failure of the first pronunciation to match the second pronunciation, continue to wait for a new recognition result to be generated to add the target character to the punctuation character, and return to the step of obtaining the pronunciation information of the target character mapped by the preset dictionary as the first pronunciation, and if the match still fails, skip the punctuation character and select the next punctuation character, and continue to execute the step of selecting the text character adjacent to the punctuation character as the target character.

10. The method according to any one of claims 1 to 9, characterized in that: The speech to be translated is a streaming speech. After the translation is performed based on the feature set to obtain a translated text, the method further includes: Return to the step of obtaining the recognition result of the speech to be translated and iterate until the streaming speech ends.

11. A speech translation device, characterized in that: include: An acquisition module, used to acquire the recognition result of the speech to be translated; wherein the recognition result includes: recognition characters and pronunciation units respectively aligned with the audio features of each speech frame in the speech to be translated; An analysis module, configured to analyze based on the recognition result and select the speech frame as a semantic segmentation point; wherein the semantic segmentation point belongs to any one of the categories of a sentence-end segmentation point and a sentence-inside segmentation point; A translation module is used to respond to the detection of the sentence end segmentation point after the historical segmentation point, determine the target segmentation point based on the latest sentence end segmentation point after the historical segmentation point, and use a semantic segmentation point no later than the historical segmentation point as the starting position of this segmentation and the target segmentation point as the ending position of this segmentation, combine the audio features as a feature set, and translate to obtain a translation text based on the feature set; wherein the historical segmentation point is the semantic segmentation point used as the ending position at the last segmentation.

12. An electronic device, characterized in that: The invention at least comprises a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech translation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech translation method according to any one of claims 1 to 10.

Images