Translation method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of speech translation, is applied to a real-time speech translation scene, and relates to a translation method, device and equipment and a storage medium thereof. Text conversion is carried out; inputting a textualized conversion result into a sentence segmentation prediction model, and performing sentence segmentation prediction; receiving all output sentences; performing text translation on all sentences according to the target language; according to the sequence information of all sentences in the textualized conversion result, performing serialized arrangement on the text translation result; and converting the text translation result into target voice data. According to the translation method provided by the invention, each sentence in the real-time voice data to be translated can be accurately segmented through the sentence segmentation prediction model, so that subsequent translation by taking the sentence as a unit is realized, the target voice data is generated, and the translated voice is more accurate and scientific.

Description

Technical Field

[0001] The present application relates to the field of speech translation technology, and is applied to real-time speech translation scenarios, and in particular to a translation method, apparatus, device, and storage medium thereof. Background Art

[0002] Voice cross-language translation usually refers to a translation method in which the translator continuously interprets the content to another target voice audience without interrupting the speaker. With the development of AI technology, more and more voice conversion technologies are replaced by AI, such as: voice to text, text to voice, etc. Therefore, voice cross-language translation can also be replaced by AI.

[0003] Currently, Ping An Group has integrated voice cross-language translation technology into its intelligent customer service system, which can realize multi-language automatic translation. However, "sentence segmentation" is very important for voice translation. If the source language speaker does not perform proper sentence segmentation, even if the voice recognition technology is advanced, AI translation will find it difficult to accurately capture the meaning, which can easily lead to inaccurate translation results. The existing sentence boundary detection method needs to wait until the entire text is input before it can divide the sentence, and there is also the problem that real-time sentence end inference is prone to misjudgment, resulting in slow translation speed and low translation accuracy. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a translation method, apparatus, device and storage medium thereof, so as to solve the problem in the prior art that when performing cross-language speech translation, real-time end-of-sentence inference is prone to misjudgment, resulting in slow translation speed and low translation accuracy.

[0005] In order to solve the above technical problems, the present application embodiment provides a translation method, which adopts the following technical solution:

[0006] A translation method comprises the following steps:

[0007] Obtain real-time voice data to be translated;

[0008] Using a preset speech-to-text technology to convert the real-time speech data into text to obtain first text data;

[0009] Inputting the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data;

[0010] Inputting all the sentences into a preset text translation model, and performing text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated;

[0011] According to the sequence information of all the sentences in the first text data, the text translation results output by the text translation model are serialized to obtain second text data;

[0012] The second text data is converted into target voice data using a preset text-to-speech technology, wherein the target voice data refers to voice data output in the target language.

[0013] Furthermore, before executing the step of inputting the first text data into the pre-trained sentence segmentation prediction model to perform sentence segmentation prediction, the method further includes:

[0014] Get batches of single complete sentences;

[0015] Integrate the batch of single complete sentences to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences;

[0016] Sequencing and arranging each sample in the training samples respectively;

[0017] According to the serialization results, mark the sequence values ​​corresponding to all characters in each sample;

[0018] According to the sequence value, different real-time sentence end scores are set for all characters in each sample to obtain a corresponding real-time sentence end score sequence;

[0019] Each sample and all real-time sentence end score sequences corresponding to each sample are input as group data into the sentence segmentation prediction model to be trained, and model training is performed to obtain a pre-trained sentence segmentation prediction model, wherein the sentence segmentation prediction model includes a standard Transformer model.

[0020] Furthermore, the step of setting different real-time sentence-end scores for all characters in each sample according to the sequence value to obtain all real-time sentence-end score sequences corresponding to each sample specifically includes:

[0021] Count the total number of characters in the current sample, denoted as N, where N is a positive integer;

[0022] Input the characters in the current sample into the preset character value encoder one by one according to the serialization result;

[0023] Through the score reference dictionary preset in the character score encoder, each input character corresponds to outputting a real-time sentence end score sequence, wherein the score reference dictionary is set with real-time sentence end scores corresponding to different sequence values ​​when different characters correspond to different characters;

[0024] Until the characters in the current sample are inputted, N real-time sentence end score sequences are obtained, the number of which is equal to the total number of characters in the current sample;

[0025] The N real-time sentence end score sequences are used as all real-time sentence end score sequences corresponding to the current sample.

[0026] Furthermore, the sentence segmentation prediction model includes a sentence end scoring component and a delay determination component, and the step of inputting the first text data into the pre-trained sentence segmentation prediction model to perform sentence segmentation prediction specifically includes:

[0027] Step 301, obtaining a current character input in a character stream manner, wherein the character stream manner refers to inputting characters one by one according to the order of all characters in the first text data;

[0028] Step 302, scoring all input characters through the sentence-end scoring component to obtain a real-time sentence-end score corresponding to each character, wherein the real-time sentence-end score is dynamically adjusted in real time according to the continuous input of text characters;

[0029] Step 303, through real-time comparison, determining whether there is a character whose real-time sentence end score exceeds a preset score threshold;

[0030] Step 304: If the real-time end-of-sentence score corresponding to the non-existing character exceeds the preset score threshold, continue to execute steps 301 to 303 to re-acquire the real-time end-of-sentence score of the input character;

[0031] Step 305: if there is a character whose real-time end-of-sentence score exceeds a preset score threshold, the character is marked, the delay determination component is started, and steps 301 to 303 are continued until the number of characters subsequently input reaches K, and the K scoring results corresponding to the marked character are obtained, where K is a positive integer and is set by the delay determination component;

[0032] Step 306: Determine whether the character is a sentence-ending character based on the K scoring results corresponding to the marked character.

[0033] Furthermore, before executing the step of scoring all input characters by the sentence-end scoring component to obtain the real-time sentence-end score corresponding to each character, the method further includes:

[0034] Counting the total number of all characters in the first text data;

[0035] Count the number of all characters input in character stream mode;

[0036] Identifying whether the current character is the last character input according to the total number of all characters in the first text data and the number of all characters input in a character stream manner;

[0037] If the current character is the last character input, directly mark the current character as the end of the sentence;

[0038] If the current character is not the last character input, step 302 is executed.

[0039] Furthermore, before executing the step of starting the delay determination component, the method further includes:

[0040] Identifying the number of characters that have not been input after the current character according to the total number of all characters in the first text data and the number of all characters input in a character stream manner;

[0041] If the number of characters that have not been input after the current character does not exceed the preset character number threshold, directly marking the current character as not being a sentence-ending character;

[0042] If the number of characters not input after the current character exceeds the preset character number threshold, the K value is dynamically adjusted, the delay determination component is started, and steps 301 to 303 are continued until the number of characters input subsequently reaches K, and the K scoring results corresponding to the marked characters are obtained.

[0043] Furthermore, the step of determining whether the character is a sentence-ending character according to the K scoring results corresponding to the marked character specifically includes:

[0044] Obtain the K scoring results corresponding to the marked characters;

[0045] According to the preset comprehensive score algorithm:

[0046]

[0047] Get the comprehensive score of the end of the sentence corresponding to the marked character, where Score represents the comprehensive score of the end of the sentence, score k represents the Kth scoring result among these K scoring results, ω k Indicates the weight value corresponding to the Kth scoring result among the K scoring results, ω1+ω2+…+ω k-1 +ω k =1, and the weight value gradually increases with the increase of K value;

[0048] By comparison, determining whether the sentence end comprehensive score exceeds the score threshold;

[0049] If the sentence end comprehensive score exceeds the score threshold, determining that the marked character is a sentence end character;

[0050] If the sentence-end comprehensive score does not exceed the score threshold, it is determined that the marked character is not a sentence-end character.

[0051] In order to solve the above technical problems, the embodiment of the present application also provides a translation device, which adopts the following technical solution:

[0052] A translation device, comprising:

[0053] The voice acquisition module to be translated is used to obtain real-time voice data to be translated;

[0054] A speech-to-text module, configured to convert the real-time speech data into text using a preset speech-to-text technology to obtain first text data;

[0055] A sentence segmentation prediction module, used for inputting the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data;

[0056] A text translation module, used for inputting all the sentences into a preset text translation model, and performing text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated;

[0057] A serialization and arrangement module, used for serializing and arranging the text translation results output by the text translation model according to the sequence information of all the sentences in the first text data, so as to obtain second text data;

[0058] The text-to-speech module is used to convert the second text data into target speech data by using a preset text-to-speech technology, wherein the target speech data refers to speech data output in the target language.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0060] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned translation method when executing the computer-readable instructions.

[0061] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0062] A computer-readable storage medium stores computer-readable instructions, which implement the steps of the translation method described above when executed by a processor.

[0063] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0064] The translation method described in the embodiment of the present application obtains real-time speech data to be translated; performs text conversion; inputs the text conversion result into a sentence segmentation prediction model to perform sentence segmentation prediction; receives all output sentences; performs text translation of all sentences according to the target language; serializes and organizes the text translation results according to the sequence information of all sentences in the text conversion result; and converts the text translation results into target speech data. The translation method of the present application can accurately segment each sentence in the real-time speech data to be translated through the sentence segmentation prediction model, thereby realizing subsequent translation based on sentences as units, generating target speech data, and making the translated speech more accurate and scientific. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0067] Figure 2 is a flow chart of an embodiment of a translation method according to the present application;

[0068] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0069] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 306 is shown;

[0070] Figure 5 is a schematic structural diagram of an embodiment of a translation device according to the present application;

[0071] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0073] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0074] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0075] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0076] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0077] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0078] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0079] It should be noted that the translation method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the translation device is generally arranged in the terminal device.

[0080] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0081] Continue to refer Figure 2 , shows a flow chart of an embodiment of a translation method according to the present application. The translation method comprises the following steps:

[0082] Step 201: Acquire real-time speech data to be translated.

[0083] In this embodiment, the real-time voice data to be translated refers to the voice to be translated obtained by a preset voice output device, for example, the real-time voice data to be translated provided by a translation headset or earphone. The translation method is applied to large-scale seminars or international conferences, or cross-national / cross-language online video exchanges, which can better assist participants in real-time communication. The translation method can be applied to the financial or medical fields, for example, in large-scale international financial seminars, or in cross-border medical surgery scenarios, cross-border medical seminars, and of course, in business exchanges of banks or enterprises, for example, in scenarios where banks use intelligent customer service translation when handling business for foreign students, and in communication and exchanges when enterprises conduct cross-border business. The translation method is combined with a sentence segmentation prediction model to output more accurate translation results and target voices.

[0084] Step 202: convert the real-time voice data into text using a preset voice-to-text technology to obtain first text data.

[0085] In this embodiment, the preset speech-to-text technology, from a concrete level, can be an online speech-to-text tool embedded in the program, such as the Whisper speech-to-text tool, or a small program or APP that can perform speech-to-text processing, or a built-in function provided by a third-party platform, such as the speech-to-text function in WeChat. From the underlying technology, the preset speech-to-text technology can directly adopt ASR technology. Automatic speech recognition ASR can identify real-time speech data to be translated, and then convert it into readable text, that is, the first text data.

[0086] It should be understood that the first text data and the speech data to be translated obtained in step 202 have not yet been converted into languages, for example, simply converting Chinese speech data into Chinese text data, or converting English speech data into English text data. By converting the real-time speech data into text, the first text data is obtained, which facilitates the subsequent computer program to directly process the text data.

[0087] Step 203: input the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data.

[0088] The pre-trained sentence segmentation prediction model is used to predict the sentence segmentation of the first text data, so that when the target language is subsequently translated, the sentences corresponding to the prediction results can be translated sentence by sentence, ensuring that the translation result is more accurate.

[0089] In this embodiment, before executing the step of inputting the first text data into the pre-trained sentence segmentation prediction model to perform sentence segmentation prediction, the method also includes: obtaining a batch of single complete sentences; integrating the batch of single complete sentences to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences; serializing each sample in the training samples; marking the sequence values ​​corresponding to all characters in each sample according to the serialization results; setting different real-time sentence end scores for all characters in each sample according to the sequence values ​​to obtain corresponding real-time sentence end score sequences; inputting each sample and all real-time sentence end score sequences corresponding to each sample as group data into the sentence segmentation prediction model to be trained, performing model training, and obtaining a pre-trained sentence segmentation prediction model, wherein the sentence segmentation prediction model includes a standard Transformer model.

[0090] Specifically, since the sentence segmentation prediction model needs to be trained, if the input sentence is a single complete sentence, it will have no practical significance for the training of the model. In order to avoid this situation, batches of single complete sentences are obtained during acquisition, and then the training samples are obtained by integrating the batches of single complete sentences. The integration process can be understood as arbitrarily selecting at least two sentences from the batch of single complete sentences, and then splicing the whole sentences in sequence, so as to obtain each sample of the training samples, so that all input samples play a role in model training.

[0091] Furthermore, each sample in the training samples is serialized and sorted respectively; based on the serialization results, the sequence values ​​corresponding to all the characters in each sample are marked. Since the subsequent training of the sentence segmentation prediction model involves scoring the real-time sentence end score for each character, that is, predicting the score of the input character as the sentence end character, therefore, in the training preparation stage, according to the different sequence values, the real-time sentence end scores corresponding to different characters in different samples are first set in combination with a preset score reference dictionary. This can be regarded as marking the training samples, so that the subsequent training of the sentence segmentation prediction model is carried out under supervision, ensuring the accuracy of the pre-trained sentence segmentation prediction model in actual prediction.

[0092] In this embodiment, the step of setting different real-time sentence-end scores for all characters in each sample according to the sequence value to obtain all real-time sentence-end score sequences corresponding to each sample specifically includes: counting the total number of characters in the current sample, recorded as N, where N is a positive integer; inputting the characters in the current sample one by one into a preset character score encoder according to the serialization result; outputting a real-time sentence-end score sequence for each input character through a score reference dictionary preset in the character score encoder, wherein the score reference dictionary is set with real-time sentence-end scores corresponding to different characters corresponding to different sequence values; until the characters in the current sample are input, N real-time sentence-end score sequences equal in number to the total number of characters in the current sample are obtained; and the N real-time sentence-end score sequences are used as all real-time sentence-end score sequences corresponding to the current sample.

[0093] Specifically, for example, the current sample contains two complete sentences, the number of characters in the first sentence is 10, and the number of characters in the second sentence is 15. After statistics, the total number of characters in the current sample is 25. Since the current sample has been serialized, when the characters in the current sample are input into the preset character score encoder one by one according to the serialization result, for example, when the first character is input, the real-time sentence end group score corresponding to the character is generated in combination with the score reference dictionary preset in the character score encoder, and it is used as the first real-time sentence end score sequence. When the second character is input, the score reference dictionary preset in the character score encoder is used. The dictionary will generate real-time sentence-end grouping scores corresponding to the character and the first character respectively, and the sequence composed of these two scores will be used as the second real-time sentence-end score sequence, and finally 25 real-time sentence-end score sequences will be obtained, that is, N real-time sentence-end score sequences equal to the total number of characters of the current sample will be obtained. In this way, each sample and all real-time sentence-end score sequences corresponding to each sample are input as group data into the sentence segmentation prediction model to be trained, and the model training is performed, so that the real-time sentence-end score corresponding to each character changes dynamically with the change of the length of the input text sequence, thereby ensuring that the trained sentence segmentation prediction model is more accurate in actual prediction.

[0094] In this embodiment, the trained sentence segmentation prediction model can score the real-time end-of-sentence value for each input character. However, in order to finally determine whether a character is an end-of-sentence character, a delayed determination component is also introduced in this embodiment. When the first real-time end-of-sentence score corresponding to a character is obtained, the real-time end-of-sentence score corresponding to the character is obtained multiple times in combination with the delayed determination component, so as to finally determine whether the character is an end-of-sentence character. This realizes the determination of the end-of-sentence character in combination with contextual information, thereby ensuring that the prediction result is more accurate.

[0095] In this embodiment, the sentence segmentation prediction model includes a sentence end scoring component and a delay determination component.

[0096] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0097] Step 301, obtaining a current character input in a character stream manner, wherein the character stream manner refers to inputting characters one by one according to the order of all characters in the first text data;

[0098] Specifically, the characters in the first text data are input one by one in a character stream manner, so that each character is dynamically scored in real time at the end of sentence as the characters are input one by one. It is no longer limited to one character corresponding to only one specific score, which ensures that the scoring can fully combine the context information, making the real-time end of sentence score more scientific and accurate.

[0099] Step 302, scoring all input characters through the sentence-end scoring component to obtain a real-time sentence-end score corresponding to each character, wherein the real-time sentence-end score is dynamically adjusted in real time according to the continuous input of text characters;

[0100] In this embodiment, before executing the step of scoring all input characters through the sentence-end scoring component to obtain the real-time sentence-end score corresponding to each character, the method also includes: counting the total number of all characters in the first text data; counting the number of all characters input in the character stream manner; identifying whether the current character is the last character input based on the total number of all characters in the first text data and the number of all characters input in the character stream manner; if the current character is the last character input, directly marking the current character as the sentence-end character; if the current character is not the last character input, executing step 302.

[0101] In essence, it is to deal with special cases, that is, if the current character is recognized as the last character input, the current character is directly marked as the sentence end character, and no real-time sentence end score prediction is performed, which is more scientific.

[0102] Step 303, through real-time comparison, determining whether there is a character whose real-time sentence end score exceeds a preset score threshold;

[0103] Step 304: If the real-time end-of-sentence score corresponding to the non-existing character exceeds the preset score threshold, continue to execute steps 301 to 303 to re-acquire the real-time end-of-sentence score of the input character;

[0104] Step 305: if there is a character whose real-time end-of-sentence score exceeds a preset score threshold, the character is marked, the delay determination component is started, and steps 301 to 303 are continued until the number of characters subsequently input reaches K, and the K scoring results corresponding to the marked character are obtained, where K is a positive integer and is set by the delay determination component;

[0105] In this embodiment, before executing the step of starting the delay determination component, the method also includes: identifying the number of characters that have not been entered after the current character based on the total number of all characters in the first text data and the number of all characters entered in a character stream manner; if the number of characters that have not been entered after the current character does not exceed a preset character number threshold, directly marking the current character as not being a sentence-ending character; if the number of characters that have not been entered after the current character exceeds a preset character number threshold, dynamically adjusting the K value, starting the delay determination component, and continuing to execute steps 301 to 303 until the number of characters subsequently entered reaches K, then obtaining the K scoring results corresponding to the marked characters.

[0106] In a special case, for example, the total number of characters contained in the first text data is 28, and the number of characters input into the sentence segmentation prediction model is 26. Then, the number of characters that have not been input after the current character is 2. Assuming that the preset character number threshold is 2 or 3, the remaining number of characters does not exceed the preset character number threshold, and it is meaningless to score the real-time end-of-sentence value for it. Therefore, no scoring prediction is performed, and the current character is directly marked as not a sentence-end character; however, if the number of characters that have been input into the sentence segmentation prediction model is 23, the number of characters that have not been input after the current character is 5. Then, the K value is dynamically adjusted so that the K value is less than 5. After that, the delay determination component is started again, and steps 301 to 303 are continued until the subsequent input characters reach the latest K. Then, the K scoring results corresponding to the marked characters are obtained. By judging whether the remaining number of characters exceeds the preset character number threshold and dynamically adjusting the K value, it is ensured that the prediction result of the sentence segmentation prediction model is more accurate and scientific.

[0107] Step 306: Determine whether the character is a sentence-ending character based on the K scoring results corresponding to the marked character.

[0108] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 306 shown includes:

[0109] Step 401, obtaining the K scoring results corresponding to the marked characters;

[0110] Step 402, according to the preset comprehensive score algorithm:

[0111]

[0112] Get the comprehensive score of the end of the sentence corresponding to the marked character, where Score represents the comprehensive score of the end of the sentence, scorek represents the Kth scoring result among these K scoring results, ω k Indicates the weight value corresponding to the Kth scoring result among the K scoring results, ω1+ω2+…+ω k-1 +ω k =1, and the weight value gradually increases with the increase of K value;

[0113] Specifically, as the number of scoring times increases, the number of characters obtained by the sentence segmentation prediction model increases. Therefore, it can be considered that more context information is obtained. Therefore, the weight value gradually increases with the increase of K value, ensuring that the prediction result is more accurate.

[0114] Step 403, through comparison, determining whether the sentence end comprehensive score exceeds the score threshold;

[0115] Step 404: if the sentence end comprehensive score exceeds the score threshold, determine that the marked character is a sentence end character;

[0116] Step 405: If the sentence-end comprehensive score does not exceed the score threshold, it is determined that the marked character is not a sentence-end character.

[0117] By integrating K scoring results and performing comprehensive calculations, it is finally determined whether the marked character is the end-of-sentence character, ensuring that the prediction results of the sentence segmentation prediction model are more accurate.

[0118] Finally, all the sentences in the first text data output by the sentence segmentation prediction model according to all the sentence end characters are received, that is, all the sentences in the first text data are obtained.

[0119] Step 204, input all the sentences into a preset text translation model, and perform text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated.

[0120] Using a sentence-by-sentence translation method, translating based on sentences as units, can retain more text semantic information and make the translation results more accurate compared to translating based on characters or words as units.

[0121] In essence, cross-language conversion between texts is implemented in step 204, for example: the first text data is in Chinese, that is, the sentence is in Chinese, and the target language is English, then the text translation result is English, or the first text data is in English, that is, the sentence is in English, and the target language is Chinese, then the text translation result is Chinese.

[0122] In this embodiment, the text translation model can be set according to actual needs. For example, if the first text data is in Chinese and the target language is English, the text translation model is a Chinese-to-English model. Similarly, if the first text data is in English and the target language is Chinese, the text translation model is an English-to-Chinese model. The text translation model includes a text translation model based on natural language processing (NLP), which will not be described in detail here.

[0123] Step 205: serialize and organize the text translation results output by the text translation model according to the sequence information of all the sentences in the first text data to obtain second text data.

[0124] Specifically, the text translation results output by the text translation model are first received, and the text translation results corresponding to all the sentences are serialized and sorted according to the sequence information of all the sentences in the first text data, thereby ensuring that the subsequently translated second text data corresponds to the first text data, that is, the second text data corresponds to the output order of the real-time speech to be translated.

[0125] Step 206: Use a preset text-to-speech technology to convert the second text data into target voice data, wherein the target voice data refers to voice data output in the target language.

[0126] In this embodiment, the preset text-to-speech technology can adopt an existing text-to-speech tool or a TTS (Text-to-Speech, TTS) technology, and input the second text data through an interface to convert the second text data into speech. Specifically, the text-to-speech technology in step 206 does not perform cross-language conversion, for example: when the second text data is in English, it is converted into English speech, or when the second text data is in Chinese, it is converted into Chinese speech.

[0127] By introducing a sentence segmentation prediction model and inputting the characters in the first text data one by one in a character stream manner, it is possible to dynamically score each character in real time at the end of a sentence as the characters are input one by one, which is no longer limited to one character corresponding to only one specific score, ensuring that the scoring can fully combine context information, making the real-time end of sentence score more scientific and accurate. At the same time, a delay determination component is introduced to perform a comprehensive calculation by integrating K scoring results, and finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, making the target speech data finally translated more accurate.

[0128] The present application uses a sentence segmentation prediction model and a character stream method to input the characters in the first text data one by one, so as to achieve dynamic real-time sentence end scoring for each character as the characters are input one by one, and is no longer limited to one character corresponding to only one specific score, ensuring that the scoring can fully combine context information, making the real-time sentence end score more scientific and accurate. At the same time, a delay determination component is introduced, and a comprehensive calculation is performed by integrating K scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, making the target speech data finally translated more accurate.

[0129] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0130] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0131] In the embodiment of the present application, by using the sentence segmentation prediction model and inputting the characters in the first text data one by one in a character stream manner, it is realized that each character is dynamically scored in real time at the end of the sentence as the characters are input one by one, and it is no longer limited to one character corresponding to only one specific score, which ensures that the scoring can fully combine the context information, making the real-time end of sentence score more scientific and accurate. At the same time, a delay determination component is introduced, and a comprehensive calculation is performed by integrating K scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, making the target speech data finally translated more accurate.

[0132] Further references Figure 5 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a translation device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0133] like Figure 5As shown, the translation device 500 described in this embodiment includes: a speech acquisition module 501 to be translated, a speech-to-text module 502, a sentence segmentation prediction module 503, a text translation module 504, a serialization and arrangement module 505, and a text-to-speech module 506. Among them:

[0134] The speech acquisition module 501 to be translated is used to acquire real-time speech data to be translated;

[0135] The speech-to-text module 502 is used to convert the real-time speech data into text using a preset speech-to-text technology to obtain first text data;

[0136] A sentence segmentation prediction module 503 is used to input the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data;

[0137] A text translation module 504, configured to input all the sentences into a preset text translation model, and perform text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated;

[0138] A serialization and arrangement module 505 is used to serialize and arrange the text translation results output by the text translation model according to the sequence information of all the sentences in the first text data to obtain second text data;

[0139] The text-to-speech module 506 is configured to convert the second text data into target speech data using a preset text-to-speech technology, wherein the target speech data refers to speech data output in the target language.

[0140] The present application uses a sentence segmentation prediction model and a character stream method to input the characters in the first text data one by one, so as to achieve dynamic real-time sentence end scoring for each character as the characters are input one by one, and is no longer limited to one character corresponding to only one specific score, ensuring that the scoring can fully combine context information, making the real-time sentence end score more scientific and accurate. At the same time, a delay determination component is introduced, and a comprehensive calculation is performed by integrating K scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, making the target speech data finally translated more accurate.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0143] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0144] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with components 6a-6c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field -Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0145] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0146] The memory 6a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 6a can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 6a is usually used to store the operating system and various application software installed on the computer device 6, such as a computer-readable instruction of a translation method, etc. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or are to be output.

[0147] The processor 6b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to run the computer-readable instructions stored in the memory 6a or process data, such as running the computer-readable instructions of the translation method.

[0148] The network interface 6c may include a wireless network interface or a wired network interface. The network interface 6c is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] The computer device proposed in this embodiment belongs to the field of speech translation technology and is applied to real-time speech translation scenarios. This application uses a sentence segmentation prediction model and a character stream method to input the characters in the first text data one by one, so as to achieve dynamic real-time sentence end scoring for each character as the characters are input one by one. It is no longer limited to a character corresponding to only one specific score, which ensures that the scoring can fully combine the context information, making the real-time sentence end score more scientific and accurate. At the same time, a delay determination component is introduced, and a comprehensive calculation is performed by integrating K scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, so that the target speech data finally translated is more accurate.

[0150] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the translation method as described above.

[0151] The computer-readable storage medium proposed in this embodiment belongs to the field of speech translation technology and is applied to real-time speech translation scenarios. This application uses a sentence segmentation prediction model and a character stream method to input the characters in the first text data one by one, so as to achieve dynamic real-time sentence end scoring for each character as the characters are input one by one. It is no longer limited to a character corresponding to only one specific score, which ensures that the scoring can fully combine the context information, making the real-time sentence end score more scientific and accurate. At the same time, a delay determination component is introduced, and a comprehensive calculation is performed by integrating K scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence segmentation prediction model is more accurate, so that the target speech data finally translated is more accurate.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0153] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A translation method, characterized in that: The steps include: Obtain real-time voice data to be translated; Using a preset speech-to-text technology to convert the real-time speech data into text to obtain first text data; Inputting the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data; Inputting all the sentences into a preset text translation model, and performing text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated; According to the sequence information of all the sentences in the first text data, the text translation results output by the text translation model are serialized to obtain second text data; The second text data is converted into target voice data using a preset text-to-speech technology, wherein the target voice data refers to voice data output in the target language.

2. The translation method according to claim 1, characterized in that: Before executing the step of inputting the first text data into the pre-trained sentence segmentation prediction model to perform sentence segmentation prediction, the method further includes: Get batches of single complete sentences; Integrate the batch of single complete sentences to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences; Sequencing and arranging each sample in the training samples respectively; According to the serialization results, mark the sequence values ​​corresponding to all characters in each sample; According to the sequence value, different real-time sentence end scores are set for all characters in each sample to obtain a corresponding real-time sentence end score sequence; Each sample and all real-time sentence end score sequences corresponding to each sample are input as group data into the sentence segmentation prediction model to be trained, and model training is performed to obtain a pre-trained sentence segmentation prediction model, wherein the sentence segmentation prediction model includes a standard Transformer model.

3. The translation method according to claim 2, characterized in that: The step of setting different real-time sentence-end scores for all characters in each sample according to the sequence value to obtain all real-time sentence-end score sequences corresponding to each sample specifically includes: Count the total number of characters in the current sample, denoted as N, where N is a positive integer; Input the characters in the current sample into the preset character value encoder one by one according to the serialization result; Through the score reference dictionary preset in the character score encoder, each input character corresponds to outputting a real-time sentence end score sequence, wherein the score reference dictionary is set with real-time sentence end scores corresponding to different sequence values ​​when different characters correspond to different characters; Until the characters in the current sample are inputted, N real-time sentence end score sequences are obtained, the number of which is equal to the total number of characters in the current sample; The N real-time sentence end score sequences are used as all real-time sentence end score sequences corresponding to the current sample.

4. The translation method according to claim 1, characterized in that: The sentence segmentation prediction model includes a sentence end scoring component and a delay determination component. The step of inputting the first text data into the pre-trained sentence segmentation prediction model to perform sentence segmentation prediction specifically includes: Step 301, obtaining a current character input in a character stream manner, wherein the character stream manner refers to inputting characters one by one according to the order of all characters in the first text data; Step 302, scoring all input characters through the sentence-end scoring component to obtain a real-time sentence-end score corresponding to each character, wherein the real-time sentence-end score is dynamically adjusted in real time according to the continuous input of text characters; Step 303, through real-time comparison, determining whether there is a character corresponding to a real-time sentence end score exceeding a preset score threshold; Step 304: If the real-time end-of-sentence score corresponding to the non-existing character exceeds the preset score threshold, continue to execute steps 301 to 303 to re-acquire the real-time end-of-sentence score of the input character; Step 305: if there is a character whose real-time end-of-sentence score exceeds a preset score threshold, the character is marked, the delay determination component is started, and steps 301 to 303 are continued until the number of characters subsequently input reaches K, and the K scoring results corresponding to the marked character are obtained, where K is a positive integer and is set by the delay determination component; Step 306: Determine whether the character is a sentence-ending character based on the K scoring results corresponding to the marked character.

5. The translation method according to claim 4, characterized in that: Before executing the step of scoring all input characters by the sentence-end scoring component to obtain a real-time sentence-end score corresponding to each character, the method further includes: Counting the total number of all characters in the first text data; Count the number of all characters input in character stream mode; Identifying whether the current character is the last character input according to the total number of all characters in the first text data and the number of all characters input in a character stream manner; If the current character is the last character input, directly mark the current character as the end of the sentence; If the current character is not the last character input, step 302 is executed.

6. The translation method according to claim 4, characterized in that: Before executing the step of starting the delay determination component, the method further includes: Identifying the number of characters that have not been input after the current character according to the total number of all characters in the first text data and the number of all characters input in a character stream manner; If the number of characters that have not been input after the current character does not exceed the preset character number threshold, directly marking the current character as not being a sentence-ending character; If the number of characters not input after the current character exceeds the preset character number threshold, the K value is dynamically adjusted, the delay determination component is started, and steps 301 to 303 are continued until the number of characters input subsequently reaches K, and the K scoring results corresponding to the marked characters are obtained.

7. The translation method according to claim 4, characterized in that: The step of determining whether the character is a sentence-ending character according to the K scoring results corresponding to the marked character specifically includes: Obtain the K scoring results corresponding to the marked characters; According to the preset comprehensive score algorithm: Get the comprehensive score of the end of the sentence corresponding to the marked character, where Score represents the comprehensive score of the end of the sentence, score k represents the Kth scoring result among these K scoring results, ω k Indicates the weight value corresponding to the Kth scoring result among the K scoring results, ω1+ω2+…+ω k-1 +ω k =1, and the weight value gradually increases with the increase of K value; By comparison, determining whether the sentence end comprehensive score exceeds the score threshold; If the sentence end comprehensive score exceeds the score threshold, determining that the marked character is a sentence end character; If the sentence-end comprehensive score does not exceed the score threshold, it is determined that the marked character is not a sentence-end character.

8. A translation device, characterized in that: include: The voice acquisition module to be translated is used to obtain real-time voice data to be translated; A speech-to-text module, configured to convert the real-time speech data into text using a preset speech-to-text technology to obtain first text data; A sentence segmentation prediction module, used for inputting the first text data into a pre-trained sentence segmentation prediction model to perform sentence segmentation prediction to obtain all sentences in the first text data; A text translation module, used for inputting all the sentences into a preset text translation model, and performing text translation on all the sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language after the real-time voice data is translated; A serialization and arrangement module, used for serializing and arranging the text translation results output by the text translation model according to the sequence information of all the sentences in the first text data, so as to obtain second text data; The text-to-speech module is used to convert the second text data into target speech data by using a preset text-to-speech technology, wherein the target speech data refers to speech data output in the target language.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the translation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the translation method according to any one of claims 1 to 7.

Images