A translation method, device, apparatus and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of voice translation, is applied to a real-time voice translation scene, and relates to a translation method, device and equipment and a storage medium thereof, and comprises the following steps: obtaining real-time voice data to be translated; performing text conversion; inputting the text conversion result into a sentence breaking prediction model to perform sentence breaking prediction; receiving all output sentences; performing text translation on all sentences according to a target language; performing serialization arrangement on the text translation result according to sequence information of all sentences in the text conversion result; and converting the text translation result into target voice data. The translation method can accurately divide each sentence in the real-time voice data to be translated through the sentence breaking prediction model, thereby realizing subsequent translation in the form of a sentence unit, generating target voice data, and making the translated voice more accurate and scientific.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice translation, and is applied to a real-time voice translation scene, and particularly relates to a translation method and device, equipment and a storage medium thereof. BACKGROUND

[0002] Voice cross-language translation generally refers to a translation mode in which an interpreter translates content to another target voice listener without interrupting the speaker. With the development of AI technology, more and more voice conversion technologies are replaced by AI, such as voice-to-text and text-to-speech. Therefore, voice cross-language translation can also be replaced by AI.

[0003] At present, Ping An Group has integrated voice cross-language translation technology in its intelligent customer service system, which can realize multilingual automatic translation function. However, punctuation is very important for voice translation. If the source language speaker does not punctuate properly, even if the voice recognition technology is advanced, the AI translation is difficult to accurately capture the meaning, which easily leads to inaccurate translation results. The existing sentence boundary detection method needs to wait for the entire text to be input before dividing the sentences, and real-time sentence end inference is prone to misjudgment, which leads to slow translation speed and low translation accuracy. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a translation method, device, equipment and storage medium to solve the problem that the existing technology is prone to misjudgment in real-time sentence end inference when performing voice cross-language translation, which leads to slow translation speed and low translation accuracy.

[0005] To solve the above technical problems, the embodiments of the present application provide a translation method, which adopts the following technical solution:

[0006] A translation method includes the following steps:

[0007] Obtain real-time voice data to be translated;

[0008] Convert the real-time voice data into text by using a preset voice-to-text technology to obtain first text data;

[0009] Input the first text data into a pre-trained sentence prediction model to perform sentence prediction and obtain all sentences in the first text data;

[0010] Input the all sentences into a preset text translation model and perform text translation on the all sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to a language after the real-time voice data is translated.

[0011] According to the sequence information of all the sentences in the first text data, the text translation result output by the text translation model is serialized and arranged to obtain second text data;

[0012] The second text data is converted into target voice data by using a preset text-to-speech technology, wherein the target voice data refers to voice data output in the target language.

[0013] Further, before performing the step of inputting the first text data into the pre-trained sentence breaking prediction model to perform sentence breaking prediction, the method further comprises:

[0014] A batch of single complete sentences is obtained;

[0015] The batch of single complete sentences is integrated to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences;

[0016] Each sample in the training samples is serialized and arranged respectively;

[0017] According to the serialization arrangement result, the sequence value corresponding to each character in each sample is marked;

[0018] According to the sequence value, different real-time sentence end score sequences corresponding to each character in each sample are set to obtain corresponding real-time sentence end score sequences;

[0019] Each sample and all real-time sentence end score sequences corresponding to each sample are input into the to-be-trained sentence breaking prediction model as group data to perform model training to obtain a pre-trained sentence breaking prediction model, wherein the sentence breaking prediction model comprises a standard Transformer model.

[0020] Further, the step of setting different real-time sentence end scores for each character in each sample according to the sequence value to obtain all real-time sentence end score sequences corresponding to each sample comprises:

[0021] The total number of characters in the current sample is counted and denoted as N, wherein N is a positive integer;

[0022] The characters in the current sample are input into a preset character score encoder one by one according to the serialization arrangement result;

[0023] Through the score reference dictionary in the character score encoder, one real-time sentence end score sequence is output for each input character, wherein the score reference dictionary sets different real-time sentence end scores corresponding to different sequence values of different characters;

[0024] until the character input in the current sample is completed, obtaining N real-time end-of-sentence scores equal in number to the total number of characters in the current sample;

[0025] taking the N real-time end-of-sentence scores as all real-time end-of-sentence scores corresponding to the current sample.

[0026] Further, the sentence breaking prediction model comprises an end-of-sentence scoring component and a delay determination component, and the step of inputting the first text data into the pre-trained sentence breaking prediction model for sentence breaking prediction specifically comprises:

[0027] Step 301: obtaining a current character input in a character stream mode, wherein the character stream mode refers to inputting characters one by one according to the order of all characters in the first text data;

[0028] Step 302: scoring all input characters by the end-of-sentence scoring component to obtain real-time end-of-sentence scores corresponding to each character respectively, wherein the real-time end-of-sentence scores are dynamically adjusted in real time according to the continuous input of text characters;

[0029] Step 303: determining whether there is a character whose real-time end-of-sentence score exceeds a preset score threshold through real-time comparison;

[0030] Step 304: if there is no character whose real-time end-of-sentence score exceeds the preset score threshold, continuing to perform steps 301 to 303 to reacquire the real-time end-of-sentence scores of the input characters;

[0031] Step 305: if there is a character whose real-time end-of-sentence score exceeds the preset score threshold, marking the character, starting the delay determination component, and continuing to perform steps 301 to 303 until K subsequent input characters are reached, and then acquiring K scoring results corresponding to the marked character, wherein K is a positive integer and is set by the delay determination component;

[0032] Step 306: determining whether the character is an end-of-sentence character according to the K scoring results corresponding to the marked character.

[0033] Further, before the step of scoring all input characters by the end-of-sentence scoring component to obtain real-time end-of-sentence scores corresponding to each character respectively, the method further comprises:

[0034] counting the total number of all characters in the first text data;

[0035] counting the number of all characters input in the character stream mode;

[0036] According to the total number of all characters in the first text data and the number of all characters input in a character stream manner, it is identified whether the current character is the last input character;

[0037] If the current character is the last input character, the current character is directly marked as a sentence ending character;

[0038] If the current character is not the last input character, step 302 is executed.

[0039] Further, before the step of starting the delay determination component is executed, the method further comprises:

[0040] According to the total number of all characters in the first text data and the number of all characters input in a character stream manner, it is identified the number of characters not input after the current character;

[0041] If the number of characters not input after the current character does not exceed a preset character number threshold, the current character is directly marked as a non-sentence ending character;

[0042] If the number of characters not input after the current character exceeds the preset character number threshold, the K value is dynamically adjusted, the delay determination component is started, and steps 301 to 303 are continuously executed until the subsequently input character reaches K, and then the K scoring results corresponding to the marked character are obtained.

[0043] Further, the step of determining whether the character is a sentence ending character according to the K scoring results corresponding to the marked character specifically comprises:

[0044] The K scoring results corresponding to the marked character are obtained;

[0045] According to a preset comprehensive score algorithm:

[0046]

[0047] The sentence ending comprehensive score corresponding to the marked character is obtained, wherein Score represents the sentence ending comprehensive score, score k represents the Kth scoring result in the K scoring results, ω k represents a weight value corresponding to the Kth scoring result in the K scoring results, ω1+ω2+…+ω k-1 +ω k =1, and the weight value gradually increases with the increase of the K value;

[0048] By comparison, it is judged whether the sentence ending comprehensive score exceeds the score threshold;

[0049] If the sentence end comprehensive score exceeds the score threshold, the marked character is determined as a sentence end character;

[0050] If the sentence end comprehensive score does not exceed the score threshold, the marked character is determined as a non-sentence end character.

[0051] To solve the above technical problems, the embodiment of the application further provides a translation device, which adopts the technical scheme as follows:

[0052] A translation device comprises:

[0053] A to-be-translated voice acquisition module is configured to acquire to-be-translated real-time voice data.

[0054] A voice-to-text module is configured to convert the real-time voice data into text by using a preset voice-to-text technology to obtain first text data.

[0055] A sentence breaking prediction module is configured to input the first text data into a pre-trained sentence breaking prediction model to perform sentence breaking prediction to obtain all sentences in the first text data.

[0056] A text translation module is configured to input the all sentences into a preset text translation model and perform text translation on the all sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to a language after translation of the real-time voice data.

[0057] A serialization and arrangement module is configured to serialize and arrange a text translation result output by the text translation model according to sequence information of the all sentences in the first text data to obtain second text data.

[0058] A text-to-voice module is configured to convert the second text data into target voice data by using a preset text-to-voice technology, wherein the target voice data refers to voice data output in the target language.

[0059] To solve the above technical problems, the embodiment of the application further provides a computer device, which adopts the technical scheme as follows:

[0060] A computer device comprises a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the above-mentioned translation method.

[0061] To solve the above technical problems, the embodiment of the application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0062] A computer readable storage medium, having stored thereon computer readable instructions, which when executed by a processor implement the steps of the translation method as described above.

[0063] Compared with the prior art, the embodiments of the application have the following beneficial effects:

[0064] The translation method described in the embodiments of the application acquires real-time speech data to be translated, performs text conversion, inputs the text conversion result into a sentence breaking prediction model to perform sentence breaking prediction, receives all the output sentences, performs text translation on all the sentences according to a target language, serializes and arranges the text translation result according to sequence information of all the sentences in the text conversion result, and converts the text translation result into target speech data. The translation method of the application can accurately segment each sentence in the real-time speech data to be translated through the sentence breaking prediction model, thereby realizing subsequent translation in units of sentences, generating target speech data, and making the translated speech more accurate and scientific. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the schemes in the application, the drawings needed in the description of the embodiments of the application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0066] Figure 1 is an exemplary system architecture diagram in which the application can be applied;

[0067] Figure 2 is a flowchart of one embodiment of the translation method according to the application;

[0068] Figure 3 is a flowchart of one specific embodiment of the step 203 shown in Figure 2

[0069] Figure 4 is a flowchart of one specific embodiment of the step 306 shown in Figure 3

[0070] Figure 5 is a structural schematic diagram of one embodiment of the translation device according to the application;

[0071] Figure 6 is a structural schematic diagram of one embodiment of the computer device according to the application. DETAILED DESCRIPTION

[0072] ​​Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the use herein of terms such as "comprise" and "comprising", "have" and "having", "include" and "including" and "contain" and "containing" are to be construed in an open, non-exhaustive way, i.e. in the sense of "including, but not limited to"; the use herein of terms such as "first", "second", and "other" are used to distinguish one object from another, and are not used to describe a particular sequential order.

[0073] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another.

[0074] In order to make the persons skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings.

[0075] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0076] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0077] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop portable computers, and desktop computers, etc.

[0078] The server 105 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101, 102, 103.

[0079] It should be noted that the translation method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the translation device is generally arranged in a terminal device.

[0080] It should be understood that Figure 1 The number of terminal devices, networks and servers in

[0081] With reference to Figure 2 , a flowchart of one embodiment of the translation method according to the present application is shown. The translation method includes the following steps:

[0082] Step 201, obtaining real-time voice data to be translated.

[0083] In the present embodiment, the real-time voice data to be translated refers to the voice to be translated obtained through a preset voice output device, for example, the real-time voice data to be translated provided by a translation headset or earphone. The translation method can be applied in a large seminar or international conference scenario, or an online video communication across countries / languages, so as to better assist the participants to communicate in real time. The translation method can be applied in the financial or medical field, for example, applied in a large international financial seminar, or a cross-country medical operation scenario, a cross-country medical seminar scenario, of course, it can also be applied to business communication of banks or enterprises, for example, the intelligent customer service translation scenario used by a bank when providing services for foreign students, or the communication in cross-country business of an enterprise. The translation method combines a sentence breaking prediction model, and can output more accurate translation results and target voice.

[0084] Step 202, converting the real-time voice data into text by using a preset voice-to-text technology, to obtain first text data.

[0085] In the present embodiment, from the aspect of concretization, the preset voice-to-text technology can be an online voice-to-text tool implanted in a program, for example, a Whisper voice-to-text tool, or a small program or APP capable of voice-to-text processing, or a built-in function provided by a third-party platform, for example, a voice-to-text function in WeChat. From the bottom technology, the preset voice-to-text technology can directly use an ASR technology. The automatic speech recognition ASR can recognize the real-time voice data to be translated, and then convert it into readable text, i.e., the first text data.

[0086] It should be understood that the first text data obtained in step 202 has not been converted in language, for example: pure Chinese speech data is converted into Chinese text data, or English speech data is converted into English text data. By converting real-time speech data into text, the first text data is obtained, which facilitates subsequent computer program processing of text data.

[0087] In step 203, the first text data is input into the pre-trained sentence breaking prediction model for sentence breaking prediction to obtain all sentences in the first text data.

[0088] By using the pre-trained sentence breaking prediction model to predict the sentence breaking of the first text data, subsequent translation in the target language can be performed sentence by sentence according to the prediction results, ensuring more accurate translation results.

[0089] In the embodiment, before the step of inputting the first text data into the pre-trained sentence breaking prediction model for sentence breaking prediction, the method further comprises: obtaining a batch of single complete sentences; integrating the batch of single complete sentences to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences; respectively serializing each sample in the training samples; according to the serialization result, marking the sequence value corresponding to each character in each sample; setting different real-time sentence tail scores for all characters in each sample according to the sequence value to obtain the corresponding real-time sentence tail score sequence; inputting each sample and all real-time sentence tail score sequences corresponding to each sample as group data into the to-be-trained sentence breaking prediction model for model training to obtain the pre-trained sentence breaking prediction model, wherein the sentence breaking prediction model comprises a standard Transformer model.

[0090] Specifically, since the sentence breaking prediction model needs to be trained, if the input sentence is a single complete sentence, the training of the model has no actual significance. In order to avoid this situation, a batch of single complete sentences are obtained, and then the batch of single complete sentences are integrated to obtain training samples. The integration process can be understood as selecting at least two sentences from the batch of single complete sentences, and then sequentially splicing the sentences to obtain each sample of the training samples, so that all input samples play a role in model training.

[0091] Further, the training samples are respectively serialized; according to the serialization result, the sequence values corresponding to all characters in each sample are marked. Since the subsequent training of the sentence breaking prediction model involves real-time sentence end score scoring of each character, that is, the score of the predicted input character as a sentence end character, in the training preparation stage, different real-time sentence end scores corresponding to different characters in different samples are set according to different sequence values, which can be regarded as marking the training samples, so that the subsequent training of the sentence breaking prediction model is carried out under supervision, and the accuracy of the pre-trained sentence breaking prediction model in actual prediction is ensured.

[0092] In the embodiment, the step of setting different real-time sentence end scores for all characters in each sample according to the sequence value to obtain all real-time sentence end score sequences corresponding to each sample specifically includes: counting the total number of characters in the current sample, denoted as N, wherein N is a positive integer; inputting the characters in the current sample one by one into a preset character score encoder according to the serialization result; through the preset score reference dictionary in the character score encoder, one real-time sentence end score sequence is output for each input character, wherein the score reference dictionary sets different real-time sentence end scores corresponding to different sequence values of different characters; until the input of the characters in the current sample is completed, N real-time sentence end score sequences equal to the total number of characters in the current sample are obtained; and the N real-time sentence end score sequences are taken as all real-time sentence end score sequences corresponding to the current sample.

[0093] Specifically, for example, the current sample contains 2 complete sentences, the number of characters of the first sentence is 10, and the number of characters of the second sentence is 15. After counting, the total number of characters in the current sample is 25. Since the current sample has been serialized and arranged, when the characters in the current sample are input into the preset character score encoder one by one according to the serialization arrangement result, for example, when the first character is input, the real-time sentence end group scoring corresponding to the character is generated in combination with the preset score reference dictionary in the character score encoder, which is taken as the first real-time sentence end score sequence. When the second character is input, the real-time sentence end group scoring corresponding to the second character and the first character is generated in combination with the preset score reference dictionary in the character score encoder, and the sequence composed of the two scores is taken as the second real-time sentence end score sequence. Finally, 25 real-time sentence end score sequences are obtained, that is, N real-time sentence end score sequences equal to the total number of characters in the current sample are obtained. In this way, each sample and all real-time sentence end score sequences corresponding to each sample are input into the to-be-trained sentence breaking prediction model as group data for model training, so that the real-time sentence end score corresponding to each character dynamically changes with the change of the input text sequence length, ensuring that the trained sentence breaking prediction model is more accurate in actual prediction.

[0094] In this embodiment, the trained sentence breaking prediction model can score the real-time sentence end score of each input character, but in the final determination of whether a character is a sentence end character, a delay determination component is also introduced in this embodiment. When the first real-time sentence end score corresponding to a character is obtained, the real-time sentence end score corresponding to the character is obtained multiple times in combination with the delay determination component, so as to finally determine whether the character is a sentence end character, realizing the determination of the sentence end character in combination with the context information and ensuring that the prediction result is more accurate.

[0095] In this embodiment, the sentence breaking prediction model includes a sentence end scoring component and a delay determination component.

[0096] With reference to Figure 3 , Figure 3 is Figure 2 a flowchart of one specific embodiment of step 203, including:

[0097] Step 301: obtaining a current character input in a character stream mode, wherein the character stream mode refers to inputting characters one by one according to the order of all characters in the first text data;

[0098] Specifically, the characters in the first text data are input one by one in a character stream mode, so that the real-time sentence end score of each character is dynamically scored as the characters are input one by one, and the scoring is no longer limited to a character corresponding to a specific score, so that the scoring can fully combine context information, and the real-time sentence end score is more scientific and accurate.

[0099] In step 302, the sentence end scoring component scores all the input characters to obtain a real-time sentence end score corresponding to each character, wherein the real-time sentence end score is dynamically adjusted in real time according to continuous input of text characters.

[0100] In the embodiment, before the step of scoring all the input characters by the sentence end scoring component to obtain a real-time sentence end score corresponding to each character, the method further includes: counting the total number of all characters in the first text data; counting the number of all characters input in a character stream mode; identifying whether the current character is the last input character according to the total number of all characters in the first text data and the number of all characters input in a character stream mode; if the current character is the last input character, directly marking the current character as a sentence end character; and if the current character is not the last input character, performing step 302.

[0101] In essence, it is to handle special cases, that is, to identify that the current character is the last input character, and directly mark the current character as a sentence end character, without real-time sentence end score prediction, which is more scientific.

[0102] In step 303, it is determined by real-time comparison whether there is a character corresponding to a real-time sentence end score exceeding a preset score threshold.

[0103] In step 304, if there is no character corresponding to a real-time sentence end score exceeding a preset score threshold, steps 301 to 303 are continuously performed to reacquire the real-time sentence end score of the input character.

[0104] In step 305, if there is a character corresponding to a real-time sentence end score exceeding a preset score threshold, the character is marked, the delay determination component is started, and steps 301 to 303 are continuously performed until K input characters are obtained, and K times of scoring results corresponding to the marked character are obtained, wherein K is a positive integer and is set by the delay determination component.

[0105] In the embodiment, before the step of starting the delay determination component, the method further comprises: identifying the number of characters not input after the current character according to the total number of all characters in the first text data and the number of all characters input in a character stream; if the number of characters not input after the current character does not exceed a preset character number threshold, directly marking the current character as a non-sentence ending character; if the number of characters not input after the current character exceeds the preset character number threshold, dynamically adjusting the K value, starting the delay determination component, and continuing to perform steps 301 to 303 until K characters are input subsequently, and then obtaining the K scoring results corresponding to the marked character.

[0106] In a special case, for example, the total number of characters contained in the first text data is 28, the number of characters input into the sentence breaking prediction model is 26, and the number of characters not input after the current character is 2. Assuming that the preset character number threshold is 2 or 3, the number of remaining characters does not exceed the preset character number threshold, and it is meaningless to perform real-time sentence ending score scoring, so the current character is directly marked as a non-sentence ending character. However, if the number of characters input into the sentence breaking prediction model is 23, the number of characters not input after the current character is 5, and the K value is dynamically adjusted to be less than 5, then the delay determination component is started again, and steps 301 to 303 are continued to be performed until K characters are input subsequently, and then the K scoring results corresponding to the marked character are obtained. By judging whether the number of remaining characters exceeds the preset character number threshold and dynamically adjusting the K value, the prediction result of the sentence breaking prediction model is more accurate and scientific.

[0107] Step 306, determining whether the character is a sentence ending character according to the K scoring results corresponding to the marked character.

[0108] With reference to Figure 4 , Figure 4 is Figure 3 a flowchart of one specific embodiment of step 306, comprising:

[0109] Step 401, obtaining the K scoring results corresponding to the marked character;

[0110] Step 402, according to a preset comprehensive score algorithm:

[0111]

[0112] obtaining a sentence ending comprehensive score corresponding to the marked character, wherein Score represents the sentence ending comprehensive score, scorek denotes the Kth scoring result in the K scoring results, ω k denotes the weight value corresponding to the Kth scoring result in the K scoring results, ω1+ω2+…+ω k-1 +ω k =1, and the weight value gradually increases with the increase of K value;

[0113] Specifically, as the number of characters obtained by the sentence breaking prediction model increases with the increase of the number of scoring, it can be considered that more context information is obtained, so the weight value gradually increases with the increase of K value, ensuring that the prediction result is more accurate.

[0114] Step 403, by comparison, it is judged whether the sentence tail comprehensive score exceeds the score threshold value;

[0115] Step 404, if the sentence tail comprehensive score exceeds the score threshold value, it is determined that the marked character is a sentence tail character;

[0116] Step 405, if the sentence tail comprehensive score does not exceed the score threshold value, it is determined that the marked character is not a sentence tail character.

[0117] By comprehensive calculation of K scoring results, it is finally determined whether the marked character is a sentence tail character, which ensures that the prediction result of the sentence breaking prediction model is more accurate.

[0118] Finally, all sentences in the first text data output by the sentence breaking prediction model according to all sentence tail characters are received, that is, all sentences in the first text data are obtained.

[0119] Step 204, input the all sentences into a preset text translation model, and perform text translation on the all sentences according to a target language, wherein the target language is deployed in the text translation model as a setting item, and the target language refers to a language after translation of the real-time voice data.

[0120] The sentence-to-sentence translation mode is adopted to translate the text in units of sentences, which can retain more text semantic information than the character or word unit translation, so that the translation result is more accurate.

[0121] In essence, cross-language conversion between texts is realized in step 204, for example: the first text data is Chinese, that is, the sentences are Chinese, and the target language is English, so the text translation result is English, or the first text data is English, that is, the sentences are English, and the target language is Chinese, so the text translation result is Chinese.

[0122] In this embodiment, the text translation model can be set according to actual needs, for example: the first text data is Chinese, and the target language is English, then the text translation model is a Chinese-English translation model, and for the same reason, the first text data is English, and the target language is Chinese, then the text translation model is an English-Chinese translation model. The text translation model includes a text translation model based on natural language processing (NLP), which will not be described in detail here.

[0123] In step 205, the text translation results output by the text translation model are serialized and arranged according to the sequence information of all sentences in the first text data, and the second text data is obtained.

[0124] Specifically, the text translation results output by the text translation model are first received, and the text translation results corresponding to all sentences are serialized and arranged according to the sequence information of all sentences in the first text data, so that the second text data translated subsequently corresponds to the first text data, that is, the output order of the second text data corresponds to the real-time speech to be translated.

[0125] In step 206, a preset text-to-speech technology is used to convert the second text data into target speech data, wherein the target speech data refers to speech data output in the target language.

[0126] In this embodiment, the preset text-to-speech technology can use existing text-to-speech tools or TTS (Text-to-Speech, TTS) technology to input the second text data through an interface and convert the second text data into speech. Specifically, the text-to-speech technology in step 206 does not perform cross-language conversion, for example: in the case of English second text data, it is converted into English speech, or in the case of Chinese second text data, it is converted into Chinese speech.

[0127] By introducing a sentence breaking prediction model and inputting characters in the first text data in a character stream manner, dynamic real-time sentence end score scoring is realized for each character as it is input, which is no longer limited to a character corresponding to only one specific score, ensuring that the scoring can fully combine context information, making the real-time sentence end score more scientific and accurate. At the same time, a delay determination component is introduced to comprehensively calculate the K times scoring results to finally determine whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence breaking prediction model is more accurate, and the final translated target speech data is more accurate.

[0128] The application realizes dynamic real-time sentence end score scoring for each character as the characters are input one by one by using a sentence breaking prediction model and inputting the characters in the first text data in a character stream mode, and no longer only restricts one character to correspond to one specific score, so as to ensure that the scoring can fully combine context information, and make the real-time sentence end score more scientific and accurate. Meanwhile, a delay determination component is introduced, the K times of scoring results are comprehensively calculated, and it is finally determined whether the marked character is a sentence end character, so as to ensure that the prediction result of the sentence breaking prediction model is more accurate, and the finally translated target voice data is more accurate.

[0129] The embodiment of the application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0130] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0131] In the embodiment of the application, the sentence breaking prediction model and the character stream mode are used to input the characters in the first text data one by one, so as to realize dynamic real-time sentence end score scoring for each character as the characters are input one by one, and no longer only restrict one character to correspond to one specific score, so as to ensure that the scoring can fully combine context information, and make the real-time sentence end score more scientific and accurate. Meanwhile, a delay determination component is introduced, the K times of scoring results are comprehensively calculated, and it is finally determined whether the marked character is a sentence end character, so as to ensure that the prediction result of the sentence breaking prediction model is more accurate, and the finally translated target voice data is more accurate.

[0132] Further reference Figure 5 , as an implementation of the method shown in Figure 2 , the application provides an embodiment of a translation device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0133] As Figure 5As shown, the translation device 500 described in this embodiment includes: a to-be-translated voice acquisition module 501, a voice-to-text module 502, a sentence breaking prediction module 503, a text translation module 504, a serialization arrangement module 505, and a text-to-voice module 506. Among them:

[0134] The to-be-translated voice acquisition module 501 is configured to acquire real-time voice data to be translated.

[0135] The voice-to-text module 502 is configured to convert the real-time voice data into text by using a preset voice-to-text technology to obtain first text data.

[0136] The sentence breaking prediction module 503 is configured to input the first text data into a pre-trained sentence breaking prediction model to perform sentence breaking prediction to obtain all sentences in the first text data.

[0137] The text translation module 504 is configured to input the all sentences into a preset text translation model and perform text translation on the all sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to a language after the real-time voice data is translated.

[0138] The serialization arrangement module 505 is configured to serialize and arrange a text translation result output by the text translation model according to sequence information of the all sentences in the first text data to obtain second text data.

[0139] The text-to-voice module 506 is configured to convert the second text data into target voice data by using a preset text-to-voice technology, wherein the target voice data refers to voice data output in the target language.

[0140] The application realizes dynamic real-time sentence end score scoring for each character as the characters are input one by one through the sentence breaking prediction model and the character stream mode, no longer limits a character to only correspond to a specific score, ensures that the scoring can fully combine context information, and makes the real-time sentence end score more scientific and accurate. At the same time, the delay determination component is introduced, the K times scoring results are comprehensively calculated, and it is finally determined whether the marked character is a sentence end character, which further ensures that the prediction result of the sentence breaking prediction model is more accurate, and the finally translated target voice data is more accurate.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through computer readable instructions, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. Among them, the storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0143] To solve the above technical problems, the embodiment of the present application also provides a computer device. For details, please refer to Figure 6 , Figure 6 The basic structure block diagram of the computer device of the present embodiment is shown in the figure.

[0144] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c, which are connected to each other through a system bus. It should be noted that only the computer device 6 with components 6a-6c is shown in the figure, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0145] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0146] The memory 6a includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 6a can also include both an internal storage unit and an external storage device of the computer device 6. In this embodiment, the memory 6a is generally used to store an operating system and various application software installed in the computer device 6, such as computer readable instructions of a translation method, or the like. In addition, the memory 6a can also be used to temporarily store various data that has been output or will be output.

[0147] The processor 6b can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to run computer readable instructions or process data stored in the memory 6a, such as computer readable instructions of the translation method.

[0148] The network interface 6c can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] The computer device provided in the embodiment belongs to the technical field of voice translation and is applied to a real-time voice translation scene. The character stream mode is adopted to input characters in the first text data one by one, and dynamic real-time sentence end score scoring is performed on each character as the characters are input one by one, so that the scoring is no longer limited to one character corresponding to one specific score, the scoring can sufficiently combine context information, the real-time sentence end score is more scientific and accurate, and meanwhile, the delay determination component is introduced, the K times scoring results are comprehensively calculated, and it is finally determined whether the marked character is a sentence end character, so that the prediction result of the sentence breaking prediction model is more accurate, and the finally translated target voice data is more accurate.

[0150] The application also provides another implementation, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by a processor to make the processor execute the steps of the translation method as described above.

[0151] The computer readable storage medium provided in the embodiment belongs to the technical field of voice translation and is applied to a real-time voice translation scene. The character stream mode is adopted to input characters in the first text data one by one, and dynamic real-time sentence end score scoring is performed on each character as the characters are input one by one, so that the scoring is no longer limited to one character corresponding to one specific score, the scoring can sufficiently combine context information, the real-time sentence end score is more scientific and accurate, and meanwhile, the delay determination component is introduced, the K times scoring results are comprehensively calculated, and it is finally determined whether the marked character is a sentence end character, so that the prediction result of the sentence breaking prediction model is more accurate, and the finally translated target voice data is more accurate.

[0152] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiment method can be realized by means of software and a general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the application.

[0153] Obviously, the above-described embodiments are only some embodiments but not all the embodiments of the present application, the preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features therein. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A method of translation, characterized by, The method comprises the following steps: obtaining real-time voice data to be translated; using a preset voice-to-text technology to convert the real-time voice data into text to obtain first text data; inputting the first text data into a pre-trained sentence prediction model to perform sentence prediction to obtain all sentences in the first text data, wherein the sentence prediction model comprises a sentence end scoring component and a delay determination component, and the step of inputting the first text data into the pre-trained sentence prediction model to perform sentence prediction specifically comprises: step 301: obtaining a current character input in a character stream mode, wherein the character stream mode refers to inputting characters one by one according to the order of all characters in the first text data; step 302: scoring all input characters by the sentence end scoring component to obtain real-time sentence end scores corresponding to each character respectively, wherein the real-time sentence end scores are dynamically adjusted in real time according to the continuous input of text characters; step 303: determining whether there is a character whose real-time sentence end score exceeds a preset score threshold through real-time comparison; step 304: if there is no character whose real-time sentence end score exceeds the preset score threshold, re-obtaining the real-time sentence end score of the input character and continuing to perform steps 301 to 303; step 305: if there is a character whose real-time sentence end score exceeds the preset score threshold, marking the character, starting the delay determination component, and continuing to perform steps 301 to 303 until K characters are input subsequently, and then obtaining K scoring results corresponding to the marked character, wherein K is a positive integer and is set by the delay determination component; step 306: determining whether the character is a sentence end character according to the K scoring results corresponding to the marked character; inputting all sentences into a preset text translation model and performing text translation on all sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to the language of the translated real-time voice data; serializing and arranging the text translation result output by the text translation model according to the sequence information of all sentences in the first text data to obtain second text data; using a preset text-to-speech technology to convert the second text data into target voice data, wherein the target voice data refers to voice data output in the target language.

2. The translation method of claim 1, wherein, Before performing the step of inputting the first text data into the pre-trained sentence prediction model to perform sentence prediction, the method further comprises: obtaining a batch of single complete sentences; integrating the batch of single complete sentences to obtain training samples, wherein each sample in the training samples is composed of at least two complete sentences; serializing and arranging each sample in the training samples respectively; according to the serialization and arrangement result, marking the sequence values corresponding to all characters in each sample respectively; Set different real-time sentence ending scores for all characters in each sample according to the sequence values, and obtain a corresponding real-time sentence ending score sequence; Input each sample and all real-time sentence ending score sequences corresponding to each sample as group data into the sentence breaking prediction model to be trained, perform model training, and obtain a pre-trained sentence breaking prediction model, wherein the sentence breaking prediction model comprises a standard Transformer model.

3. The translation method of claim 2, wherein, The step of setting different real-time sentence ending scores for all characters in each sample according to the sequence values, and obtaining all real-time sentence ending score sequences corresponding to each sample, specifically comprises: Count the total number of characters in the current sample, denoted as N, wherein N is a positive integer; Input the characters in the current sample into the preset character score encoder one by one according to the serialization arrangement result; Through the score reference dictionary preset in the character score encoder, output a real-time sentence ending score sequence for each input character, wherein the score reference dictionary sets different real-time sentence ending scores corresponding to different sequence values; Until the input of the characters in the current sample is completed, N real-time sentence ending score sequences equal in number to the total number of characters in the current sample are obtained; The N real-time sentence ending score sequences are taken as all real-time sentence ending score sequences corresponding to the current sample.

4. The translation method of claim 1, wherein, Before the step of performing scoring on all input characters by the sentence ending scoring component to obtain real-time sentence ending scores corresponding to each character, the method further comprises: Count the total number of characters in the first text data; Count the number of all characters input in the character stream mode; According to the total number of characters in the first text data and the number of all characters input in the character stream mode, identify whether the current character is the last input character; If the current character is the last input character, directly mark the current character as a sentence ending character; If the current character is not the last input character, perform step 302.

5. The translation method of claim 1, wherein, Before the step of starting the delay determination component, the method further comprises: According to the total number of characters in the first text data and the number of all characters input in the character stream mode, identify the number of characters not input after the current character; If the number of characters not input after the current character does not exceed the preset character number threshold, directly mark the current character as a non-sentence ending character; If the number of characters not input after the current character exceeds the preset character number threshold, dynamically adjust the K value, start the delay determination component, and continue to perform steps 301 to 303 until the subsequently input characters reach K, and then obtain the K scoring results corresponding to the marked characters.

6. The translation method of claim 1, wherein, The step of determining whether the character is a sentence ending character according to the K scoring results corresponding to the marked characters, specifically comprises: Obtain the K scoring results corresponding to the marked characters; According to the preset comprehensive score algorithm: ， obtaining a sentence-end comprehensive score corresponding to the marked character pair, wherein indicating the sentence-end comprehensive score, indicating the Kth scoring result among the K scoring results, indicating a weight value corresponding to the Kth scoring result among the K scoring results, and the weight value gradually increases with the increase of the value of K. By comparison, determine whether the sentence ending comprehensive score exceeds the score threshold; If the sentence end comprehensive score exceeds the score threshold, the marked character is determined as a sentence end character; If the sentence end comprehensive score does not exceed the score threshold, the marked character is determined as a non-sentence end character.

7. A translation device, characterized by The method comprises the following steps: acquiring real-time speech data to be translated; performing text conversion on the real-time speech data by using a preset speech-to-text technology to obtain first text data; inputting the first text data into a pre-trained sentence breaking prediction model to perform sentence breaking prediction to obtain all sentences in the first text data, wherein the sentence breaking prediction model comprises a sentence end scoring component and a delay determination component, and the step of inputting the first text data into the pre-trained sentence breaking prediction model to perform sentence breaking prediction specifically comprises the following steps: Step 301: acquiring a current character input in a character stream mode, wherein the character stream mode refers to inputting characters one by one according to the order of all characters in the first text data; Step 302: scoring all input characters by using the sentence end scoring component to obtain real-time sentence end scores corresponding to each character respectively, wherein the real-time sentence end scores are dynamically adjusted in real time according to the continuous input of text characters; Step 303: determining whether there is a character whose real-time sentence end score exceeds a preset score threshold by real-time comparison; Step 304: if there is no character whose real-time sentence end score exceeds the preset score threshold, re-acquiring the real-time sentence end scores of the input characters by continuing to perform steps 301 to 303; Step 305: if there is a character whose real-time sentence end score exceeds the preset score threshold, marking the character, starting the delay determination component, and continuing to perform steps 301 to 303 until K input characters are obtained, and then acquiring K scoring results corresponding to the marked character, wherein K is a positive integer and is set by using the delay determination component; Step 306: determining whether the character is a sentence end character according to the K scoring results corresponding to the marked character; inputting all sentences into a preset text translation model and performing text translation on the all sentences according to a target language, wherein the target language is pre-deployed in the text translation model as a setting item, and the target language refers to a language of translated real-time speech data; serializing and arranging the text translation result output by the text translation model according to sequence information of the all sentences in the first text data to obtain second text data; converting the second text data into target speech data by using a preset text-to-speech technology, wherein the target speech data refers to speech data output in the target language.

8. A computer device, comprising: A computer program product comprising a computer readable storage medium having computer readable program instructions stored therein, the computer readable program instructions being executable by a processor to cause execution of the steps of the translation method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer readable storage medium having computer readable program instructions stored therein, the computer readable program instructions being executable by a processor to cause execution of the steps of the translation method of any one of claims 1 to 6.

Images