Speech translation methods, devices, and electronic equipment

Abstract

This application provides a speech translation method, including: acquiring the original text and sentence segment timestamps of an original audio source; translating the original text of the original audio source into target language text; synthesizing the target language text into target audio; and comparing the sentence segment timestamps of the target audio source and the original audio source to perform a secondary translation of the original text of the original audio source, so that the target audio source and the original audio source have consistent timing. The speech translation method provided by this application can effectively improve the matching and coordination between target audio and original audio. This application also provides a speech translation device and an electronic device.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and more specifically, to a speech translation method, apparatus, and electronic device. Background Technology

[0002] With increasing globalization, cross-border exchanges and cooperation are becoming more frequent, and video has become a primary way for people to communicate and obtain information. However, due to language barriers, people often encounter difficulties in understanding foreign language videos. Previously, this problem required professional voice actors to manually translate and dub the original video content, which was costly and inefficient.

[0003] In recent years, with the continuous breakthroughs in AI technologies such as machine translation, STT, and TTS, automated multilingual dubbing has been achieved by performing text recognition, text translation, and finally synthesizing audio from the text. However, the dubbing result differs greatly in rhythm from the original audio. For example, the person in the video has finished speaking, but the automated dubbing result is still speaking, or the person in the video is speaking, but the automated dubbing result has already finished speaking. This leads to extreme incoordination between the voice and the picture in the video. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of this application provide a speech translation method, apparatus, and electronic device that can effectively improve the matching and coordination between target audio and original audio.

[0005] In a first aspect, embodiments of this application provide a speech translation method, including:

[0006] Obtain the raw text and sentence timestamps of the original audio;

[0007] Translate the original text of the original audio into the target language text;

[0008] Synthesize the target language text into target audio; and

[0009] The sentence break timestamps of the target audio and the original audio are compared, and the original text of the original audio is translated a second time to make the timing of the target audio consistent with that of the original audio.

[0010] Further, translating the original text of the original audio into target language text includes:

[0011] The original text of the original audio is translated into the target language text using a large text model.

[0012] Furthermore, obtaining the original text and sentence-segment timestamps of the original audio includes:

[0013] The original text and sentence timestamps of the original audio are obtained using the STT algorithm.

[0014] Further, the step of synthesizing the target language text into target audio includes:

[0015] The target language text is synthesized into the target audio using a TTS algorithm.

[0016] Furthermore, the step of comparing the sentence break timestamps of the target audio and the original audio to perform a secondary translation of the original text of the original audio includes:

[0017] Compare the sentence break timestamps of the target audio and the original audio, and based on the text lengths of the target audio and the original audio, perform a secondary translation of the original text of the original audio by setting a secondary translation prompt program.

[0018] Furthermore, after comparing the sentence break timestamps of the target audio and the original audio, and performing a secondary translation of the original text of the original audio, the process further includes:

[0019] The target language text, after secondary translation, is synthesized into the target audio.

[0020] Furthermore, the target language text after secondary translation is synthesized into target audio using a TTS algorithm.

[0021] Secondly, embodiments of this application also provide a voice translation device, including:

[0022] The text acquisition module is used to acquire the raw text and sentence timestamps of the original audio.

[0023] The text translation module is used to translate the original text of the original audio into text in the target language;

[0024] An audio synthesis module is used to synthesize the target language text into target audio; and

[0025] The secondary translation module is used to compare the sentence break timestamps of the target audio and the original audio, and to perform a secondary translation of the original text of the original audio so that the timing of the target audio and the original audio are consistent.

[0026] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the program to implement the speech translation method according to the first aspect described above.

[0027] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, the computer program being used to implement the speech translation method according to the first aspect described above.

[0028] Fifthly, embodiments of this application also provide a computer program product having a computer program stored thereon, the computer program being used to implement the speech translation method according to the first aspect described above.

[0029] The embodiments of this application bring the following beneficial effects:

[0030] The speech translation method provided in this application, after obtaining the original text and sentence timestamps of the original audio, translates the original text of the original audio into target language text, and synthesizes the target language text into target audio. Furthermore, by comparing the sentence timestamps of the target audio and the original audio, a second translation is performed on the original text of the original audio to ensure that the timing of the target audio matches that of the original audio. This ensures the rhythm and timing matching between the target audio generated from the translated target text and the original audio, thereby prioritizing the improvement of the matching and coordination between speech and visuals. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0032] Figure 1 A schematic flowchart illustrating a speech translation method provided in an embodiment of this application;

[0033] Figure 2 A structural block diagram of the speech translation device provided in the embodiments of this application;

[0034] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0035] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0037] In the specification, claims, and accompanying drawings of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0038] Figure 1 This is a flowchart of a speech translation method according to an embodiment of this application. Figure 1 As shown, the speech translation method of this application includes the following steps:

[0039] S101: Obtain the original text and sentence timestamps of the original audio;

[0040] Specifically, audio usually refers to files that store sound content, which includes human voices and natural sounds. Here, raw audio specifically refers to audio files containing human speech. When processing video files, they need to be separated into audio files before processing.

[0041] After obtaining the original audio, speech recognition is required to obtain the original text file, and the sentence-breaking timestamps of the original audio are also obtained.

[0042] S102: Translate the original text of the original audio into the target language text;

[0043] Specifically, once the original text of the audio is obtained, it needs to be translated into the target language text. Translation methods include human translation or intelligent machine translation, and the target language includes various languages ​​or other languages.

[0044] S103: Synthesize the target language text into target audio;

[0045] In other words, after translating the original text of the original audio into the target language text, it is necessary to use track splitting operations to synthesize the target language text into the target audio. However, the target audio may have inconsistent timing with the original audio.

[0046] S104: Compare the sentence break timestamps of the target audio and the original audio, and perform a secondary translation of the original text of the original audio so that the timing of the target audio and the original audio are consistent.

[0047] Specifically, by comparing the length of the original text of the original audio with the length of the text of the target audio, which is the sentence break timestamp of the two, a secondary translation of the original text of the original audio is performed, so that the text lengths of the two correspond, that is, the timing of the target audio and the original audio are consistent.

[0048] In practical applications, for example, when the target audio is the dubbing audio of the original audio, the speech translation method provided in this application can solve the problem of matching and coordinating the dubbing audio with the video.

[0049] Therefore, the speech translation method provided in this application, after obtaining the original text and punctuation timestamps of the original audio, translates the original text of the original audio into target language text, synthesizes the target language text into target audio, and performs a second translation of the original text of the original audio by comparing the punctuation timestamps of the target audio and the original audio, so that the timing of the target audio and the original audio are consistent, thereby ensuring the rhythm and timing matching of the target audio generated from the translated target text with the original audio, and can prioritize improving the matching and coordination of speech and image.

[0050] Furthermore, in some embodiments of this application, translating the original text of the original audio into target language text includes:

[0051] The original text of the original audio is translated into the target language text using a large text model.

[0052] Specifically, large-scale text models, also known as AI large-scale text models, refer to powerful text understanding and generation models built based on deep learning and natural language processing technologies. By training on large-scale text datasets, they can automatically learn and understand the semantics, contextual information, and grammatical structure of text. AI large-scale text models possess high accuracy and intelligence, enabling tasks such as text classification, sentiment analysis, and machine translation.

[0053] This approach utilizes large text models, such as GPT and Wenxin Yiyan. By setting a translation prompt, the large text model acts as a multilingual text translator, translating the original text into the target language. Compared to human and machine translation, large text models offer higher accuracy and better naturalness, thus ensuring the quality of the original audio-to-text translation.

[0054] Furthermore, in some embodiments of this application, obtaining the original text and sentence-segment timestamps of the original audio includes:

[0055] The original text and sentence timestamps of the original audio are obtained using the STT algorithm.

[0056] Specifically, STT (Speech to Text) is a technology that converts speech signals into text. Its basic principle is to convert speech signals into a series of audio features, and then use machine learning algorithms to map these features into text. Compared with other natural language generation technologies, STT natural language generation technology has the following advantages: (1) Simple implementation: It is implemented using programming languages ​​such as Python and does not require special hardware devices; (2) High accuracy: It can achieve high-precision text generation by training with large-scale text data; (3) Strong customizability: Different algorithms and parameters can be flexibly selected and customized according to different application scenarios and needs; (4) Support for multiple generation methods: It supports multiple text generation methods, such as rule-based generation, model-based generation, and generative model-based generation.

[0057] Therefore, the speech translation method provided in this application, which obtains the original text and sentence timestamps of the original audio through the STT algorithm, can improve the accuracy and effectiveness of the operation.

[0058] It should be noted that the STT algorithm here can be a mainstream open-source technology such as Whisper or wav2vec, or a mature commercial interface such as those from Microsoft or Alibaba.

[0059] Furthermore, in some embodiments of this application, the step of synthesizing the target language text into target audio includes:

[0060] The target language text is synthesized into the target audio using a TTS algorithm.

[0061] Specifically, TTS (Text to Speech) algorithm belongs to speech synthesis technology, which can convert text information generated by the computer itself or input from the outside into fluent spoken output that can be understood. The speech translation method provided in this application uses TTS algorithm to synthesize the target language text into the target audio, which is highly accurate and efficient.

[0062] It should be noted that the TTS algorithm used here can be open source such as CoQuiTTS or OpenTTS, or it can be a commercial TTS interface such as Microsoft's.

[0063] Furthermore, in some embodiments of this application, the step of comparing the sentence break timestamps of the target audio and the original audio to perform a secondary translation of the original text of the original audio includes:

[0064] Compare the sentence break timestamps of the target audio and the original audio, and based on the text lengths of the target audio and the original audio, perform a secondary translation of the original text of the original audio by setting a secondary translation prompt program.

[0065] Specifically, a prompt typically refers to an input text paragraph or phrase that serves as the starting point or guide for generating the model's output. A prompt can be a question, a text description, a dialogue, or any form of text input. The model will generate the corresponding output text based on the context and semantic information provided by the prompt.

[0066] The speech translation method provided in this application compares the sentence break timestamps of the target audio and the original audio. Based on the text length of the target audio and the original audio, a secondary translation prompt program is set to perform a secondary translation of the original text of the original audio, so that the timing of the target audio and the original audio are consistent. This ensures the matching of the rhythm timing of the target audio generated from the translated target text with the original audio, and can prioritize improving the matching and coordination of speech and image.

[0067] Furthermore, in some embodiments of this application, after comparing the sentence break timestamps of the target audio and the original audio, and performing a secondary translation of the original text of the original audio, the method further includes:

[0068] The target language text, after secondary translation, is synthesized into the target audio.

[0069] Specifically, after the target language text has been translated a second time, it needs to be synthesized into target audio to ensure that the target audio matches the original audio in terms of timing. This ensures that the target audio generated from the translated target text matches the rhythm of the original audio, thus prioritizing the improvement of the matching and coordination between speech and visuals.

[0070] Furthermore, in some embodiments of this application, the target language text after secondary translation is synthesized into target audio using a TTS algorithm.

[0071] Specifically, TTS (Text to Speech) algorithm belongs to speech synthesis technology, which can convert text information generated by the computer itself or input from the outside into fluent spoken output that can be understood. The speech translation method provided in this application uses TTS algorithm to synthesize the target language text into the target audio, which is highly accurate and efficient.

[0072] Figure 2 This is a structural block diagram of the voice translation device 200 according to an embodiment of this application. Figure 2 As shown, the speech translation device 200 of this application embodiment includes: a text acquisition module 210, a text translation module 220, an audio synthesis module 230, and a secondary translation module 240, wherein:

[0073] Text acquisition module 210 is used to load the browser's page to be rendered;

[0074] The text translation module 220 is used to execute a pre-set global variable function to perform rendering operations on the page to be rendered;

[0075] Audio synthesis module 230 is used to take screenshots of the page after rendering; and

[0076] The secondary translation module 240 is used to compare the sentence break timestamps of the target audio and the original audio, and to perform secondary translation on the original text of the original audio so that the timing of the target audio and the original audio are consistent.

[0077] The speech translation device provided in this application, after acquiring the original text and punctuation timestamps of the original audio, translates the original text of the original audio into target language text, and synthesizes the target language text into target audio. Furthermore, by comparing the punctuation timestamps of the target audio and the original audio, a secondary translation is performed on the original text of the original audio to ensure that the timing of the target audio matches that of the original audio. This ensures the rhythm and timing matching between the target audio generated from the translated target text and the original audio, thereby prioritizing the improvement of the matching and coordination between speech and visuals.

[0078] It should be noted that the specific implementation of the speech translation device in this application embodiment is similar to the specific implementation of the speech translation method in this application embodiment. Please refer to the description in the method section for details, which will not be repeated here.

[0079] Figure 3 This is a schematic diagram of the structure of the electronic device 300 according to an embodiment of this application.

[0080] like Figure 3As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage section 302 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0081] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0082] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the electronic device of this application.

[0083] It should be noted that the computer-readable medium shown in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electronic device, apparatus, or device that is electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0084] In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used or combined with an electronic device, apparatus, or device by instructions. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use or combined with an electronic device, apparatus, or device by instructions. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of processing and receiving devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based electronic device that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor, which executes the program to implement the speech translation method.

[0087] Obtain the raw text and sentence timestamps of the original audio;

[0088] Translate the original text of the original audio into the target language text;

[0089] Synthesize the target language text into target audio; and

[0090] The sentence break timestamps of the target audio and the original audio are compared, and the original text of the original audio is translated a second time to make the timing of the target audio consistent with that of the original audio.

[0091] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs, which, when used by one or more processors, execute the speech translation method described in this application:

[0092] Obtain the raw text and sentence timestamps of the original audio;

[0093] Translate the original text of the original audio into the target language text;

[0094] Synthesize the target language text into target audio; and

[0095] The sentence break timestamps of the target audio and the original audio are compared, and the original text of the original audio is translated a second time to make the timing of the target audio consistent with that of the original audio.

[0096] In another aspect, this application also provides a computer program product, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer program product stores one or more programs, which, when used by one or more processors, execute the speech translation method described in this application:

[0097] Obtain the raw text and sentence timestamps of the original audio;

[0098] Translate the original text of the original audio into the target language text;

[0099] Synthesize the target language text into target audio; and

[0100] The sentence break timestamps of the target audio and the original audio are compared, and the original text of the original audio is translated a second time to make the timing of the target audio consistent with that of the original audio.

[0101] The above description is merely a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made based on the content of this application's specification and drawings under the concept of this application, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.

Claims

1. A speech translation method, characterized in that, include: Obtain the raw text and sentence timestamps of the original audio; Translate the original text of the original audio into the target language text; The target language text is synthesized into target audio; and The sentence break timestamps of the target audio and the original audio are compared. Based on the text lengths of the target audio and the original audio, a secondary translation prompt program is set to perform a secondary translation of the original text of the original audio so that the timing of the target audio and the original audio is consistent.

2. The speech translation method according to claim 1, characterized in that, The step of translating the original text of the original audio into target language text includes: The original text of the original audio is translated into the target language text using a large text model.

3. The speech translation method according to claim 1, characterized in that, The process of obtaining the original text and sentence timestamps of the original audio includes: The original text and sentence timestamps of the original audio are obtained using the STT algorithm.

4. The speech translation method according to claim 1, characterized in that, The step of synthesizing the target language text into target audio includes: The target language text is synthesized into the target audio using a TTS algorithm.

5. The speech translation method according to claim 1, characterized in that, After comparing the sentence break timestamps of the target audio and the original audio, and performing a secondary translation of the original text of the original audio, the process further includes: The target language text, after secondary translation, is synthesized into the target audio.

6. The speech translation method according to claim 5, characterized in that, The target language text, after secondary translation, is synthesized into target audio using a TTS algorithm.

7. A voice translation device, characterized in that, include: The text acquisition module is used to acquire the raw text and sentence timestamps of the original audio. The text translation module is used to translate the original text of the original audio into text in the target language; An audio synthesis module is used to synthesize the target language text into target audio; and The secondary translation module is used to compare the sentence break timestamps of the target audio and the original audio. Based on the text length of the target audio and the original audio, the module performs a secondary translation of the original text of the original audio by setting a secondary translation prompt program, so that the timing of the target audio and the original audio is consistent.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the speech translation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for implementing the speech translation method according to any one of claims 1-6.

Images