End-to-end video translation method and device based on artificial intelligence and medium

Abstract

The invention belongs to the technical field of machine translation, and provides an end-to-end video translation method and device based on artificial intelligence and a medium, and the method comprises the steps: (1) processing a video file uploaded by a user: extracting an original audio file in the video file, and obtaining a human voice audio file and a background music audio file after processing; (2) generating a customizable bilingual subtitle file; (3) performing speech synthesis based on the bilingual subtitle file in the step (2) to obtain synthesized speech; and (4) performing video synthesis to generate a final translated video. End-to-end video translation is realized through technologies such as voice recognition, machine translation, voice synthesis and video synthesis, complete automatic video translation is realized, and great convenience is brought to users.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular, relates to an end-to-end video translation method, device and medium based on artificial intelligence. Background Art

[0002] With the development of artificial intelligence and the increase in international exchanges, machine translation has demonstrated its high efficiency. However, existing machine translation technology solutions are basically reflected in text translation, and there is still a lack of complete and high-quality technical solutions for video translation.

[0003] Video translation mainly includes two aspects: video content translation and speech synthesis. The existing video translation technology solutions have the following problems: (1) Video translation only supports subtitle translation or speech synthesis, and does not achieve end-to-end video translation.

[0004] (2) Subtitle translation does not support editing functions, and users cannot customize and proofread subtitle translation content.

[0005] (3) Speech synthesis is difficult, including problems such as synthesis fluency, audio and video synchronization, and dubbing of multiple characters. Summary of the invention

[0006] The purpose of the present invention is to provide an end-to-end video translation method based on artificial intelligence to solve the technical problems existing in the prior art.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows: An end-to-end video translation method based on artificial intelligence comprises the following steps: (1) Process the video files uploaded by users: extract the original audio files from the video files, and obtain the human voice audio files and background music audio files after processing; (2) Generate a customizable bilingual subtitle file: (21) Use the whisper model to perform speech recognition and convert the human voice audio file into an original subtitle file, where the original subtitle file distinguishes the start and end time of each speaking interval and the subtitle content; (22) Use the SpeakerRecognition model to perform speaker recognition on each speaking interval in the human voice audio file, and provide a default timbre for each speaker; (23) Use the LanMT machine translation engine to translate the subtitle content of the original subtitle file; (24) Integrate the contents obtained in (21), (22) and (23), with each spoken area corresponding to a subtitle content, integrate all the spoken areas, and generate a user-customizable bilingual subtitle file; (3) Based on the bilingual subtitle file in step (2), perform speech synthesis to obtain synthesized speech; (4) Video synthesis: generating the final translated video.

[0008] Preferably, in (1), the human voice and background music are separated by a spleeter model, and the audio file is subjected to noise reduction and dereverberation processing by using an MDX-Net and a DeEcho-Aggressive model to obtain a human voice audio file and a background music audio file.

[0009] Preferably, in (2), the generated bilingual subtitle files are returned to the user, and the user can modify all contents in each subtitle file to obtain a user-defined subtitle file.

[0010] Preferably, in (3), the specific method of speech synthesis is as follows: (31) using Tacotron and WaveNet algorithms to perform basic speech synthesis to obtain basic synthesized speech data; (32) based on the basic synthesized speech data, using the VALL-E algorithm to simulate the speaker's intonation, emotion, and accent; (33) using the VITS algorithm to perform voiceprint simulation; (34) using the HP2 model to perform noise reduction processing, eliminate electronic sound, and obtain the final synthesized speech.

[0011] Preferably, the (3) further comprises: performing a silent segment cutting and adding process on the synthesized speech, and then performing a speed-changing but pitch-invariant process on the processed synthesized speech by using a WSLOA algorithm.

[0012] Preferably, in (4), the video synthesis method is as follows: (41) replacing the original audio file with the synthesized speech obtained in step (3); (42) attaching the background music audio file to the video; (43) adding the user-defined subtitle file to the video to generate the final translated video.

[0013] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the end-to-end video translation method based on artificial intelligence.

[0014] In order to achieve the above-mentioned objectives, the present invention also provides an end-to-end video translation device based on artificial intelligence, comprising: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the end-to-end video translation device based on artificial intelligence performs the end-to-end video translation method based on artificial intelligence.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention realizes end-to-end video translation through speech recognition, machine translation, speech synthesis, video synthesis and other technologies, and realizes automatic dubbing of complete videos, which greatly facilitates user use.

[0016] (2) The present invention provides a subtitle editing function, and users can customize the content and format of subtitle files and adjust and proofread subtitles.

[0017] (3) The present invention uses audio separation and noise reduction algorithms to ensure the accuracy of speech recognition; and uses speech synthesis, timbre cloning, and voiceprint simulation algorithms to make the synthesized audio effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the process of Example 1 of the present invention.

[0019] Figure 2 This is a principle block diagram of Example 3 of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to have a clearer understanding and knowledge of the present invention, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described below are only used to explain the present invention, and are convenient for understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following embodiments, and the technical solutions provided by the embodiments should not limit the protection scope of the present invention.

[0021] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] Example 1 like Figure 1 As shown, this embodiment provides an end-to-end video translation method based on artificial intelligence, the principle of which is as follows: end-to-end video translation is achieved through speech recognition, machine translation, speech synthesis, video synthesis and other technologies.

[0023] In this embodiment, the specific method of the end-to-end video translation method based on artificial intelligence is as follows: 1. Get the original video and process the original video file uploaded by the user Extract the original audio file from the video file, and obtain the human voice audio file and background music audio file after processing; the processing method is as follows: separate the human voice and background music through the spleeter model, and use the MDX-Net and DeEcho-Aggressive models to reduce noise and dedeverberation on the audio file, and retain the clean human voice audio file and background music audio file. It should be noted that there is no need to extract video data in this step. The subsequent video synthesis is to directly replace the audio data and subtitle data of the original video file with the synthesized audio data and subtitle data.

[0024] 2. Generate customizable bilingual subtitle files In order to meet the personalized needs of users and facilitate users to edit subtitles, a customizable bilingual subtitle file is generated in this step. The generated bilingual subtitle file is returned to the user. The user can modify all the contents in each subtitle file to obtain a user-customized subtitle file.

[0025] In this step, the specific method of generating a customizable bilingual subtitle file is as follows: First, the whisper model is used for speech recognition to convert the human voice audio file into an original subtitle file, which distinguishes the start and end time of each speaking interval and the subtitle content; it should be noted that those skilled in the art can also use other open source models that can achieve the above purpose; Then, the SpeakerRecognition model is used to perform speaker recognition on each speaking interval in the human voice audio file, and a default timbre is provided for each speaker. It should be noted that the default timbre of different speakers is different. Then, the subtitle content of the original subtitle file is translated using the LanMT machine translation engine. Those skilled in the art may also use other machine translation engines or models. Finally, the above contents are integrated, each audio area corresponds to a subtitle content, and all audio areas are integrated to generate a user-customizable bilingual subtitle file. Users can review, edit, and reselect the timbre based on the returned bilingual subtitle file; the specific format of a single subtitle is as follows: {"id": 0, "Start Time": 6.42, "End Time": 11.64, "speaker id": "speaker_1", "Speaker voice": "clone", "Original subtitle": "What do you think of the future growth and development of LanMT translation engine?", "Translated subtitle": "What do you think of the future growth and development of LanMT credit engine?" }.

[0026] 3. Perform speech synthesis to obtain synthesized speech Based on the above bilingual subtitle file, speech synthesis is performed to obtain synthesized speech; the specific method of speech synthesis is as follows: First, in order to ensure the fluency and timbre similarity of speech synthesis, Tacotron and WaveNet algorithms are used to perform basic speech synthesis to obtain basic synthesized speech data; Then, based on the basic synthesized speech data, the VALL-E algorithm is used to simulate the speaker's intonation, emotion, and accent to make the synthesized speech more natural; Then, the VITS algorithm is used to simulate the voiceprint to ensure the stability of the tone; Finally, the HP2 model is used to perform noise reduction and eliminate electronic sounds to obtain the final synthesized speech, making the synthesized speech closer to the voice of a real person.

[0027] When each subtitle in the subtitle file is completed with speech synthesis, there will be a problem of audio and video being out of sync because the synthesized audio and the original audio file have different durations. In order to deal with this problem, the audio duration is first processed by cutting and adding silent segments to ensure that the duration of the synthesized audio and the original audio will not differ too much, and there will be no sudden changes in the speaking speed. Then, the entire synthesized audio file is processed with variable speed but not pitch using the WSLOA algorithm to ensure audio and video synchronization.

[0028] 4. Video synthesis to generate the final translated video First, the original audio file is replaced with the synthesized speech; then, the background music audio file is attached to the video; finally, the user-defined subtitle file is added to the video to generate the final translated video; the customized subtitle file refers to the file that the user has modified and edited according to the subtitle file, and these modifications and edits include subtitle color, font, size, position, etc.

[0029] Example 2 This embodiment provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the end-to-end video translation method based on artificial intelligence provided in Embodiment 1. A person of ordinary skill in the art can understand that all or part of the steps of implementing the method provided in Embodiment 1 can be completed by hardware related to the computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the method provided in Embodiment 1 are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0030] Example 3 like Figure 2 As shown, this embodiment provides an end-to-end video translation device based on artificial intelligence, including: a processor and a memory; the memory is used to store computer programs; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the end-to-end video translation device based on artificial intelligence executes the end-to-end video translation method based on artificial intelligence provided in Example 1.

[0031] Specifically, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0032] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0033] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An end-to-end video translation method based on artificial intelligence, characterized in that: The following steps are involved: (1) Process the video files uploaded by users: extract the original audio files from the video files, and obtain the human voice audio files and background music audio files after processing; (2) Generate a customizable bilingual subtitle file: (21) Use the whisper model to perform speech recognition and convert the human voice audio file into an original subtitle file, where the original subtitle file distinguishes the start and end time of each speaking interval and the subtitle content; (22) Use the SpeakerRecognition model to perform speaker recognition on each speaking interval in the human voice audio file, and provide a default timbre for each speaker; (23) Use the LanMT machine translation engine to translate the subtitle content of the original subtitle file; (24) Integrate the contents obtained in (21), (22) and (23), with each spoken area corresponding to a subtitle content, integrate all the spoken areas, and generate a user-customizable bilingual subtitle file; (3) Based on the bilingual subtitle file in step (2), perform speech synthesis to obtain synthesized speech; (4) Video synthesis: generating the final translated video.

2. The end-to-end video translation method based on artificial intelligence according to claim 1, characterized in that: In (1), the human voice and background music are separated by the spleeter model, and the audio files are denoised and de-reverberated by the MDX-Net and DeEcho-Aggressive models to obtain human voice audio files and background music audio files.

3. The end-to-end video translation method based on artificial intelligence according to claim 2 is characterized in that: In (2), the generated bilingual subtitle files are returned to the user, and the user can modify all the contents in each subtitle file to obtain a user-defined subtitle file.

4. The end-to-end video translation method based on artificial intelligence according to claim 3 is characterized in that: In (3), the specific method of speech synthesis is as follows: (31) using Tacotron and WaveNet algorithms to perform basic speech synthesis to obtain basic synthesized speech data; (32) based on the basic synthesized speech data, using VALL-E algorithm to simulate the speaker's intonation, emotion, and accent; (33) Use the VITS algorithm to simulate voiceprints; (34) Use the HP2 model to perform noise reduction and eliminate electronic sounds to obtain the final synthesized speech.

5. The end-to-end video translation method based on artificial intelligence according to claim 4 is characterized in that: The (3) also includes: performing silent segment cutting and adding processing on the synthesized speech, and then performing speed-changing but pitch-invariant processing on the processed synthesized speech through the WSLOA algorithm.

6. The end-to-end video translation method based on artificial intelligence according to claim 5, characterized in that: In (4), the video synthesis method is as follows: (41) replacing the original audio file with the synthesized speech obtained in step (3); (42) attaching the background music audio file to the video; (43) adding the user-defined subtitle file to the video to generate the final translated video.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the artificial intelligence-based end-to-end video translation method as described in any one of claims 1 to 6.

8. An end-to-end video translation device based on artificial intelligence, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the artificial intelligence-based end-to-end video translation device performs the artificial intelligence-based end-to-end video translation method as described in any one of claims 1 to 6.

Images