Video dubbing system and method, electronic equipment and storage medium

Abstract

The invention provides a video dubbing system and method, electronic equipment and a storage medium. The system comprises a video separation module, a human voice separation module, a voice transcription module, a text translation module, a dubbing module and an audio and video synthesis module. Performing audio-video separation on the audio-video to be dubbed through a video separation module to obtain an original video and an original audio; and separating the original audio based on the human voice separation module to obtain a to-be-dubbed human voice audio and a background audio. And then, based on a voice transcription and translation module, obtaining a translated text segment, and carrying out dubbing and speech speed adjustment on the translated text segment to realize audio and video synthesis. According to the system provided by the embodiment of the invention, the audio and video separation module, the human voice separation module, the voice transfer module, the text translation module, the dubbing module and the audio and video synthesis module are integrated into the video dubbing system, so that the automatic processing of the whole film and television dubbing process is realized, and the dubbing production efficiency and quality are greatly improved; and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video dubbing system, method, electronic equipment and storage medium. Background Art

[0002] Against the backdrop of increasing globalization and cross-language communication, film and television translation and dubbing have become an indispensable part of film and television production. Currently, films, TV series, cartoons, documentaries and other film and television works need to meet the multilingual needs of the international market. Accurate and efficient film and television translation and dubbing are essential for literature, technology, law, business and other fields. In the traditional film and television dubbing industry, the film and television dubbing process relies on a large number of manual operations, including audio and video separation, manual transcription, dubbing and audio and video synthesis.

[0003] However, the current reliance on manual large-scale dubbing for film and television is not only time-consuming and labor-intensive, but also prone to human errors, which affects the final dubbing quality. Summary of the invention

[0004] The present invention provides a video dubbing system, method, electronic device and storage medium, which are used to solve the defects in the prior art that large-scale film and television dubbing relying on manual labor is not only time-consuming and labor-intensive, but also has low dubbing quality.

[0005] The present invention provides a video dubbing system, comprising an audio-video separation module, a human voice separation module, a speech transcription module, a text translation module, a dubbing module and an audio-video synthesis module, wherein the audio-video separation module, the human voice separation module, the speech transcription module, the text translation module, the dubbing module and the audio-video synthesis module are connected in sequence, wherein: The audio and video separation module is used to obtain the audio and video to be dubbed, parse the audio and video to be dubbed, and extract the original video and original audio; The human voice separation module is used to perform audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; The speech transcription module is used to perform speech recognition on the human voice audio to be dubbed, and obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; The text translation module is used to translate the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training the Transformer model; The dubbing module is used to dub the translated text segment to obtain a dubbing audio segment, and compare the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjust the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; The audio and video synthesis module is used to perform audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

[0006] According to a video dubbing system provided by the present invention, the dubbing module includes a voice conversion module and a time synchronization module, wherein: The speech conversion module is used to obtain the dubbing timbre based on the spectral features of the original audio, and dub the translated text segment using the deep learning-based text-to-speech model and the dubbing timbre to obtain the dubbing audio segment; The time synchronization module is used to calculate the dubbing duration ratio of each dubbing audio segment based on the original duration of each text segment to be dubbed and the dubbing duration of the corresponding dubbing audio segment, and to perform linear interpolation on each dubbing audio segment based on the dubbing duration ratio to obtain the synchronized dubbing audio segment.

[0007] According to a video dubbing system provided by the present invention, the time synchronization module includes a splicing module; The splicing module is used to splice the synchronized dubbing audio segments based on the silent segments of different lengths to obtain the complete dubbing audio; The length of the silent segment is determined based on the time interval between two adjacent synchronized dubbing audio segments in the audio to be dubbed; The audio and video synthesis module is specifically used for: The dubbing audio and video is synthesized based on the original video, the background audio and the complete dubbing audio to obtain the dubbing audio and video.

[0008] According to a video dubbing system provided by the present invention, the text translation module is specifically used for: Based on the text translation model and the preset professional terminology library, the text segment to be dubbed is translated into the target language to obtain the translated text segment.

[0009] According to a video dubbing system provided by the present invention, the human voice separation module includes a background sound detection module before the human voice separation module; The background sound detection module is used to filter the original audio to obtain a mixed sound audio based on the spectrum characteristics of the original audio and the sound classification model, and transmit the mixed sound audio to the human voice separation module; The human voice separation module is specifically used for: Receiving the mixed sound audio, performing audio processing on the mixed sound audio based on a sound separation model, and separating the mixed sound audio to obtain the human voice audio to be dubbed and the background audio; The sound classification model and the sound separation model are obtained based on training of a neural network model.

[0010] According to a video dubbing system provided by the present invention, the speech transcription module is specifically used for: Performing noise suppression and / or vocal enhancement on the human voice audio to be dubbed to obtain enhanced human voice audio; Based on the spectral features of the enhanced human voice audio and the speech recognition model, the enhanced human voice audio is subjected to speech recognition to obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the enhanced human voice audio; The speech recognition model is trained based on the Transformer encoder-decoder model.

[0011] The present invention also provides a video dubbing method, comprising: Obtaining the audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; Performing audio processing on the original audio to separate the audio of the human voice to be dubbed and the background audio; Performing speech recognition on the audio of the human voice to be dubbed to obtain text segments to be dubbed and the original duration of each text segment to be dubbed in the audio of the human voice to be dubbed; Performing text translation of the to-be-dubbed text segment into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training a Transformer model; Dubbing the translated text segment to obtain a dubbing audio segment, and comparing the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; Audio and video synthesis is performed based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned video dubbing methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video dubbing method described in any one of the above methods is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video dubbing method described above is implemented.

[0015] The video dubbing system, method, electronic device and storage medium provided by the present invention integrate an audio and video separation module, a human voice separation module, a speech transcription module, a text translation module, a dubbing module and an audio and video synthesis module into a video dubbing system, thereby realizing automated processing of the entire process of film and television dubbing, thereby greatly improving the efficiency and quality of dubbing production and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a structural schematic diagram of the video dubbing system provided by the present invention; Figure 2 This is one of the flow charts of the video dubbing method provided by the present invention; Figure 3 This is the second flow chart of the video dubbing method provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that the current traditional manual dubbing method for film and television has the following problems: 1. Complexity of audio and video separation: In traditional methods, audio and video separation requires manual operation, which is prone to errors and time-consuming. 2. Background sound interference problem: In audio containing background sound, it is difficult to extract pure human voice audio. 3. Low efficiency of recording and transcription: Manual recording and transcription is not only inefficient, but also prone to errors. 4. Accuracy and speed of subtitle translation: Manual translation of subtitles is not only time-consuming, but also may be inaccurate. 5. The cumbersomeness and cost of dubbing recording: Traditional dubbing recording requires professional dubbing actors and recording equipment, and the process is cumbersome and costly. The present invention uses an AI dubbing module and text-to-speech technology to automatically generate high-quality dubbing audio, reducing dependence on professional dubbing resources. 6. Difficulty in audio and video synchronization: In manual operation, the synchronous synthesis of audio and video is prone to asynchronous problems. The present invention automatically synchronizes the silent video with the dubbing audio through the audio and video synthesis module, automatically adjusts the speech speed to match the duration and lines, and ensures that the output dubbing film and television files are synchronized with audio and video.

[0020] In view of the above problems, the present invention provides a video dubbing system to realize an integrated system of efficient and high-quality film and television dubbing. Figure 1 Schematic diagram of the structure of the video dubbing system provided by the present invention. Figure 1 As shown, the system includes: an audio and video separation module 110, a human voice separation module 120, a speech transcription module 130, a text translation module 140, a dubbing module 150 and an audio and video synthesis module 160, wherein the audio and video separation module 110, the human voice separation module 120, the speech transcription module 130, the text translation module 140, the dubbing module 150 and the audio and video synthesis module 160 are connected in sequence, wherein: The audio and video separation module 110 is used to obtain the audio and video to be dubbed, parse the audio and video to be dubbed, and extract the original video and original audio; The human voice separation module 120 is used to perform audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; The speech transcription module 130 is used to perform speech recognition on the human voice audio to be dubbed, and obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; The text translation module 140 is used to translate the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training the Transformer model; The dubbing module 150 is used to dub the translated text segment to obtain a dubbing audio segment, and compare the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjust the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; The audio and video synthesis module 160 is used to perform audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

[0021] Specifically, in the audio and video separation module 110, first, the audio and video to be dubbed is obtained, and the audio and video to be dubbed is parsed to extract the original video and the original audio. For example, the audio and video to be dubbed input by the user can be received, and then the input film and television file can be parsed based on the multimedia parsing tool to extract and separate the audio stream and the video stream therein, and the original video and the original audio can be parsed. The multimedia parsing tool here can be FFmpeg. Finally, the original video obtained by parsing can be transmitted to the audio and video synthesis module 160 for the final audio and video synthesis. In addition, the original audio can be transmitted to the audio and video separation module 120.

[0022] The audio and video to be dubbed here can be in MP4, MKV and other formats. The audio and video to be dubbed here can include human voice, background sound and corresponding video, and can include multiple human voices. The original video here refers to a silent video.

[0023] It should be noted that, compared with the prior art, audio and video separation relies on manual operation and is inefficient. Traditional audio and video editing software cannot achieve fast and automatic audio and video separation, and requires a lot of manpower and time. The audio and video separation module 110 provided by the present invention performs audio and video analysis based on multimedia analysis tools, and the automated audio and video separation improves processing efficiency and accuracy, reduces manual intervention, and improves the efficiency of preprocessing in the video dubbing process.

[0024] In the audio and video separation module 120, first, the original audio transmitted by the audio and video separation module 110 is received. Then, the original audio is separated from the human voice and the background sound to obtain the human voice audio and background audio to be dubbed. For example, the original audio can be processed by a human voice and background sound separation model based on a neural network, such as U-Net, Wave-U-Net, and BiLSTM to separate the human voice and the background sound, that is, to obtain the human voice audio and background audio to be dubbed. Finally, the separated background audio is transmitted to the audio and video synthesis module 160 for the final audio and video synthesis. And the human voice audio to be dubbed is transmitted to the speech transcription module 130.

[0025] Here, the human voice audio to be dubbed refers to the audio stream of all the human voices contained in the audio and video to be dubbed. For example, if video A contains characters A, B, and C, the human voice audio to be dubbed can be the audio stream containing characters A, B, and C. The background audio here refers to the audio that does not need to be translated or dubbed, such as environmental sounds.

[0026] It is understandable that compared with the prior art, spectrum analysis and filtering are often used, but due to the complexity and diversity of background sound, the existing methods cannot achieve the ideal separation effect while ensuring the sound quality. The present invention processes the original audio through the human voice separation module 120 based on neural networks, such as U-Net, Wave-U-Net, BiLSTM human voice and background sound separation models, effectively separates the human voice and background sound, improves the purity of the human voice audio, and ensures the quality of subsequent processing.

[0027] In the speech transcription module 130, first, the audio of the human voice to be dubbed is received. Then, the spectral features or logarithmic Mel spectrogram of the audio of the human voice to be dubbed can be extracted, and then the extracted spectral features or logarithmic Mel spectrogram and the audio of the human voice to be dubbed can be input into the speech recognition model, and speech recognition can be performed through the speech recognition model, or speech recognition can be implemented through IBM Watson ASR to obtain the text segment to be dubbed and the original duration of each text segment to be dubbed in the audio of the human voice to be dubbed. Finally, the text segment to be dubbed obtained by speech recognition and the original duration of each text segment to be dubbed in the audio of the human voice to be dubbed are transmitted to the text translation module 140.

[0028] It should be noted that the speech recognition model here can be a Conformer-based or transformer-based encoder-decoder model. In addition, the text segments to be dubbed here can be divided by speaker. For example, the text obtained by speech recognition of speaker A's voice audio to be dubbed is used as a separate text segment to be dubbed. The original duration of each text segment to be dubbed in the voice audio to be dubbed here represents the audio duration corresponding to the recognized text.

[0029] In the text translation module 140, first, the text segments to be dubbed and the original duration of each text segment to be dubbed in the audio of the human voice to be dubbed can be received. Then, the text segments to be dubbed can be translated into the target language based on the text translation model, or the DeepL technology can be applied to translate the text to obtain the translated text segments. For example, the text segments to be dubbed can be input into the text translation model, and the text translation model can be used to translate the text into the target language. Then, the quality of the text obtained after translation can be evaluated and automatically corrected to obtain the final translated text segments to ensure the accuracy, fluency and consistency of the translated content. Finally, the translated text segments obtained by translating each text segment to be dubbed and the original duration of the corresponding text segments to be dubbed in the audio of the human voice to be dubbed can be stored, and can be stored as SRT (SubRip Text) files for subsequent use by the dubbing module 150.

[0030] It should be noted that the text translation model obtained here based on the text translation module 140 can be obtained based on the sample text pair and the initial Transformer model training. The target language here can include different languages ​​and dialects, such as Chinese, English, Korean and other languages, and can also include Sichuan dialect, Minnan dialect and other dialects.

[0031] In the dubbing module 150, first, the original duration of the translated text segment and each translated text segment and the corresponding text segment to be dubbed in the human voice audio to be dubbed can be obtained. Then, the translated text segment can be dubbed by the AI ​​dubbing model to obtain a dubbing audio segment. The original duration of each text segment to be dubbed is compared with the dubbing duration of the corresponding dubbing audio segment, and the speech speed of each dubbing audio segment is adjusted based on the duration comparison result to obtain a synchronized dubbing audio segment. Thus, the playback duration of the dubbing audio segment can be synchronized with the playback duration of the corresponding original audio segment to improve the quality of the audio and video combination for dubbing, reduce the dependence on professional manual dubbing actors, and reduce the dubbing cost. Finally, the synchronized dubbing audio segment obtained by dubbing is transmitted to the audio and video synthesis module 160.

[0032] In the audio and video synthesis module 160, first, the synchronous dubbing audio segment, the original video, and the background audio are received. Then, a multimedia processing tool such as FFmpeg can be used to synthesize the silent original video, the synchronous dubbing audio segment, and the background audio to obtain the dubbing audio and video. The synthesized dubbing audio and video can also be quality checked to ensure the synchronization of audio and video and the playback effect, so as to obtain the final dubbing audio and video.

[0033] It should be noted that the present invention integrates multiple processing steps such as audio and video separation, human voice separation processing, transcription, translation, dubbing and audio and video synthesis, and realizes the automatic processing of the whole process through the automation system, which reduces the errors caused by manual intervention and multiple conversions, and improves the coherence and consistency of the overall processing. In addition, the system adopts a modular design, and each functional module can be optimized and upgraded independently, which is convenient for integrating the latest technology. The modular design has a high degree of flexibility and scalability, and can be customized and expanded according to user needs to meet the needs of different users and markets. In addition, through intelligent background sound detection and separation technology, automatic speech recognition, machine translation and text-to-speech conversion, the present invention significantly improves the intelligence level and processing efficiency of the film and television dubbing process, and provides an efficient, low-cost, high-quality dubbing solution.

[0034] The system provided by the embodiment of the present invention integrates an audio and video separation module, a human voice separation module, a speech transcription module, a text translation module, a dubbing module and an audio and video synthesis module into a video dubbing system, thereby realizing automated processing of the entire process of film and television dubbing, thereby greatly improving the efficiency and quality of dubbing production and reducing labor costs.

[0035] Based on any of the above embodiments, the dubbing module 150 includes a voice conversion module and a time synchronization module, wherein: The speech conversion module is used to obtain the dubbing timbre based on the spectral features of the original audio, and dub the translated text segment using the deep learning-based text-to-speech model and the dubbing timbre to obtain the dubbing audio segment; The time synchronization module is used to calculate the dubbing duration ratio of each dubbing audio segment based on the original duration of each text segment to be dubbed and the dubbing duration of the corresponding dubbing audio segment, and to perform linear interpolation on each dubbing audio segment based on the dubbing duration ratio to obtain the synchronized dubbing audio segment.

[0036] Specifically, in the speech conversion module, first, the spectral features of the original audio can be extracted. Then, based on the spectral features of the original audio, the dubbing timbre is determined. Next, the translated text segment can be dubbed using a deep learning-based text-to-speech model and dubbing timbre, or text-to-speech (TTS) technology, such as Amazon Polly, can be applied to obtain a dubbing audio segment that is consistent with the original audio in timbre. Alternatively, the spectral features of the original audio and the dubbing timbre of the translated text segment can be input into an end-to-end text-to-speech model to directly generate dubbing audio segments in different languages ​​but with the same timbre.

[0037] Here, the text-to-speech model can be a model based on Variational Inference Text-to-Speech (VITS) or Transformer+Vocoder structure. In addition, the dubbing audio segment here corresponds to each translated text segment. It is understandable that there may be differences in text length or pronunciation between the translated text segment and its corresponding text segment to be dubbed. Therefore, there may be differences in the duration of the dubbing audio segment and the duration of the corresponding original audio segment, and this problem can be solved by a time synchronization module to achieve synchronization of the dubbing audio segment and the corresponding original audio segment.

[0038] In the time synchronization module, first, the dubbing duration ratio of each dubbing audio segment can be calculated by the original duration of each to-be-dubbed text segment and the dubbing duration of the corresponding dubbing audio segment. Then, the linear adjustment function can be defined using the librosa library according to the dubbing duration ratio, and each dubbing audio segment can be linearly interpolated, the speech speed can be adjusted to match the duration and the lines, and the playback speed of each dubbing audio segment can be adjusted to ensure audio and video synchronization, so as to obtain the synchronized dubbing audio segment corresponding to each dubbing audio segment.

[0039] The dubbing duration ratio here can reflect the difference between the original duration of the dubbing text segment to be dubbed and the dubbing duration of the corresponding dubbing audio segment. In addition, the synchronized dubbing audio segment here can be regarded as the dubbing audio in different languages ​​synchronized with the original audio, and the synchronized dubbing audio segment is also synchronized with the playback progress of the original video.

[0040] Based on any of the above embodiments, the time synchronization module includes a splicing module; The splicing module is used to splice the synchronized dubbing audio segments based on the silent segments of different lengths to obtain the complete dubbing audio; The length of the silent segment is determined based on the time interval between two adjacent synchronized dubbing audio segments in the audio to be dubbed; The audio and video synthesis module 160 is specifically used for: The dubbing audio and video is synthesized based on the original video, the background audio and the complete dubbing audio to obtain the dubbing audio and video.

[0041] Specifically, in the splicing module, each synchronized dubbing audio segment can be spliced ​​by using silent segments of different lengths to obtain a complete dubbing audio. It should be noted that the length of the silent segment here is determined based on the time interval between two adjacent synchronized dubbing audio segments in the audio of the human voice to be dubbed. Thus, the complete dubbing audio obtained by splicing has the same playing time as the original audio, and then in the audio and video synthesis module 160, the original video, background audio and complete dubbing audio can be synthesized to obtain a dubbing audio and video with the video screen synchronized with the dubbing.

[0042] Based on any of the above embodiments, the text translation module is specifically used for: Based on the text translation model and the preset professional terminology library, the text segment to be dubbed is translated into the target language to obtain the translated text segment.

[0043] Here, the preset professional terminology library may contain the original text and translation text of professional vocabulary and phrases of each industry to ensure the accuracy and professionalism of the translation. Thus, the text segment to be dubbed can be input into the text translation model containing the preset professional terminology library, and the text translation model can output an accurate and professional translation text segment. It is understandable that the translated translation text segment can be stored together with the timeline information corresponding to the text segment to be dubbed into an SRT (SubRip Text) file for use by the subsequent dubbing module.

[0044] Based on any of the above embodiments, the human voice separation module 120 includes a background sound detection module; The background sound detection module is used to filter the original audio to obtain a mixed sound audio based on the spectrum characteristics of the original audio and the sound classification model, and transmit the mixed sound audio to the human voice separation module; The human voice separation module is specifically used for: Receiving the mixed sound audio, performing audio processing on the mixed sound audio based on a sound separation model, and separating the mixed sound audio to obtain the human voice audio to be dubbed and the background audio; The sound classification model and the sound separation model are obtained based on training of a neural network model.

[0045] Specifically, in the background sound detection module, first, the spectral features of the original audio can be extracted. Then, the spectral features of the original audio and the original audio can be applied through a sound classification model, such as a convolutional neural network, a support vector machine, etc., to determine whether the original audio contains background sound, so as to filter out audio that does not contain pure background sound from the original audio, that is, to filter out mixed sound audio. It should be noted that the mixed sound audio here refers to the audio after the background sound has been preliminarily removed, that is, the audio after the pure background sound has been removed, and the mixed sound audio may contain a mixture of background sound and human voice. In addition, the sound classification model and the sound separation model here can both be obtained based on training of the neural network model.

[0046] In order to further eliminate the background sound in the mixed sound audio and obtain a purer human voice audio. The mixed sound audio can be transmitted to the human voice separation module, so that the mixed sound audio is received by the human voice separation module, and the mixed sound audio is processed based on the sound separation model to further separate the human voice audio to be dubbed and the background audio. The separated human voice audio stream can be saved in MP3 format and passed to the speech transcription module. The background sound is temporarily stored and added when the audio and video synthesis module is finally generated.

[0047] It can be understood that by setting a background sound detection module before the human voice separation module, preliminary background sound screening is performed to eliminate the audio of pure background sound and obtain a mixed sound audio for secondary human voice and background sound separation, so that the human voice audio to be dubbed obtained by voice separation based on the mixed sound audio is purer, which is conducive to improving the accuracy of subsequent speech transcription based on the human voice audio to be dubbed, and then improving the accuracy of subsequent other operations to obtain high-quality dubbing audio and video.

[0048] It should be noted that the background sound detection module has been introduced, which uses a machine learning model to perform spectrum analysis and feature extraction on the audio stream to automatically determine whether the audio contains background sound. For audio containing background sound, advanced neural network technology (such as U-Net, Wave-U-Net, etc.) is used to achieve efficient separation of human voice and background sound, ensuring the quality of human voice audio in subsequent processing steps.

[0049] Based on any of the above embodiments, the speech transcription module is specifically used for: Performing noise suppression and / or vocal enhancement on the human voice audio to be dubbed to obtain enhanced human voice audio; Based on the spectral features of the enhanced human voice audio and the speech recognition model, the enhanced human voice audio is subjected to speech recognition to obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the enhanced human voice audio; The speech recognition model is trained based on the Transformer encoder-decoder model.

[0050] Specifically, in the speech transcription module 130, first, the human voice audio to be dubbed can be subjected to noise suppression and / or human voice enhancement to obtain enhanced human voice audio. Then, based on the spectral features of the enhanced human voice audio and the speech recognition model, speech recognition can be performed on the enhanced human voice audio to obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the enhanced human voice audio.

[0051] It is understandable that before speech transcription, pre-processing operations such as noise suppression and / or voice enhancement are performed to enhance the vocal part and reduce noise interference, so as to further improve the accuracy of speech recognition and thus improve the accuracy of subsequent dubbing tasks.

[0052] It should be noted that the automatic speech recognition (ASR) technology is used to identify pure human voice audio and generate highly accurate text segments to be dubbed. Compared with the traditional manual transcription method, this module significantly improves the efficiency and accuracy of generating text segments to be dubbed.

[0053] Based on any of the above embodiments, audio and video separation, background sound detection, voice and background sound separation, transcription, subtitle translation, AI dubbing and audio and video synthesis modules can be integrated into a unified system. Then, the system optimization algorithm can be used to improve the processing efficiency and accuracy of each module to ensure the consistency and stability of the overall process. Provide a user-friendly interface to facilitate users to input film and television files and obtain the final dubbing film and television files, and provide output result files for each step for users to intervene manually when necessary. According to the latest technological developments, continuously optimize and upgrade each module, optimize the training of each module model, improve the processing speed of each module model from aspects such as model distillation and memory optimization, and maintain the advancement and efficiency of the system. Monitor the entire system in real time, collect user feedback, make continuous improvements, and improve the stability of the system and user experience.

[0054] It should be noted that the system provided by the embodiment of the present invention, by realizing the full automation of the film and television dubbing process, brings significant beneficial effects, as follows: 1. Improve production efficiency: The traditional dubbing production process involves a number of complex and time-consuming manual operations, while the present invention can automate multiple steps such as audio and video separation, background sound detection and separation, audio recording and transcription, subtitle translation, AI dubbing and audio and video synthesis through an automated system, significantly shortening the production cycle and improving overall production efficiency. 2. Reduce production costs: Reduce dependence on professional dubbing actors, translators and recording equipment, and reduce manpower and equipment costs. The automated system reduces manual intervention and reduces the cost of rework and correction caused by human errors, thereby further reducing overall production costs. 3. Improve dubbing quality: By using advanced neural network technology for background sound separation and speech recognition, the purity of human voice audio and the accuracy of subtitle transcription are guaranteed. The AI ​​dubbing module uses high-quality text-to-speech conversion technology to generate natural and smooth dubbing audio to ensure the professional level and consistency of dubbing. 4. Multi-language support: The subtitle translation and AI dubbing of the embodiment of the present invention can support the processing of multiple languages. The user only needs to input the film and television files in the original language, and the system can automatically generate multi-language versions of dubbing film and television files to meet the needs of different language markets and facilitate the international dissemination of film and television works. 5. Improve processing accuracy and consistency: The systematic processing flow ensures the accuracy and consistency of each step. Background sound detection and separation technology can accurately extract human voice audio, and automatic speech recognition and machine translation technology improve the accuracy of subtitle transcription and translation, ensuring the consistency of dubbing content with the original text. 6. Enhance user experience: Through the rapid response capability of the automated system, users can obtain high-quality dubbing works in a relatively short time. Especially when batch processing of film and television files is required, the system can provide efficient and stable dubbing services, significantly improving user experience and satisfaction. 7. System flexibility and scalability: The present invention adopts a modular design, and each functional module can be independently optimized and upgraded, which is convenient for integrating the latest technology, has high flexibility and scalability, can be customized and expanded according to needs, and meets the needs of different users and markets. 8. Reduce human errors: Automated processing reduces mistakes and omissions in manual operations and ensures the stability and consistency of each processing step. The system is able to perform highly repetitive and tedious tasks, reducing the uncertainty caused by manual operation and improving the reliability of processing. 9. Environmental protection and energy saving: The automated system reduces the need for professional recording equipment and manual sites, reducing energy consumption and environmental burden. The system's efficient processing capacity reduces the use time and maintenance frequency of equipment, further saves resources, and has the advantages of environmental protection and energy saving.

[0055] Based on any of the above embodiments, Figure 2 FIG. 1 is one of the flow charts of the video dubbing method provided by the present invention, such as Figure 2As shown, the method includes: Step 210, obtaining the audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; Step 220, performing audio processing on the original audio to separate the audio of the human voice to be dubbed and the background audio; Step 230, performing speech recognition on the audio to be dubbed to obtain text segments to be dubbed and the original duration of each text segment to be dubbed in the audio to be dubbed; Step 240, translating the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained based on training of a Transformer model; Step 250, dubbing the translated text segment to obtain a dubbing audio segment, and comparing the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; Step 260, performing audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain dubbing audio and video.

[0056] The method provided by the embodiment of the present invention integrates an audio and video separation module, a human voice separation module, a speech transcription module, a text translation module, a dubbing module and an audio and video synthesis module into a video dubbing system, thereby realizing automated processing of the entire process of film and television dubbing, thereby greatly improving the efficiency and quality of dubbing production and reducing labor costs.

[0057] Based on any of the above embodiments, Figure 3 FIG. 2 is a flow chart of the video dubbing method provided by the present invention. Figure 3 As shown, the method includes: first, receiving a video uploaded by a user. Then, the uploaded video is automatically separated from the video and audio to obtain the audio and the silent video. The silent video can be used for the final automatic synthesis dubbing video, and the audio is automatically analyzed to determine whether it contains background sound. If it does not contain background sound, it is classified as human voice; then the audio containing background sound and human voice is separated from the human voice and background sound. Furthermore, by performing AI audio recording and subtitle recording on the human voice, speech transcription into text is realized.

[0058] Then, AI subtitles are translated for the transcribed text. It should be noted that the translated text can be aligned with the length of the audio and adjusted in accordance with the speech speed. In addition, the translated content is adjusted in accordance with the habits of the film and television characters. Finally, AI subtitles are output.

[0059] Furthermore, AI subtitles can be used to automatically identify the speaker's role, perform AI dubbing, and output dubbing audio. Finally, the background sound, dubbing audio, and silent video are automatically synthesized into a dubbing video.

[0060] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the video dubbing method, which includes: obtaining audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; performing audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; performing speech recognition on the human voice audio to be dubbed to obtain the text segment to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; performing text translation on the text segment to be dubbed into the target language based on the text translation model to obtain the translated text segment; the text translation model is obtained by training the Transformer model; dubbing the translated text segment to obtain the dubbing audio segment, and comparing the original duration of each text segment to be dubbed with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain the synchronized dubbing audio segment; performing audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

[0061] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0062] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video dubbing method provided by the above methods, which includes: obtaining audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; performing audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; performing speech recognition on the human voice audio to be dubbed to obtain a text segment to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; performing text translation on the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training a Transformer model; dubbing the translated text segment to obtain a dubbing audio segment, and comparing the original duration of each text segment to be dubbed with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; performing audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain a dubbing audio and video.

[0063] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to execute the video dubbing method provided by the above methods, the method comprising: obtaining audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; performing audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; performing speech recognition on the human voice audio to be dubbed to obtain a text segment to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; performing text translation on the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training a Transformer model; dubbing the translated text segment to obtain a dubbing audio segment, and comparing the original duration of each text segment to be dubbed with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; performing audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain a dubbing audio and video.

[0064] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video dubbing system, characterized in that: It includes an audio and video separation module, a human voice separation module, a voice transcription module, a text translation module, a dubbing module and an audio and video synthesis module, wherein the audio and video separation module, the human voice separation module, the voice transcription module, the text translation module, the dubbing module and the audio and video synthesis module are connected in sequence, wherein: The audio and video separation module is used to obtain the audio and video to be dubbed, parse the audio and video to be dubbed, and extract the original video and original audio; The human voice separation module is used to perform audio processing on the original audio to separate the human voice audio to be dubbed and the background audio; The speech transcription module is used to perform speech recognition on the human voice audio to be dubbed, and obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the human voice audio to be dubbed; The text translation module is used to translate the text segment to be dubbed into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training the Transformer model; The dubbing module is used to dub the translated text segment to obtain a dubbing audio segment, and compare the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjust the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; The audio and video synthesis module is used to perform audio and video synthesis based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

2. The video dubbing system according to claim 1, characterized in that: The dubbing module includes a voice conversion module and a time synchronization module, wherein: The speech conversion module is used to obtain the dubbing timbre based on the spectral features of the original audio, and dub the translated text segment using the deep learning-based text-to-speech model and the dubbing timbre to obtain the dubbing audio segment; The time synchronization module is used to calculate the dubbing duration ratio of each dubbing audio segment based on the original duration of each text segment to be dubbed and the dubbing duration of the corresponding dubbing audio segment, and to perform linear interpolation on each dubbing audio segment based on the dubbing duration ratio to obtain the synchronized dubbing audio segment.

3. The video dubbing system according to claim 2, characterized in that: The time synchronization module includes a splicing module afterwards; The splicing module is used to splice the synchronized dubbing audio segments based on the silent segments of different lengths to obtain the complete dubbing audio; The length of the silent segment is determined based on the time interval between two adjacent synchronized dubbing audio segments in the audio to be dubbed; The audio and video synthesis module is specifically used for: The dubbing audio and video is synthesized based on the original video, the background audio and the complete dubbing audio to obtain the dubbing audio and video.

4. The video dubbing system according to any one of claims 1 to 3, characterized in that: The text translation module is specifically used for: Based on the text translation model and the preset professional terminology library, the text segment to be dubbed is translated into the target language to obtain the translated text segment.

5. The video dubbing system according to any one of claims 1 to 3, characterized in that: The human voice separation module includes a background sound detection module before the human voice separation module; The background sound detection module is used to filter the original audio to obtain a mixed sound audio based on the spectrum characteristics of the original audio and the sound classification model, and transmit the mixed sound audio to the human voice separation module; The human voice separation module is specifically used for: Receiving the mixed sound audio, performing audio processing on the mixed sound audio based on a sound separation model, and separating the mixed sound audio to obtain the human voice audio to be dubbed and the background audio; The sound classification model and the sound separation model are obtained based on training of a neural network model.

6. The video dubbing system according to any one of claims 1 to 3, characterized in that: The speech transcription module is specifically used for: Performing noise suppression and / or vocal enhancement on the human voice audio to be dubbed to obtain enhanced human voice audio; Based on the spectral features of the enhanced human voice audio and the speech recognition model, the enhanced human voice audio is subjected to speech recognition to obtain the text segments to be dubbed and the original duration of each text segment to be dubbed in the enhanced human voice audio; The speech recognition model is trained based on the Transformer encoder-decoder model.

7. A video dubbing method, characterized in that: include: Obtaining the audio and video to be dubbed, parsing the audio and video to be dubbed, and extracting the original video and original audio; Performing audio processing on the original audio to separate the audio of the human voice to be dubbed and the background audio; Performing speech recognition on the audio of the human voice to be dubbed to obtain text segments to be dubbed and the original duration of each text segment to be dubbed in the audio of the human voice to be dubbed; Performing text translation of the to-be-dubbed text segment into the target language based on a text translation model to obtain a translated text segment; the text translation model is obtained by training a Transformer model; Dubbing the translated text segment to obtain a dubbing audio segment, and comparing the original duration of each of the to-be-dubbed text segments with the dubbing duration of the corresponding dubbing audio segment, and adjusting the speech speed of each dubbing audio segment based on the duration comparison result to obtain a synchronized dubbing audio segment; Audio and video synthesis is performed based on the original video, the background audio and the synchronized dubbing audio segment to obtain the dubbing audio and video.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video dubbing method as claimed in claim 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video dubbing method as claimed in claim 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video dubbing method as claimed in claim 7 is implemented.

Images