Automatic video translation system based on FFMPEG, TTS and Wav2Lip

Abstract

The invention discloses an automatic video translation system based on FFMPEG, TTS and Wav2Lip, and relates to the technical field of automatic translation processing, the automatic video translation system comprises a video processing unit, a translation synthesis unit and a mobile terminal, the video processing unit comprises a video input module, a video acquisition module, a preprocessing module, an audio extraction module and a video extraction module; the video input module is sequentially connected with the video acquisition module, the preprocessing module and the audio extraction module; the video extraction module is connected with the translation synthesis unit; according to the invention, face key points in a video frame are detected and tracked by using a computer vision technology through a mouth shape synchronization module, related voice features are extracted based on a face tracking result, and accurate synchronization of figure mouth shapes and voice contents in a translated video is ensured through technical means of image-to-image feedback circulation and the like; the translated video looks more natural and smooth, and the watching experience of audiences is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic translation processing, and in particular relates to an automatic video translation system based on FFMPEG, TTS and Wav2Lip. Background Art

[0002] With the development of globalization, the demand for the dissemination of video content in different language regions is growing. However, existing video translation methods have many limitations. The traditional manual video translation method is inefficient, requires a lot of manpower and time costs, and it is difficult to ensure the timeliness of translation; some automated video translation tools can only simply translate the text information in the video, such as subtitle translation, but the translated voice content in the video cannot match the lip shape of the characters in the original video screen well, resulting in a poor viewing experience. In addition, when dealing with related operations such as video format conversion, audio extraction and synthesis, there is a lack of efficient and integrated solutions, which makes the entire video translation process cumbersome and prone to compatibility problems. Therefore, we propose an automated video translation system based on FFMPEG, TTS, and Wav2Lip. Summary of the invention

[0003] The purpose of the present invention is to provide an automatic video translation system based on FFMPEG, TTS and Wav2Lip, thereby solving the existing problems.

[0004] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0005] The present invention is an automatic video translation system based on FFMPEG, TTS and Wav2Lip, comprising a video processing unit, a translation synthesis unit and a mobile terminal, and is characterized in that:

[0006] The video processing unit includes a video input module, a video acquisition module, a preprocessing module, an audio extraction module and a video extraction module; the video input module is sequentially connected to the video acquisition module, the preprocessing module, the audio extraction module and the video extraction module and connected to the translation synthesis unit;

[0007] The translation and synthesis unit includes a speech recognition and translation module, a text-to-speech module, an audio synthesis module, a lip synchronization module, a subtitle retention module and a video output module; the video extraction module is sequentially connected to the speech recognition and translation module, the text-to-speech module, the audio synthesis module, the lip synchronization module, the subtitle retention module and finally electrically connected to the video output module;

[0008] The speech recognition and translation module is used to automatically detect the language type of the audio through the model after deep learning, select the appropriate speech recognition model and translation engine, adjust and optimize the parameters of the speech recognition model according to the speech intonation and vocabulary pronunciation rules of different languages, and integrate a variety of professional translation engines, including API interfaces of mainstream translation services such as Google Translate, Baidu Translate, and Youdao Translation, as well as some professional translation engines for specific fields, to ensure efficient and accurate speech recognition in various language environments; the text-to-speech module is used to generate natural and fluent speech according to the text content through the text-to-speech model after deep learning, and make fine adjustments to the intonation, speaking speed, and pause elements of the speech to make it closer to the real human voice expression, and adjust the personalized speech output according to the voice style and intonation for different scenarios; the audio synthesis module is used to convert the text-to-speech model into a natural and fluent speech according to the text content ... voice intonation, speaking speed, and pause elements to make it closer to the real human voice expression, and adjust the personalized speech output according to the voice style and intonation for different scenarios. The target language speech generated by the audio module is re-synthesized with the video processed by the video content processing module to obtain a complete video file after translation, and the audio processing function of FFMPEG is used to ensure the quality and compatibility of the synthesis; the lip synchronization module is used to accurately synchronize the lip shape of the characters in the translated video with the speech content, and uses computer vision technology to detect and track the key points of the face in the video frame, mainly the outline and shape of the mouth part. Based on the face tracking results, the voice features, pitch, voice duration and voice features that describe different lip patterns are extracted. Through the image-to-image feedback loop, each generated image frame is used as input to generate the next frame of the animation, and the lip animation is aligned in time and space to ensure that it is consistent with the facial movements in the original video. The speed, amplitude and details of the animation are adjusted to match the facial movements in the original video; by extracting the position information of the facial feature points, the formula is as follows:

[0009] The subtitle retention module is used to dynamically set a suitable subtitle retention time for each semantic unit according to the above language results, combined with the video playback speed and the average speed of the general audience reading subtitles; the video output module is used to output the translated video file processed by the audio synthesis module according to the format and parameters specified by the user, which can be used directly by the user or further distributed and disseminated. Before outputting the video, the translated video is subjected to a final quality check to check whether there is any problem of picture quality degradation or audio distortion.

[0010] As a preferred technical solution, the video input module is used to receive original video files in various formats, supports common video formats, and is also compatible with niche video formats that still have usage scenarios. The module uses FFMPEG video format processing capabilities to perform preliminary analysis and format standardization on the input video; the video acquisition module is used to obtain the input original video material to be translated, and provide the required original video data for the subsequent processing of each module in the system, ensuring that the system has accurate and appropriate video content for translation and related processing.

[0011] As a preferred technical solution, the preprocessing module is used to convert input videos of different formats into a standard format that is easy to process within the system, while ensuring the reasonable adjustment and recording of key video parameters, performing integrity checks on the video stream, detecting whether there are data loss or image damage problems, and optimizing processing for some common video defects, color deviation, and image flickering through built-in image repair to ensure that the quality of the input video meets the basic requirements of subsequent processing; the audio extraction module is used to accurately extract the audio stream from the video processed by the video input module; the video extraction module is used to accurately separate the video stream from the original video file, and the video stream separation operation can be successfully completed regardless of the common format of the video input.

[0012] As a preferred technical solution, the mobile terminal includes a visualization window, a log record and a storage module; the video output module is respectively connected to the visualization window and the storage module through a network; the visualization window is respectively connected to the log directory and the storage module through a network.

[0013] As a preferred technical solution, the visualization window is used to display the main functions and operation procedures of the system to the user in a clear and intuitive manner when the user first starts the system or enters a specific functional module. When the system translates the video, the visualization window will display the running progress of each processing stage in real time; the storage module is used to provide a data storage space for the system, which is used to receive and save the original video files uploaded or imported by the user, and is also responsible for managing the original video files, recording the file name, file size, upload time, and file source of each video file, so as to facilitate subsequent query, retrieval, and tracing of the history of video translation tasks; the log record is used to record in detail each operation step performed by the system from startup to completion of the video translation task, recording when the video input module receives the original video file, what kind of format standardization and preprocessing operations are performed on the file; when the audio extraction module extracts the audio stream from the preprocessed video, and the specific parameters involved in the extraction process.

[0014] The present invention has the following beneficial effects:

[0015] The lip synchronization module in the present invention uses computer vision technology to detect and track key points of the face in the video frame, extracts relevant voice features based on the facial tracking results, and ensures that the lip shape of the character in the translated video is accurately synchronized with the voice content through technical means such as image-to-image feedback loop, so that the translated video looks more natural and smooth, and improves the audience's viewing experience. The audio synthesis module uses the audio processing function of FFMPEG to re-synthesize the generated target language voice with the processed video, which can ensure the quality and compatibility of the synthesis, thereby obtaining a complete translated video file with good audio effect.

[0016] The speech recognition and translation module of the system of the present invention can automatically detect the language type of the audio through the model after deep learning, and can adjust and optimize the corresponding parameters of the speech recognition model according to the voice intonation and vocabulary pronunciation rules of different languages. At the same time, it integrates multiple mainstream and professional translation engines, such as the API interfaces of Google Translate, Baidu Translate, Youdao Translation, etc., as well as professional translation engines in specific fields, to ensure efficient and accurate speech recognition and translation in various language environments, greatly improving the accuracy and comprehensiveness of video content translation, and being able to meet the translation needs of videos in different fields and languages.

[0017] The visualization window of the mobile terminal in the present invention displays the main functions and operation processes of the system in a clear and intuitive manner when the user starts the system for the first time or enters a specific function module, and displays the running progress of each processing stage in real time during the video translation process, so that the user can understand the working status of the system and it is easy to operate. The storage module can not only receive and save the original video files uploaded or imported by the user, but also be responsible for managing and operating them, recording detailed information for subsequent query, retrieval and tracing of the history of video translation tasks. At the same time, the log records in detail each operation step of the system from startup to completion of the video translation task, so that the user can review and troubleshoot the entire translation process, and provide users with a convenient and orderly data management and operation experience as a whole.

[0018] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 This is a flow chart of an automatic video translation system based on FFMPEG, TTS, and Wav2Lip of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 As shown, the present invention is an automatic video translation system based on FFMPEG, TTS, and Wav2Lip, including a video processing unit, a translation synthesis unit, and a mobile terminal:

[0023] The video processing unit includes a video input module, a video acquisition module, a preprocessing module, an audio extraction module and a video extraction module; the video input module is sequentially connected to the video acquisition module, the preprocessing module, the audio extraction module and the video extraction module and connected to the translation synthesis unit; the translation synthesis unit includes a speech recognition translation module, a text-to-speech module, an audio synthesis module, a lip synchronization module, a subtitle retention module and a video output module; the video extraction module is sequentially connected to the speech recognition translation module, the text-to-speech module, the audio synthesis module, the lip synchronization module, the subtitle retention module and finally electrically connected to the video output module;

[0024] The speech recognition and translation module is used to automatically detect the language type of the audio through the model after deep learning, select the appropriate speech recognition model and translation engine, adjust and optimize the parameters of the speech recognition model according to the speech intonation and vocabulary pronunciation rules of different languages, and integrate a variety of professional translation engines, including API interfaces of mainstream translation services such as Google Translate, Baidu Translate, and Youdao Translate, as well as some professional translation engines for specific fields, to ensure efficient and accurate speech recognition in various language environments; the text-to-speech module is used to generate natural and fluent speech according to the text content through the text-to-speech model after deep learning, and make fine adjustments to the intonation, speed, and pause elements of the speech to make it closer to the real human voice expression. It has a variety of voice styles and intonations to choose from, which can not only simulate the voices of different genders, but also automatically generate a matching target language voice style according to the expression style of the original video audio, such as formal, informal, friendly, serious, etc., by analyzing the intonation, speed, emotional color and other characteristics of the original video audio. Users can also adjust the voice style according to their preferences. The module is further customized to meet different application scenarios and personal needs, and the personalized voice output is adjusted according to the voice style and intonation for different scenarios; the audio synthesis module is used to re-synthesize the target language speech generated by the text-to-speech module with the video processed by the video content processing module to obtain a complete video file after translation, and the audio processing function of FFMPEG is used to ensure the quality and compatibility of the synthesis; the lip synchronization module is used to accurately synchronize the lip shape of the characters in the translated video with the voice content, and uses computer vision technology to detect and track the key points of the face in the video frame, mainly the outline and shape of the mouth part. Based on the facial tracking results, the voice features, pitch, voice duration and voice features that describe different lip patterns are extracted. Through the image-to-image feedback loop, each generated image frame is used as input to generate the next frame of the animation, and the lip animation is aligned in time and space to ensure that it is consistent with the facial movements in the original video. The speed, amplitude and details of the animation are adjusted to match the facial movements in the original video; by extracting the position information of the facial feature points, the formula is as follows:

[0025]

[0026] The subtitle retention module is used to dynamically set the appropriate subtitle retention time for each semantic unit based on the above language results, combined with the video playback speed and the average speed at which the general audience reads subtitles; the video output module is used to output the translated video file processed by the audio synthesis module in accordance with the format and parameters specified by the user, which can be used directly by the user or further distributed and disseminated. Before outputting the video, the translated video is subjected to a final quality check to check whether there is any problem of picture quality degradation or audio distortion.

[0027] The video input module is used to receive raw video files in various formats, supports common video formats, and is also compatible with niche but still useful video formats. This module uses FFMPEG video format processing capabilities to perform preliminary analysis and format standardization on the input video; the video acquisition module is used to obtain the input raw video material to be translated, and provide the required raw video data for the subsequent processing of each module in the system, ensuring that the system has accurate and appropriate video content for translation and related processing; the preprocessing module is used to convert the input videos of different formats into a standard format that is easy to process within the system, while ensuring the reasonable adjustment and recording of key video parameters, performing integrity checks on the video stream, and detecting whether there are data The built-in image repair module is used to optimize the processing of some common video defects, such as color deviation and screen flicker, so as to ensure that the quality of the input video meets the basic requirements of subsequent processing; the audio extraction module is used to accurately extract the audio stream from the video processed by the video input module; the video extraction module is used to accurately separate the video stream from the original video file. No matter what common format of video is input, the video stream separation operation can be completed smoothly, and the metadata information of the video, such as shooting time, location, equipment model, etc., can be extracted, and the information can be analyzed. On the one hand, it can be used for subsequent log records and data statistics, and on the other hand, it can provide a reference for other modules in certain specific scenarios.

[0028] The mobile terminal includes a visualization window, a log record and a storage module; the video output module is connected to the visualization window and the storage module network respectively; the visualization window is connected to the log directory and the storage module network respectively; the visualization window is used to show the main functions and operation procedures of the system to the user in a clear and intuitive manner when the user starts the system for the first time or enters a specific functional module. When the system translates the video, the visualization window will display the running progress of each processing stage in real time; the storage module is used to provide the system with a data storage space for receiving and saving the original video files uploaded or imported by the user, and is also responsible for managing the original video files, recording the file name, file size, upload time, and file source of each video file, so as to facilitate subsequent query, retrieval and tracing of the history of video translation tasks; the log record is used to record in detail every operation step performed by the system from startup to completion of the video translation task, recording when the video input module receives the original video file, what kind of format standardization and preprocessing operations are performed on the file; when the audio extraction module extracts the audio stream from the preprocessed video, and the specific parameters involved in the extraction process.

[0029] A specific application of this embodiment is: the user uploads the original video file through the mobile terminal, the system records the file information, the video processing unit uses FFMPEG to parse and standardize the video format, and converts it into a common format that is easy to process within the system. The preprocessing module further adjusts the video parameters, optimizes the picture quality, and separates the audio stream and the video stream. During the operation of the translation and synthesis unit, the audio stream is transmitted to the speech recognition and translation module, which automatically detects the language type and performs speech recognition and translation to generate text content in the target language. The text-to-speech module generates natural and fluent speech based on the text content, and adjusts the intonation, speech speed, etc. The audio synthesis module re-synthesizes the target language speech with the processed video to ensure quality and compatibility, and the lip shape is the same. The first step module accurately synchronizes the lip shape and voice content of the characters in the translated video. The subtitle retention module dynamically sets the appropriate subtitle retention time for each semantic unit. Finally, the video output module will perform a final quality check on the translated video before outputting the translated video file processed by the audio synthesis module in the format and parameters specified by the user. If the test passes, it can be used directly by the user or further distributed and disseminated. The output video file can view the playback progress and other conditions in the visualization window of the mobile terminal. At the same time, the storage module will also perform corresponding record management on the output video file. In addition, the user can view the log records to understand in detail every operation step performed by the system from startup to completion of the video translation task.

[0030] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0031] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.

[0032] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An automatic video translation system based on FFMPEG, TTS, and Wav2Lip, comprising a video processing unit, a translation synthesis unit, and a mobile terminal, characterized in that: The video processing unit includes a video input module, a video acquisition module, a preprocessing module, an audio extraction module and a video extraction module; the video input module is sequentially connected to the video acquisition module, the preprocessing module, the audio extraction module and the video extraction module and connected to the translation synthesis unit; The translation and synthesis unit includes a speech recognition and translation module, a text-to-speech module, an audio synthesis module, a lip synchronization module, a subtitle retention module and a video output module; the video extraction module is sequentially connected to the speech recognition and translation module, the text-to-speech module, the audio synthesis module, the lip synchronization module, the subtitle retention module and finally electrically connected to the video output module; The speech recognition and translation module is used to automatically detect the language type of the audio through the model after deep learning, select the speech recognition model and the translation engine, and optimize the speech recognition model accordingly according to the speech intonation and vocabulary pronunciation rules of different languages; the text-to-speech module is used to generate speech according to the text content through the text-to-speech model after deep learning, and achieve the real human speech expression by adjusting the intonation, speaking speed and pause elements of the speech, and adjust the personalized speech output according to the speech style and intonation for different scenarios; the audio synthesis module is used to re-synthesize the target language speech generated by the text-to-speech module with the video processed by the video content processing module to obtain a translated video file; The lip sync module is used to synchronize the lip shapes of the characters in the translated video with the speech content, using computer vision technology to detect and track the key points of the face in the video frame, extracting speech features, pitch, speech duration and speech features to describe the characteristics of different lip patterns based on the face tracking results, and using each generated image frame as input to generate the next frame of the animation through an image-to-image feedback loop, aligning the lip animation in time and space to make the animation consistent with the facial movements in the original video, and adjusting the speed, amplitude and details of the animation to match the facial movements in the original video; The facial feature points are extracted as follows: n sample images with facial feature points are labeled in the training set. The facial feature point coordinates of each sample image form a vector Xi, and the average shape quantity The calculation formula is: di is the deviation of each sample from the mean shape, and then the covariance matrix C is constructed: The shape vector X is represented as: Where P is a matrix consisting of eigenvectors corresponding to the principal components, and b is a shape parameter vector; When matching the position of feature points in an image, search for the position along the grayscale contour of the image around the feature point. The grayscale point of the contour is I(x, y). The coordinates of a point along the search direction are (x+Δx, y+Δy). The formula for updating the position of the feature point is: ∑[I(x, y)-I(x+Δx, y+Δy)]2; The subtitle retention module is used to dynamically set a suitable subtitle retention time for each semantic unit according to the lip animation alignment result, combined with the video playback speed and the average speed of the audience reading the subtitles; The video output module is used to output the translated video file processed by the audio synthesis module according to the format and parameters specified by the user.

2. The automatic video translation system based on FFMPEG, TTS, and Wav2Lip according to claim 1, characterized in that: The video input module is used to receive the original video file, and the video input module uses the FFMPEG video format processing capability to parse and standardize the input video; The video acquisition module is used to obtain the input original video material to be translated.

3. The automatic video translation system based on FFMPEG, TTS and Wav2Lip according to claim 1, characterized in that: The preprocessing module is used to convert the input videos of different formats into a standard format for internal processing of the system, while ensuring the reasonable adjustment and recording of key video parameters, checking the integrity of the video stream, and optimizing the video defects through built-in image repair; The audio extraction module is used to extract the audio stream from the video processed by the video input module; the video extraction module is used to separate the video stream from the original video file.

4. The automatic video translation system based on FFMPEG, TTS, and Wav2Lip according to claim 1, characterized in that: The mobile terminal includes a visualization window, a log record and a storage module; the video output module is respectively connected to the visualization window and the storage module through a network; the visualization window is respectively connected to the log directory and the storage module through a network.

5. The automatic video translation system based on FFMPEG, TTS and Wav2Lip according to claim 4, characterized in that: The visualization window is used to display the system's functions and operating procedures to the user when the user first starts the system or enters a functional module. When the system translates the video, the visualization window displays the running progress of each processing stage; the storage module is used to receive and save the original video files uploaded or imported by the user, and is also used to manage the original video files and record the file name, file size, upload time, and file source of each video file; the log record is used to record each operation step performed by the system from startup to completion of the video translation task, record the time when the video input module receives the original video file, and perform format standardization and preprocessing operations on the original video file; the audio extraction module is used to extract the audio stream from the preprocessed video.

Images