Automatic dubbing method and system based on generative AI

Abstract

The invention discloses an automatic dubbing method and system based on generative AI. The method comprises the following steps: separating a human voice audio, a background voice audio and a silent video part from an original video, automatically identifying and translating text content in the audio, and combining a translated text with a corresponding time period to generate a translated subtitle text; segmenting the voice audio into segments, marking character attributes for each segmented segment according to voiceprint features, and evaluating the voice quality at the same time; converting the segmented subtitle text into timbre audios of a target language by using a TTS technology, and connecting and merging the audios according to the starting and ending time of the subtitles; and finally, synthesizing the generated target audio with the background sound audio and the silent video to obtain a complete target video. The above steps efficiently complete dubbing work of multiple languages, multiple roles and multiple scenes, and improve dubbing quality and dubbing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer speech processing, and in particular to an automatic dubbing method and system based on generative AI. Background Art

[0002] With the wave of globalization, the demand for cross-language communication is growing. As an intuitive and effective way to convey information, video is increasingly required to cross language barriers, so video translation and dubbing has become an important field. Traditional translation and dubbing work uses manual translation and real-person dubbing. Although the sound is natural and smooth and the overall effect is exquisite, it is extremely costly, inflexible, and has a long production cycle. It can only be used in small-scale high-quality video translation and dubbing application scenarios. In addition, many companies use TTS speech synthesis technology to try automatic dubbing of videos. Although the cost is low and the efficiency is high, the timbre characteristics are single, the tone and emotion are monotonous, and the effect is very poor.

[0003] With the rapid development of generative AI big models in natural language processing, images, audio, video and other sub-fields in recent years, the field of video dubbing has also ushered in new breakthroughs. The AI ​​big model's superb reasoning and generation capabilities can more accurately identify the voice content of the audio and generate more fluent and natural content text or translation text. Furthermore, the speech synthesis, conversion and other functions of speech-based generative AI can generate high-quality dubbing audio for translated texts in multiple languages.

[0004] In summary, both the traditional manual dubbing method and the TTS speech synthesis dubbing method have the problem of not being able to achieve both cost, efficiency, and sound quality. Based on the generative AI model, corresponding methods and systems are provided in the field of automated dubbing, which can complete automated high-quality dubbing in multiple languages, multiple roles, and multiple scenarios, and have significant advantages in accuracy, fluency, efficiency, etc. In practical applications, the automated dubbing system can complete batch and personalized dubbing work, greatly saving labor costs. Summary of the invention

[0005] In response to the deficiencies of the above-mentioned prior art, the embodiments of the present application provide an automated dubbing method and system based on generative AI. The method and system combine generative AI technology and a variety of audio processing techniques, use the MDX network model to separate human voice and background sound audio, use speech recognition related models to generate translated subtitle text, analyze and mark the voiceprint features of the audio, generate segmented audio by applying TTS speech synthesis and timbre conversion models, and finally use connection synthesis technology to generate a high-quality target video.

[0006] The present invention provides an automatic dubbing method based on generative AI, which comprises the following steps: S101, separating the original video into human voice audio, background sound audio and silent video; S102, automatically identifying and translating the text content of the audio, and merging the translated language text and the time period into a translated subtitle text; S103, segmenting the human voice audio, and marking the person's attributes according to the voiceprint features of the segmented segments; S104, calculating the voiceprint features of the segmented audio to evaluate the sound quality of each segmented audio; S105, converting each line of segmented subtitle text into segmented generated text corresponding to the target language, and generating corresponding timbre audio using TTS technology; S106, generating a timbre-converted audio of the segmented text; S107, connecting and merging all the segmented audios after the timbre conversion according to the start and end time of each line of the segmented subtitle text of the translated subtitle text; S108, synthesizing the previously generated target audio with the background audio and the silent video through video editing software to obtain a final target video.

[0007] Preferably, the step S107, connecting and merging all the segmented audios after the timbre conversion, includes: S201, extracting audio features from each segment of audio after timbre conversion; S202, identifying audio segments with special tones, abnormal emotional points, and abnormal speech speeds based on the extracted features, and marking these locations with special marks; S203, placing the segmented audio in the [starttm, endtm] time period of the time axis, and matching the start time of the audio special mark with the start time of the video segment at the special mark position; S204, aligning the start time of the special audio segment of the original audio with the start time of the special audio mark, correcting and adjusting the time difference, and connecting the segments with blank sounds.

[0008] 3. The method for automatic dubbing based on generative AI according to claim 1, characterized in that the step S102, merging the translated language text and the time period into a translated subtitle text, comprises: The translated language text and the corresponding time period are combined into translated subtitle text lines by lines, in the format of [[starttm,endtm,subtitle],[starttm,endtm,subtitle],...], where each line of subtitles contains the line subtitle, the start time starttm and the end time endtm.

[0009] Preferably, the S103 specifically includes: Call the segmentation model Segmentation in the Python package Pyannote to process the human voice audio to obtain the start and end times of all the voiced parts; Use Pydub's segmentation function to split the audio into small segments based on these start and end times, marked as srcsegaudio; Call Pyannote's voiceprint model to extract voiceprint features for each audio segment; Apply a clustering algorithm to these voiceprint features to form multiple clusters, and mark the audio segments in the same cluster with the same character attribute label, thereby marking the character attributes of all audio segments; Merge the start and end times and character attribute labels generated in the first two steps with the translated subtitle text in step S102; Compare the intersection of the start and end times obtained in the first step with the start and end times of each line of subtitles to complete the one-to-one correspondence of segments or the merging of adjacent segments.

[0010] Preferably, completing the segment merging includes: The final merged time is based on the start and end time of the first step, and the label is added to each line of subtitles. The updated format is as follows: [[starttm,endtm,subtitle,label],[starttm,endtm,subtitle,label],…].

[0011] Preferably, the S104 specifically includes: Segmented audio is aggregated according to different character attribute labels, and the voiceprint model in the Coqui-ai / TTS library is called to generate voiceprint features of all segmented audio. The arithmetic average of all voiceprint features with the same label is taken to obtain the voiceprint feature center of each label. Calculate the cosine similarity between all voiceprint features and the center of the voiceprint feature under each label, and sort the segmented audio under each label in descending order according to the cosine similarity. The cosine similarity can be approximated as the score of the sound quality of the segmented audio; Filter out the first five audio segments under each label, use Pydub to connect and generate a new audio segment as the basic audio labelaudio of the character label, and save the voiceprint feature results of all audio segments.

[0012] Preferably, the step S201 extracts audio features, including but not limited to speech speed, pitch, and volume, including: By performing speech recognition on the audio signal, the speech signal is converted into text, and then the number of syllables or phonemes in the text is counted and divided by the length of the corresponding time period to obtain the speech rate feature; By performing fundamental frequency detection or pitch tracking on the audio signal, the fundamental frequency sequence of the audio signal, i.e., the pitch feature, is obtained; The volume characteristics of the audio signal are obtained by dividing the audio signal into frames and then calculating the sum of the square amplitudes or the energy value of each frame signal.

[0013] Preferably, the S204 includes: The start time of the special mark is extracted from the audio and video, and they are compared to obtain the calculated time difference. The blank sound in the audio is adjusted according to the time difference. After the blank sound adjustment, the audio and video are synchronized and checked.

[0014] Preferably, adjusting the blank sound in the audio according to the calculated time difference includes: If the time difference is positive, it means that the audio is slower than the video and the length of the blank sound needs to be shortened; if the time difference is negative, it means that the audio is faster than the video and the length of the blank sound needs to be increased.

[0015] This application also provides an automated dubbing system based on generative AI, including: The audio extraction module is used to separate the original video into three parts: human voice audio, background sound audio and silent video; The automatic speech recognition module is used to automatically recognize and translate the speech content of human voice audio and generate translated subtitle text in the target language; The speech segmentation and calculation module is used to divide the human voice audio into small segments, mark the character attributes of the segmented audio, further calculate the voiceprint features of the segmented audio with the same character attributes, and evaluate the voice quality of each character; The TTS module is used to generate segmented audio in the target language corresponding to each line of segmented subtitle text; The timbre conversion module is used to obtain new segmented audio by timbre conversion of segmented audio; The video synthesis module is used to connect all the segmented audios to synthesize the target audio, and further synthesize the final target video with the background audio and silent video.

[0016] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By introducing a series of automated processing steps, the application scenarios of traditional manual translation and real-person dubbing have been completely changed, the work efficiency of the entire dubbing process has been greatly improved, and labor costs have been saved; compared with the TTS speech synthesis dubbing method, this method and system, due to the use of generative AI related technologies, can complete multi-language, multi-role, and multi-scene automated high-quality dubbing, greatly improving the speaker's timbre, tone, emotion and other sound effects. Furthermore, the automated dubbing system improves the dubbing efficiency of a large number of video works in various application scenarios, provides a powerful system tool for video-related content communication, and opens a window for people from different language backgrounds to share information and culture.

[0017] By accurately matching the time periods and dynamically adjusting the positions of the segmented audio, the dubbing audio is ensured to be highly consistent with the action of the characters opening their mouths to speak in the video. This greatly enhances the audience's audio-visual experience and makes the dubbing more natural and smooth. Since the start and end times of the subtitle files correspond to the start and end times of the sounds in the original audio, the subtitles and dubbing audio can be perfectly synchronized, avoiding the trouble caused by the subtitles and audio being out of sync during viewing. Through voiceprint feature extraction, timbre conversion and other technologies, the generated dubbing audio can better restore the characteristics of the original speaker in terms of timbre, tone, emotion, etc., thus achieving a professional-level dubbing effect.

[0018] By using the technology of deep analysis of audio features, it is possible to accurately identify audio segments containing special tones, abnormal emotions, and changes in speech speed, and to apply special marks to these important nodes, which greatly facilitates the subsequent fine-tuning and adjustment of these special segments. In terms of strategy design, this solution not only ensures the precise alignment between special marks of audio and video, but also cleverly combines greedy algorithms and time threshold adjustment to process audio segments at non-special mark positions. This comprehensive strategy not only maintains strict synchronization of key positions, but also significantly improves the efficiency and flexibility of the overall synchronization processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an automated dubbing method based on generative AI according to an embodiment of the present invention; Figure 2 The present invention is a block diagram of an automatic dubbing system based on generative AI according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items.

[0021] Embodiment 1: Figure 1 This is a flow chart of an automated dubbing method based on generative AI according to an embodiment of the present invention.

[0022] like Figure 1 As shown, an automatic dubbing method based on generative AI specifically includes the following steps: S101, separating the original video into human voice audio, background sound audio and silent video, and obtaining a clear original human voice after removing the background sound.

[0023] Specifically, first use common audio and video editing software to separate the audio part and the video part of the original video. The separated video part has no sound at all, that is, the silent video is marked as silentvideo, and the other part is the audio part, including the human voice part (only contains human voice) marked as vocalaudio and the background sound part (background sound audio includes accompaniment, harmony, etc.) marked as bgmaudio; Take ffmpeg to separate video and audio as an example: open the command line or terminal, enter the ffmpeg command instructions on the page to separate video and audio; for example: use the command ffmpeg -i input.mp4 -an output_silentvideo.mp4 -vn output_audio.wav, where input.mp4 is the original video file, output_silentvideo.mp4 is the separated silent video file, and output_audio.wav is the separated audio file. The audio part and the video part of the original video are separated, and the separated video part has no sound at all, that is, the silent video; The other part of the audio with sound includes the vocal part and the background sound part. The vocal part and the background sound part of the audio with sound are separated to obtain the vocal audio and the background sound audio. Use the various acoustic models in Ultimate Vocal Remover, adjust the parameters of each acoustic model, introduce the acoustic models of vocals, accompaniment, harmony, etc. based on the deep learning network to process the audio with sound in sequence, and obtain completely clear vocal audio. If the audio with sound often contains special situations such as telephone tones, empty ringtones, crying and laughing sounds, one or more rounds of acoustic model processing are required, which will not be described in detail here. The adjustment of specific acoustic model parameters needs to be adjusted iteratively according to the actual separation effect. When using the model, it is necessary to load the pre-trained model weights and input the pre-processed audio data according to the model requirements. The details of the detailed introduction of the deep learning-based vocal separation model can be referred to the existing technology, which will not be repeated in this article.

[0024] S102, automatically identifying and translating the text content of the audio, and merging the translated language text and the time period into a translated subtitle text.

[0025] Specifically, the human voice audio is first converted into original language text using an automatic speech recognition model. Then, the original language text is translated into the translated language text of the target language through a large language model. Finally, the translated language text and the corresponding time period are merged into lines of translated subtitle text in the following format: [[starttm,endtm,subtitle],[starttm,endtm,subtitle],...], where each line of subtitles contains the lines (subtitle), start time (starttm), and end time (endtm). It should be noted that these times may be inconsistent with the actual start and end times of the sound, which may affect the alignment of the target audio and the subtitles. Therefore, these time values ​​need to be further corrected in subsequent steps.

[0026] Among them, there are two optional methods for implementing the above steps. First: first identify the original language text through Whisper, and attach the time period information of the sound, and then construct a dialogue translation instruction through Chatgpt to convert the original language text into a translated language text; second: directly transcribe the human voice audio through Whisper to obtain the translated language text, and attach the time period information of the sound. The former method is basically not restricted by the original and target languages, and the latter method is restricted by the language, but it is particularly effective in some languages. In practical applications, a certain method is selected based on the language and efficiency. For example, the second method will be used in the case of Chinese-English translation.

[0027] S103, segmenting the human voice audio, and marking the person's attributes according to the voiceprint features of the segmented segments.

[0028] Specifically, the first step is to segment the human voice audio. By calling the segmentation model Segmentation in the Python package Pyannote, the human voice audio is processed to obtain the start and end times of all the voiced parts. Then, using the segmentation function of Pydub, the audio is segmented into small segments according to these start and end times, marked as srcsegaudio; The second step is to call Pyannote's voiceprint model to extract voiceprint features for each audio segment. Then, a clustering algorithm is applied to these voiceprint features to form multiple clusters, and the audio segments in the same cluster are marked with the same character attribute label, thereby annotating the character attributes of all audio segments; The third step is to merge the start and end times and character attribute labels generated in the first two steps with the translated subtitle text in step S102. By comparing the intersection of the two time periods (i.e., the start and end times obtained in the first step and the start and end times of each line of subtitles), one-to-one correspondence or merging of adjacent segments is achieved. The final merged time is based on the start and end times of the first step, and the label is added to each line of subtitles. The updated format is as follows: [[starttm,endtm,subtitle,label],[starttm,endtm,subtitle,label],……]; Among them, the Python packages Pyannote and Pydub, the former provides audio analysis, classification, speaker recognition and other functions, and the latter provides audio reading, writing, segmentation, merging and other functions; although starttm and endtm represent the same, the values ​​have been corrected to ensure the alignment of the audio, subtitles and video timelines, thereby improving the overall effect of the target video; The voiceprint feature is a multi-dimensional floating-point vector group in data form, which is represented as a tensor in the deep learning framework Pytorch. Calculate the arithmetic mean or weighted average of multiple voiceprint features, that is, perform corresponding operations between multiple multi-dimensional floating-point vector groups. To calculate the cosine similarity of two voiceprint features, first flatten each multi-dimensional floating-point vector group into a one-dimensional floating-point vector, and then calculate the cosine similarity of the two one-dimensional floating-point vectors; It should be noted that when the intervals between different speakers in the audio are short or the speakers’ emotions vary greatly, the annotated character labels may not be completely accurate, so a small amount of manual correction is required.

[0029] S104, calculating the voiceprint features of the audio segments to evaluate the sound quality of each audio segment.

[0030] Specifically, the segmented audio is aggregated according to different character attribute labels, and the voiceprint model in the Coqui-ai / TTS library is called to generate the voiceprint features of all segmented audio. The arithmetic average of all voiceprint features with the same label is taken to obtain the voiceprint feature center of each label. Then, calculate the cosine similarity between all voiceprint features under each label and the center of the voiceprint feature, and sort the segmented audio under each label in descending order according to the cosine similarity. The cosine similarity can be approximated as the score of the sound quality of the segmented audio; Finally, filter out the first five audio segments under each label, connect them with Pydub to generate a new audio segment as the basic audio labelaudio of the character label, and save the voiceprint feature results of all audio segments for later use. The Coqui-ai / TTS voiceprint model used here is different from the Pyannote voiceprint model in the previous step. First, it is consistent with the subsequent TTS model, and second, it is better in representing features such as timbre and emotion.

[0031] For example, suppose that a conversation video containing multiple characters is being processed, and the separated human voice audio, translated subtitle text, and character attribute labels annotated according to voiceprint features have been successfully obtained through steps S101 to S103; Through step S103, a translated subtitle text with start and end time and character attribute tags is generated, in a format such as: [[0.0,2.5,"Hello,how are you?","CharacterA"],[2.5,5.0,"I'm fine,thank you.","CharacterB"],...]. At the same time, the original audio is segmented using Pyannote and Pydub tools to obtain short human voice audio srcsegaudio classified by character attribute tags; Next, for each character label, all the segmented audios belonging to that character are summarized. For example, for CharacterA, five segmented audios may be obtained. Then, the voiceprint model in the Coqui-ai / TTS library is used to extract the voiceprint features of these segmented audios. These features may cover multiple dimensions such as timbre, pitch, and speaking speed. In order to obtain the "average voice" of each character, the arithmetic mean of all segmented audio voiceprint features of each character was calculated to form the voiceprint feature center of the character. Subsequently, the cosine similarity between the voiceprint feature of each segmented audio and the voiceprint feature center of the corresponding character was calculated. This indicator is used to measure the similarity between two vectors. The higher the value, the more similar it is. Next, the segmented audios of each character were sorted in descending order according to the cosine similarity, so that the audios at the top were most similar to the character's "average voice". Then, for each character, the top 5 segmented audios were selected, which were considered to be the highest quality and most representative segments of the character's voice; Finally, Pydub is used to connect these filtered audios together to generate a new audio as the basic sound labelaudio of the character. For example, for CharacterA, a 5-second basic sound may be obtained. Finally, these generated basic sounds and corresponding voiceprint feature results are saved to the specified location for subsequent use.

[0032] Among them, cosine similarity is a common method to measure the similarity between two vectors. Its value range is [-1,1]. The closer the value is to 1, the more similar the two vectors are. In practical application calculations, the dot product of two vectors (i.e., the voiceprint feature and feature center of the audio segment) can be taken, and then divided by the product of their respective moduli. The result is the cosine similarity.

[0033] S105, converting each line of segmented subtitle text into segmented generated text corresponding to the target language, and generating corresponding timbre audio using TTS technology.

[0034] Specifically, first, for each character label, the candidate roles of the Coqui-ai / TTS model are screened. By calculating the cosine similarity between each character's basic sound labelaudio and the voiceprint features of all candidate roles in the model, the cosine similarity is sorted and selected, that is, the one with the largest cosine similarity, that is, the one with the closest voiceprint features, as the candidate role of the character. Then, the candidate role corresponding to each line of subtitle and label is passed as a parameter to the Coqui-ai / TTS model to generate the corresponding segmented audio ttssegaudio; At present, the timbre characteristics of all generated ttssegaudio are only similar to the candidate characters. Due to the difference between the original language and the translated language, the duration of the segmented audio ttssegaudio should be quite different from the duration of the original segmented audio (endtm-starttm). Therefore, it is also necessary to control the speech speed parameter speed and repeatedly generate ttssegaudio with a duration not much different from the original to achieve good results.

[0035] Among them, the subtitle text is the above-mentioned [starttm, endtm, subtitle, label].

[0036] S106, generating a timbre-converted audio of the segmented text.

[0037] Specifically, first calculate the voiceprint features of the three audio segments of TTS segmented audio ttssegaudio, its corresponding character label's basic sound labelaudio and the original segmented audio srcsegaudio. Then take the weighted average of the voiceprint features of the audios labelaudio and srcsegaudio with a weight of 0.5:0.5 to obtain a new voiceprint feature. The weight ratio can be adjusted according to the actual situation, such as labelaudio accounts for 0.6, srcsegaudio accounts for 0.4, etc. Finally, the voiceprint features of the segmented audios ttssegaudio, ttssegaudio and the new voiceprint features are passed as parameters to the timbre conversion model Coqui-ai / VC to generate a new segmented audio vcsegaudio.

[0038] The purpose of weighted averaging is to obtain the average timbre characteristics of the original speaker and to highlight the tone, emotion and other characteristics of the current segmented audio srcsegaudio. The timbre characteristics of the segmented audio vcsegaudio are similar to those of the original speaker, and the tone and emotion effects are similar to the original segmented audio.

[0039] S107, according to the start and end time of each line of segmented subtitle text of the translated subtitle text, all segmented audios vcsegaudio after timbre conversion are connected and merged.

[0040] Among them, the segmented audio vcsegaudio is placed in the [starttm, endtm] time period of the timeline, and the segments are connected by blank sounds. However, since the speaking speed and rhythm of the translated language may not match those of the original language, the duration of the segmented audio vcsegaudio is usually inconsistent with the subtitle duration endtm-starttm. Direct connection will inevitably result in two mismatches: the blank sound is too long or the segmented audio overlaps. In order to achieve a one-to-one correspondence between the start and end time of the audio (i.e. the speaking time period) and the mouth-opening speaking action in the video, the start and end time starttm and endtm of the translated subtitle text must be used as a reference standard, and the start and end points of the audio cannot deviate too far from starttm and endtm; When the video audio duration is much longer than the subtitle duration, over-compression of the audio duration will result in poor sound quality. If the start and end time of the audio exceeds the set start time (starttm) or the end time (endtm) is short (for example, less than 0.8 seconds), this will basically not affect the overall audio-visual experience of the audience listening to the audio and watching the video subtitles at the same time.

[0041] Therefore, the position of the segmented audio vcsegaudio on the timeline is dynamically adjusted by moving, compressing or stretching it under a time threshold (such as 0.8 seconds). In terms of specific implementation, first calculate the difference between the vcsegaudio duration of each line of subtitles and the subtitle duration endtm-starttm. If the value is a positive number and the larger it is, the greater the compression ratio is.

[0042] Specifically, first calculate the difference between the duration of vcsegaudio of each line of subtitles and the duration of endtm-starttm of the subtitles. If the value is a positive number and the larger it is, the greater the compression ratio is. Then, a greedy strategy is used to determine the maximum duration that the current subtitle line can be moved forward and backward, taking into account the duration difference of the next two lines of subtitles. This duration is the maximum value that the audio can exceed starttm and endtm and does not intersect with the previous and next segmented audio. Finally, starting from the first line of subtitles, the compressed or stretched segmented audio vcsegaudio is connected with blank audio until the target audio is finally synthesized.

[0043] S108, synthesizing the previously generated target audio with the background audio and the silent video through video editing software to obtain a final target video.

[0044] Among them, the final target video can continue to be synthesized using Ffmpeg. If the translated subtitle text is converted into SRT format, it can also be merged into the target video. The starttm and endtm of the subtitle file correspond to the start and end time of the sound in the original audio, that is, the start and end time of the mouth opening action in the video, so the subtitle time corresponds to the video action one by one. Finally, a target video with audio, subtitles, and video fully aligned is obtained.

[0045] It should be noted that the embodiment of the present invention applies multiple generative AI models: the MDX network model is used in the human voice and background sound separation stage; the Whisper language model and the Chatgpt large model are used in the automatic speech recognition stage; and the TTS speech synthesis model and the timbre conversion model are used in the speech synthesis stage.

[0046] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: 1. By introducing a series of automated processing steps, the application scenarios of traditional manual translation and real-person dubbing have been completely changed, the work efficiency of the entire dubbing process has been greatly improved, and labor costs have been saved; compared with the TTS speech synthesis dubbing method, this method and system, due to the use of generative AI related technologies, can complete multi-language, multi-role, and multi-scene automated high-quality dubbing, greatly improving the speaker's timbre, tone, emotion and other sound effects. Furthermore, the automated dubbing system improves the dubbing efficiency of a large number of video works in various application scenarios, provides a powerful system tool for video-related content communication, and opens a window for people from different language backgrounds to share information and culture.

[0047] 2. Through precise time period matching and dynamic adjustment of the position of segmented audio, the dubbing audio is ensured to be highly consistent with the action of the characters opening their mouths to speak in the video. This greatly enhances the audience's audio-visual experience and makes the dubbing more natural and smooth. Since the start and end time of the subtitle file corresponds to the start and end time of the sound in the original audio, the subtitles and dubbing audio can be perfectly synchronized, avoiding the trouble caused by the subtitles and audio being out of sync during the viewing process. Through voiceprint feature extraction, timbre conversion and other technologies, the generated dubbing audio can better restore the characteristics of the original speaker in terms of timbre, tone, emotion, etc., thereby achieving a professional dubbing effect.

[0048] Embodiment 2: The above embodiment 1 completely changes the application scenarios of traditional manual translation and real-person dubbing through audio and video separation, automatic speech recognition, translation, voiceprint feature extraction and timbre conversion, greatly improves the work efficiency of the entire dubbing process, and saves labor costs.

[0049] On this basis, the above step S107 is improved and upgraded to address the problem that the speed and rhythm of the translated language do not match the original language, and further solve the problem of one-to-one correspondence between the start and end time of the audio and the opening and speaking action in the video.

[0050] S201, extracting audio features from each segmented audio vcsegaudio after the timbre conversion, including but not limited to speech speed, pitch, volume, etc.

[0051] Specifically, in audio feature extraction, the audio signal is converted into text by performing speech recognition on the audio signal, and then the number of syllables or phonemes in the text is counted and divided by the corresponding time period length to obtain the speaking rate feature; the audio signal is subjected to fundamental frequency detection or pitch tracking to obtain the fundamental frequency sequence of the audio signal, that is, the tone feature; the audio signal is framed and the sum of the squares of the amplitude or the energy value of each frame signal is calculated to obtain the volume feature of the audio signal.

[0052] In addition to speech rate, pitch and volume features, other features such as spectrum centroid, zero-crossing rate, Mel-frequency cepstral coefficient (MFCC) etc. can be extracted according to actual needs.

[0053] S202, based on the extracted features, identify the audio segments with special tone, abnormal emotional points and abnormal speech speed, and mark these positions with special marks as specialtm.

[0054] S203, the segmented audio vcsegaudio is placed in the [starttm, endtm] time period of the time axis, and the start time of the audio special mark specialtm matches the start time and position of the video segment of the special mark position.

[0055] Among them, for the audio segments at non-specially marked positions, the greedy strategy and time threshold method in the first embodiment are still used for adjustment and connection.

[0056] S204, aligning the start time of the special audio segment of the original audio with the start time of the special audio mark, correcting and adjusting the time difference, and connecting the segments with blank sounds.

[0057] Specifically, first, the start time of the special mark needs to be extracted from the audio and video. In audio, the start time of the special mark can be determined by analyzing the mutation point of audio features (such as speech speed, pitch, volume, etc.); in video, the start time of the special mark can be obtained by analyzing the content or metadata of the video frame; After obtaining the start time of the special mark in the audio and video, you need to compare them and calculate the time difference. This time difference is the error of the blank sound, which needs to be adjusted to ensure the synchronization of audio and video; Adjust the blank sound in the audio based on the calculated time difference. If the time difference is positive, it means that the audio is slower than the video, and the length of the blank sound needs to be shortened; if the time difference is negative, it means that the audio is faster than the video, and the length of the blank sound needs to be increased. The adjusted blank sound should be able to compensate for the time difference so that the audio and video can be played in complete synchronization; After adjusting the blank sound, you need to check the synchronization of the audio and video to ensure that the start time of the special mark in the audio and video is exactly the same. If you find that there is still a synchronization error, you can repeat the above adjustment process until the error is satisfied.

[0058] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages: By deeply analyzing the extracted audio features, the solution can accurately identify audio segments with special tones, abnormal emotional points, and abnormal speech speeds, and mark these key locations with special marks. This helps to optimize and adjust these special segments in a targeted manner in subsequent processing.

[0059] The solution not only takes into account the precise matching of special audio and video tags, but also takes into account the greedy strategy and time threshold method adjustment of audio segments without special tags. This strategy not only ensures the synchronization accuracy of key locations, but also improves the efficiency and flexibility of overall synchronization.

[0060] Embodiment three: Figure 2 It is a block diagram of a composition of an automatic dubbing system based on generative AI according to an embodiment of the present invention.

[0061] like Figure 2 As shown, an automatic dubbing system based on generative AI includes: Audio extraction module: used to separate the original video into three parts: vocal audio vocalio, background audio bgmaudio and silent video.

[0062] Automatic speech recognition module: used to automatically recognize and translate the speech content of human voice audio vocalaudio and generate translated subtitle text in the target language.

[0063] Speech segmentation and calculation module: used to divide the human voice audio vocalio into small segmented audio srcsegaudio, mark the character attributes label of the segmented audio, further calculate the voiceprint features of the segmented audio with the same character attributes, and evaluate the voice quality of each character.

[0064] TTS module: used to generate segmented audio ttssegaudio in the target language corresponding to each line of segmented subtitle text.

[0065] Tone conversion module: used to convert the segmented audio ttssegaudio into a new segmented audio vcsegaudio through tone conversion.

[0066] Video synthesis module: used to connect all segmented audio vcsegaudio to synthesize the target audio, and further synthesize the final target video with the background audio bgmaudio and silent video silentvideo.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An automated dubbing method based on generative AI, characterized in that: The specific steps include: S101, separating the original video into human voice audio, background audio and silent video; S102, automatically identifying and translating the text content of the audio, and merging the translated language text and the time period into a translated subtitle text; S103, segmenting the human voice audio, and marking the person's attributes according to the voiceprint features of the segmented segments; S104, calculating the voiceprint features of the segmented audio to evaluate the sound quality of each segmented audio; S105, converting each line of segmented subtitle text into segmented generated text corresponding to the target language, and generating corresponding timbre audio using TTS technology; S106, generating a timbre-converted audio of the segmented text; S107, connecting and merging all the segmented audios after the timbre conversion according to the start and end time of each line of the segmented subtitle text of the translated subtitle text; S108, synthesizing the previously generated target audio with the background audio and the silent video through video editing software to obtain a final target video.

2. The automated dubbing method based on generative AI as claimed in claim 1, characterized in that: The step S107, connecting and merging all the segmented audios after the timbre conversion, includes: S201, extracting audio features from each segment of audio after timbre conversion; S202, identifying audio segments with special tones, abnormal emotional points, and abnormal speech speeds based on the extracted features, and marking these locations with special marks; S203, placing the segmented audio in the [starttm, endtm] time period of the time axis, and matching the start time of the audio special mark with the start time of the video segment at the special mark position; S204, aligning the start time of the special audio segment of the original audio with the start time of the special audio mark, correcting and adjusting the time difference, and connecting the segments with blank sounds.

3. The automatic dubbing method based on generative AI as claimed in claim 1, characterized in that: The step S102, combining the translated language text and the time period into a translated subtitle text, includes: The translated language text and the corresponding time period are combined into translated subtitle text lines by lines, in the format of [[starttm,endtm,subtitle],[starttm,endtm,subtitle],...], where each line of subtitles contains the line subtitle, the start time starttm and the end time endtm.

4. The automatic dubbing method based on generative AI as claimed in claim 1, characterized in that: The S103 specifically includes: Call the segmentation model Segmentation in the Python package Pyannote to process the human voice audio to obtain the start and end times of all the voiced parts; Use Pydub's segmentation function to split the audio into small segments based on these start and end times, marked as srcsegaudio; Call Pyannote's voiceprint model to extract voiceprint features for each audio segment; Apply a clustering algorithm to these voiceprint features to form multiple clusters, and mark the audio segments in the same cluster with the same character attribute label, thereby marking the character attributes of all audio segments; Merge the start and end times and character attribute labels generated in the first two steps with the translated subtitle text in step S102; Compare the intersection of the start and end times obtained in the first step with the start and end times of each line of subtitles to complete the one-to-one correspondence of segments or the merging of adjacent segments.

5. The automatic dubbing method based on generative AI as claimed in claim 4, characterized in that: Complete segment merging, including: The final merged time is based on the start and end time of the first step, and the label is added to each line of subtitles. The updated format is as follows: [[starttm,endtm,subtitle,label],[starttm,endtm,subtitle,label],…].

6. The automatic dubbing method based on generative AI as claimed in claim 5, characterized in that: The S104 specifically includes: Segmented audio is aggregated according to different character attribute labels, and the voiceprint model in the Coqui-ai / TTS library is called to generate voiceprint features of all segmented audio. The arithmetic average of all voiceprint features with the same label is taken to obtain the voiceprint feature center of each label. Calculate the cosine similarity between all voiceprint features and the center of the voiceprint feature under each label, and sort the segmented audio under each label in descending order according to the cosine similarity. The cosine similarity can be approximated as the score of the sound quality of the segmented audio; Filter out the first five audio segments under each label, use Pydub to connect and generate a new audio segment as the basic audio labelaudio of the character label, and save the voiceprint feature results of all audio segments.

7. The automatic dubbing method based on generative AI as claimed in claim 2, characterized in that: The step S201 extracts audio features, including but not limited to speech speed, pitch, and volume, including: By performing speech recognition on the audio signal, the speech signal is converted into text, and then the number of syllables or phonemes in the text is counted and divided by the length of the corresponding time period to obtain the speech rate feature; By performing fundamental frequency detection or pitch tracking on the audio signal, the fundamental frequency sequence of the audio signal, i.e., the pitch feature, is obtained; The volume characteristics of the audio signal are obtained by dividing the audio signal into frames and then calculating the sum of the square amplitudes or the energy value of each frame signal.

8. The method for automatic dubbing based on generative AI as claimed in claim 2, characterized in that: The S204 includes: The start time of the special mark is extracted from the audio and video, and the time difference is calculated by comparing them. The blank sound in the audio is adjusted according to the time difference. After the blank sound adjustment, the audio and video are synchronized and checked.

9. The method for automatic dubbing based on generative AI as claimed in claim 8, characterized in that: According to the calculated time difference, the blank sound in the audio is adjusted, including: If the time difference is positive, it means that the audio is slower than the video and the length of the blank sound needs to be shortened; if the time difference is negative, it means that the audio is faster than the video and the length of the blank sound needs to be increased.

10. An automated dubbing system based on generative AI, characterized in that: include: The audio extraction module is used to separate the original video into three parts: human voice audio, background sound audio and silent video; The automatic speech recognition module is used to automatically recognize and translate the speech content of human voice audio and generate translated subtitle text in the target language; The speech segmentation and calculation module is used to divide the human voice audio into small segments, mark the character attributes of the segmented audio, further calculate the voiceprint features of the segmented audio with the same character attributes, and evaluate the voice quality of each character; The TTS module is used to generate segmented audio in the target language corresponding to each line of segmented subtitle text; The timbre conversion module is used to obtain new segmented audio by timbre conversion of segmented audio; The video synthesis module is used to connect all the segmented audios to synthesize the target audio, and further synthesize the final target video with the background audio and silent video.

Images