Dubbing method and dubbing system
By extracting audio feature vectors, clustering processing and text-to-speech models, the problem of inconsistent line duration in multi-character dubbing is solved, and automated audio synchronization processing is realized, which eliminates sound overlap and voice conflicts, and improves the sound quality of film and television works and Audience experience.
Patent Information
- Application Number
- CN202411276640.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
AI Technical Summary
In the production of multimedia works such as film and television dramas, cartoons, when multiple characters are dubbing, the length of lines is inconsistent due to differences in speech speed and intonation, resulting in overlapping voices and voice conflicts. The existing manual adjustment methods are time-consuming and labor-intensive and prone to errors, which affects the impact of Quality of the work.
By obtaining the target language subtitle file and original audio, extracting audio feature vectors, performing clustering processing to determine the audio role type, performing line alignment, using the text-to-speech model to generate dubbing audio, and adjusting speech speed and matching time, using personalized voice Synthetic model and digital signal processing technology realize automated audio synchronization processing.
It realizes audio synchronization processing in multi-character dialogue, eliminates sound overlap and voice conflicts, improves the efficiency and quality of dubbing processing, ensures perfect synchronization between audio and video, and enhances the audience's immersive experience.
Smart Images

Figure CN119091895A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of audio processing, and in particular to a dubbing method and system. Background Art
[0002] In the production process of film and television dramas, cartoons, documentaries and other multimedia works, dubbing is a crucial task, especially in multi-character dialogue scenes, the voices of different characters need to be precisely synchronized to ensure that the audience can clearly understand the content of the dialogue.
[0003] In practice, due to the differences in speech speed, intonation and pronunciation habits of each voice actor, the duration of lines of multiple characters is often inconsistent. This mismatch in duration may cause overlapping voices, voice conflicts or asynchrony between dubbing and the picture in the post-processing process, seriously affecting the overall quality of the work.
[0004] In order to solve the problems existing in multi-character dubbing, the existing technology usually relies on manual adjustment, which requires audio engineers to make detailed time adjustments to the recordings of each voice actor so that the voices of different characters do not overlap. However, this manual processing method is not only time-consuming and laborious, but also prone to errors when dealing with complex dialogue scenes, making it difficult to ensure the synchronization and naturalness of the dubbing. In addition, manual adjustment may change the character's tone and emotional expression to a certain extent, resulting in a decrease in the quality of dubbing. Therefore, there is an urgent need for a dubbing method and system to solve the above problems. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides a dubbing method and system.
[0006] The present invention provides a dubbing method, comprising: Acquire a target language subtitle file and an original audio corresponding to the target language subtitle file, and perform feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; Performing clustering processing on the audio feature vectors in the audio feature vector set, and determining the audio role type of each line time axis in the original audio according to the clustering processing result; Aligning the audio role type with the lines in the corresponding time axis of the target language subtitle file to obtain an aligned target language subtitle file; Add the voice audio sequence number of the audio role type in the dialogue time axis corresponding to the original audio to the position of the corresponding dialogue in the aligned target language subtitle file to obtain a dubbing language subtitle file; Based on the text-to-speech model, the lines of each line timeline in the subtitle file of the dubbing language are converted into corresponding dubbing audios, and based on the multiple dubbing audios, a dubbing file corresponding to the original audio in the target voice is generated.
[0007] According to a dubbing method provided by the present invention, the feature extraction of the speech audio corresponding to the time axis of each segment of the lines in the original audio to obtain an audio feature vector set includes: Annotate the speech audio of each section of the dialogue time axis in the original audio with the speech audio sequence number corresponding to the speech audio, and obtain the speech audio annotated with the speech audio sequence number; Extracting Mel-frequency cepstral coefficient features of the speech audios marked with the speech audio serial numbers; The Mel-cepstral coefficient feature is input into an X-Vector model to obtain the audio feature vector set output by the X-Vector model.
[0008] According to a dubbing method provided by the present invention, clustering the audio feature vectors in the audio feature vector set comprises: Based on a hierarchical clustering algorithm, the audio feature vectors in the audio feature vector set are clustered to obtain similarities between the respective audio feature vectors, and the clustering result is obtained according to the similarities and a preset threshold.
[0009] According to a dubbing method provided by the present invention, the method further comprises: Obtain the voice audio sequence number and the audio role type of each line time axis in the dubbing language subtitle file; Acquire original voice audio, wherein the original voice audio is the voice audio corresponding to the voice audio sequence number in the original audio; Obtaining target dubbing audio, wherein the target dubbing audio is the dubbing audio of the original voice audio corresponding to the line time axis in the dubbing language subtitle file; Performing short-time Fourier transform on the original speech audio and the target dubbing audio to obtain corresponding spectrogram features; The spectrogram features are input into a personalized speech synthesis model to obtain the character dubbing audio output by the personalized speech synthesis model, wherein the personalized speech synthesis model is trained based on a Transformer model.
[0010] According to a dubbing method provided by the present invention, the method further comprises: Based on digital signal processing technology, multiple dubbing audios are denoised to obtain multiple dubbing audios after denoising.
[0011] According to a dubbing method provided by the present invention, before generating a dubbing file corresponding to the original audio in the target voice according to the plurality of dubbing audios, the method further comprises: Obtaining a time length ratio result according to time axis length information of each line time axis in the original audio and corresponding length information of the dubbing audio; According to the duration ratio result, the dubbing audio is subjected to linear interpolation processing, or resampled after reducing the sampling rate, to obtain processed dubbing audio.
[0012] The present invention also provides a dubbing system, comprising: An audio acquisition module, used to acquire a target language subtitle file and the original audio corresponding to the target language subtitle file, and perform feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; A processing module, configured to perform clustering processing on the audio feature vectors in the audio feature vector set, and determine the audio role type of each line time axis in the original audio according to the clustering processing result; A matching module, used for aligning the audio role type with the lines in the corresponding time axis of the target language subtitle file to obtain an aligned target language subtitle file; A dubbing subtitle file generation module, used for adding the voice audio sequence number of the audio role type in the line time axis corresponding to the original audio to the position of the corresponding line in the aligned target language subtitle file to obtain a dubbing language subtitle file; The dubbing audio generation module is used to convert the lines of each line time axis in the dubbing language subtitle file into corresponding dubbing audio based on the text-to-speech model, and generate a dubbing file corresponding to the original audio under the target voice based on multiple dubbing audios.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the dubbing methods described above is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the dubbing method described in any one of the above is implemented.
[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the dubbing method described above is implemented.
[0016] The dubbing method and system provided by the present invention can automatically adjust the dubbing speed and accurately match the duration of the lines, and realize audio synchronization processing in multi-role dialogues through intelligent algorithms, thereby eliminating the problems of sound overlap and voice conflict, and greatly improving the efficiency and quality of dubbing processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 A schematic flow chart of the dubbing method provided by the present invention; Figure 2 A schematic diagram of the structure of the dubbing system provided by the present invention; Figure 3 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0020] With the rapid development of the film and television production industry, especially in modern film and television works with high tempo and frequent multi-character dialogues, the demand for efficient and high-quality dubbing processing has become increasingly strong. At present, although some audio processing software has the function of adjusting the speech speed, most of these tools are general and lack optimization specifically for the synchronous processing of multi-character dubbing, which makes it difficult to meet the complex and changing needs in film and television production. Especially when it comes to scenes involving rapid or cross-conversational dialogues among multiple people, for example, in a certain film and television drama, there is a complex dialogue scene involving the interaction of three characters. In the original script, the dialogues of these three characters have different time periods on the timeline. In the dubbing recording, due to the differences in the actors' speech speed, the recorded audio tracks have overlapping and asynchronous problems. Existing audio processing tools are often difficult to cope with, and still require a lot of manual intervention for post-adjustment.
[0021] In multi-character dubbing, different actors often speak at different speeds, resulting in different times for the same lines to be dubbed by different actors. This not only causes sound overlap, but also causes the dubbing to be out of sync with the picture, making it difficult for the audience to hear the dialogue and affecting the viewing experience. Existing manual adjustment methods are inefficient and prone to errors, often affecting the voice actor's tone and emotional expression, resulting in distortion of the final dubbing effect and failure to truly reflect the character's emotions.
[0022] The present invention first analyzes the speech speed of each dubbing character, and adjusts the dubbing speed according to the difference between the length of the character's lines in the script and the actual dubbing time to match the timeline of the original script, ensuring that the emotional expression and tone of the character are not affected after the adjustment of the speech speed. For multi-person dialogue scenes, the dubbing time of each character is analyzed in turn, and after adjusting the speech speed, the lines of each character are matched in time to ensure that the appearance time of each line in the audio is consistent with the time in the original script, thereby avoiding sound overlap.
[0023] Furthermore, after the initial adjustment, the adjusted audio is detected using a synchronization detection algorithm to identify any possible overlapping areas, and further fine-tuning and track staggering are performed to completely eliminate the overlapping phenomenon and ensure the overall smoothness and synchronization of the dubbing. Finally, the adjusted multi-character dubbing is synthesized to generate the final audio output synchronized with the film and television works, ensuring that the audience can experience a natural and smooth dialogue effect when watching.
[0024] The dubbing method provided by the present invention relates to the field of film and television production and digital audio processing technology, in particular to the dubbing of multi-character dialogue scenes in the post-production of film and television dramas, cartoons, documentaries and other multimedia works. The present invention solves the common problems of track overlap and voice conflict in multi-character dubbing through voice analysis, speech rate adjustment, duration matching and audio synchronization technology, so that when multiple characters are talking at the same time, the speech rate of each character's dubbing can be accurately controlled, and the duration of the lines can be adjusted to achieve seamless docking of audio tracks, ensuring that the final output audio is perfectly synchronized with the original video picture, improving the sound quality of film and television works and the audience's immersive experience. Specifically, the dubbing method provided by the present invention is suitable for the fields of film and television post-production, animation dubbing, game sound effect synthesis and advertising audio production.
[0025] Figure 1 A schematic diagram of the process of the dubbing method provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a dubbing method, comprising: Step 101, obtaining a target language subtitle file and the original audio corresponding to the target language subtitle file, and performing feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set.
[0026] In the present invention, the target language subtitle file refers to a subtitle file corresponding to a specific language (such as English, Chinese, etc.), which usually contains the text content of the dialogue or commentary in the video to be dubbed and the time point (timestamp) when they appear. These subtitle files can be extracted from the video or specially made for the video. In the present invention, the subtitle file can be in multiple formats, such as SRT (Sub Rip Subtitle), SSA (Sub Station Alpha), etc.
[0027] The audio part in the original audio video includes all audio elements such as character dialogue, environmental sounds and background music. In this processing flow, the main focus is on the dialogue or commentary part, because they are the main content of the subtitle file. In the present invention, firstly, the voice audio corresponding to each line time axis in the original audio is feature extracted, wherein the line time axis refers to the time period corresponding to each subtitle in the subtitle file, that is, the start time and end time of the subtitle content appearing in the video or audio. By parsing the subtitle file, these time axis information can be obtained.
[0028] Furthermore, according to the timeline of the lines, audio clips corresponding to the time period of the lines are intercepted from the original audio. These clips are the speech audio of the lines. For each segment of speech audio that is intercepted, feature extraction is required, such as short-time Fourier transform (STFT), Mel frequency cepstral coefficient (MFCC), Mel filter bank (Mel Filter Bank) and other methods to extract key information that can represent the speech content from the original audio signal, such as pitch, tone and frequency. These features are represented in the form of vectors, namely audio feature vectors. By extracting features from each segment of speech audio, a set of all the audio feature vectors of the lines is obtained, which will be used in the subsequent dubbing generation process.
[0029] Step 102: clustering the audio feature vectors in the audio feature vector set, and determining the audio role type of each line time axis in the original audio according to the clustering result.
[0030] In the present invention, the samples in the data set are divided into several groups or "clusters" through a clustering algorithm, so that the similarity of samples in the same cluster is high, while the similarity of samples between different clusters is low. Since different speakers have different voice characteristics (such as pitch, timbre, speaking speed, etc.), the present invention clusters the audio feature vectors to group audio clips with similar voice characteristics together.
[0031] After the clustering process is completed, several clusters are obtained, each of which contains a set of similar audio feature vectors. These clusters can represent different speakers or roles in the original audio. Further, the audio role type of each line timeline in the original audio is determined based on the clustering results, and a corresponding role ID is assigned to each audio role type, thereby determining the number of roles in the original audio.
[0032] Step 103, aligning the audio role type with the lines on the corresponding time axis in the target language subtitle file to obtain an aligned target language subtitle file.
[0033] In the present invention, the character dialogues in the audio (i.e., audio character types) are precisely aligned with the lines in the corresponding target language subtitle file in the time axis. First, it is necessary to identify the types of each character in the audio. This process is accomplished by extracting features from the original audio in the above embodiment, thereby clustering the identified sound features, intonations, and speech speeds of different characters.
[0034] Furthermore, a subtitle file that has been translated into the target language is prepared or obtained. This subtitle file contains all the lines that need to be displayed, but in the initial state, these lines may not be time-aligned with the specific lines in the audio. Then, the lines in each time axis in the subtitle file are aligned with the corresponding audio role type, that is, the lines under each time axis in the subtitle file are matched to the role ID, thereby completing the line time axis alignment processing based on the audio role type.
[0035] Step 104, adding the voice audio sequence number of the audio role type in the dialogue time axis corresponding to the original audio to the position of the corresponding dialogue in the aligned target language subtitle file to obtain a dubbing language subtitle file.
[0036] In the present invention, after obtaining the identity of each character in the original audio (i.e., the audio character type) and the specific time point in the audio through the above embodiment, it can be determined that the lines of each character correspond to one or more time periods in the original audio (i.e., the line timeline).
[0037] Further, in the summary of the present invention, each audio clip is assigned a unique serial number or identifier (for example, file name, clip number ID, etc.). This serial number is used to quickly locate a specific line or character speech in the audio file. In this step, it is necessary to associate these serial numbers with the line timeline of each character. In the present invention, after completing the alignment process of the audio role type and the lines of the corresponding timeline in the target language subtitle file, the voice audio serial number corresponding to each character in the original audio is added to the corresponding position of the aligned target language subtitle file. After completing the above steps, the obtained subtitle file not only contains the lines of the target language and their time positions, but also contains the voice audio serial numbers corresponding to the character speeches in the original audio, thereby obtaining a dubbing language subtitle file, which not only provides the translated lines, but also retains information about which lines are spoken by which characters.
[0038] In view of the problem that existing multi-character dubbing processing usually requires a lot of manual adjustments, resulting in low processing efficiency, the present invention greatly reduces the workload of post-production personnel through an automated processing flow, can quickly process multi-character dialogues, shorten the production cycle, and reduce labor costs.
[0039] Step 105, based on the text-to-speech model, convert the lines of each line time axis in the dubbing language subtitle file into corresponding dubbing audio, and generate a dubbing file corresponding to the original audio in the target voice based on the multiple dubbing audios.
[0040] In the present invention, a text-to-speech (TTS) model is used to generate dubbing audio, and these dubbing audios are further combined into a complete dubbing file under the target voice, so as to convert the lines in the translated subtitle file into dubbing with the characteristics of the target language to match the content and emotion of the original audio.
[0041] In the present invention, multiple role audios are merged into a single audio file to ensure that the lines of each role are arranged according to the script timeline, and for the blank segments between adjacent audios, the missing blank segments between the audios are spliced with silent segments on the premise of ensuring the uniform start position of the timeline in the SRT file. Furthermore, the merged audio file is exported to the target format (such as WAV, MP3 or other audio formats), and format conversion and compression are performed to meet the requirements of different platforms to ensure the quality and compatibility of the output audio.
[0042] Optionally, in one embodiment, an audio quality detection model is used to conduct a comprehensive test on the final audio file to classify and evaluate the sound quality, synchronization, and possible audio defects. Some key dialogue scenes or complex dialogue passages can be manually reviewed to ensure that the audio processing meets the highest quality standards. Finally, the final processed audio files and related project files are stored and backed up to ensure the security and integrity of the data, and a detailed report on the dubbing process is generated, including information such as processing steps, algorithm parameters, and adjustment records, so as to record the adjustments and technical parameters of each step, provide a transparent processing process and detailed reference materials, and help post-production personnel trace and review the dubbing process to ensure the controllability and consistency of the final audio quality.
[0043] The text-to-speech model used in the present invention is a technology that can convert text content into natural speech, such as Variational Inference Text-to-Speech (VITS) or Transformer+Vocoder structure models, etc. These models can simulate the pronunciation, intonation and speaking speed of human speech, and convert each line into dubbing audio in an end-to-end manner, or by first generating MFCC features and then generating audio. Through training, the TTS model can learn to convert the input text into natural and fluent speech. After all the lines are converted into dubbing audio, these audio clips need to be combined in the timeline order of the original audio to generate the final dubbing file to ensure seamless connection between the audio clips and overall fluency.
[0044] The dubbing method provided by the present invention can automatically adjust the dubbing speed and accurately match the duration of the lines, and realize audio synchronization processing in multi-role dialogues through an intelligent algorithm, thereby eliminating the problems of sound overlap and voice conflict, and greatly improving the efficiency and quality of dubbing processing.
[0045] Based on the above embodiment, the feature extraction of the speech audio corresponding to the time axis of each segment of the lines in the original audio to obtain an audio feature vector set includes: Annotate the speech audio of each section of the dialogue time axis in the original audio with the speech audio sequence number corresponding to the speech audio, and obtain the speech audio annotated with the speech audio sequence number; Extracting Mel-frequency cepstral coefficient features of the speech audios marked with the speech audio serial numbers; The Mel-cepstral coefficient features are input into an X-Vector model to obtain the audio feature vector set output by the X-Vector model.
[0046] In the present invention, according to the input target language subtitle file, the corresponding speech audio under each line time axis is intercepted from the audio of the original language of the film and television, and the speech audio serial number ID is marked for the speech audio of each time axis, and then each standardized speech audio is preprocessed and Mel-frequency cepstral coefficient features (MFCC) are extracted.
[0047] Furthermore, the extracted Mel-frequency cepstral coefficient features are input into the X-Vector model, and the model generates the feature vector Embedding of each speech audio to form a set of audio feature vectors of the entire original audio about the lines.
[0048] The present invention accurately annotates the speech audio of each segment of the dialogue timeline in the original audio and assigns a serial number, which can ensure that each speech segment is accurately recognized and distinguished, and helps to reduce recognition errors caused by segment overlap or confusion in subsequent processing; at the same time, the Mel-frequency cepstral coefficient features are extracted as the input of audio analysis, which can capture subtle changes in the audio, not only analyze the speech content, but also further identify the identity characteristics of the speaker.
[0049] Based on the above embodiment, the step of clustering the audio feature vectors in the audio feature vector set includes: Based on a hierarchical clustering algorithm, the audio feature vectors in the audio feature vector set are clustered to obtain similarities between the respective audio feature vectors, and the clustering result is obtained according to the similarities and a preset threshold.
[0050] In the present invention, the hierarchical clustering algorithm calculates the similarity between two feature vectors in the entire audio feature vector set, thereby performing adaptive clustering. When the thresholds of different categories reach a certain level, the category merging is stopped, and the number of characters and the character ID (i.e., the audio character type) in the original audio are determined based on the categories obtained by the final clustering. The present invention adopts a hierarchical clustering algorithm, which can gradually merge or split clusters according to the similarity between feature vectors until the preset threshold conditions are reached, so that it can flexibly respond to complex audio scenes, effectively identify and separate the audio features of each character, and ensure the accuracy and rationality of the clustering results.
[0051] Based on the above embodiment, the method further includes: Obtain the voice audio sequence number and the audio role type of each line time axis in the dubbing language subtitle file; Acquire original voice audio, wherein the original voice audio is the voice audio corresponding to the voice audio sequence number in the original audio; Obtaining target dubbing audio, wherein the target dubbing audio is the dubbing audio of the original voice audio corresponding to the line time axis in the dubbing language subtitle file; Performing short-time Fourier transform on the original speech audio and the target dubbing audio to obtain corresponding spectrogram features; The spectrogram features are input into a personalized speech synthesis model to obtain the character dubbing audio output by the personalized speech synthesis model, wherein the personalized speech synthesis model is trained based on a Transformer model.
[0052] In the present invention, the role ID (i.e., audio role type) and audio ID (i.e., voice audio sequence number) corresponding to each segment (timeline) of the lines in the dubbing language subtitle file are obtained, and then the original film and television audio (i.e., original voice audio) corresponding to the audio ID is obtained. Furthermore, the original voice audio of this segment of lines and the target dubbing audio generated based on the original voice audio are used as the data to be processed, and the spectrogram features are respectively converted using the short-time Fourier transform (STFT) to obtain the spectrogram features, and the spectrogram features are input into the personalized speech synthesis model obtained by Transformer training to perform personalized synthetic dubbing of the corresponding role, so as to obtain the dubbing audio that conforms to the timbre of the original role.
[0053] Based on the above embodiment, the method further includes: Based on digital signal processing technology, multiple dubbing audios are denoised to obtain multiple dubbing audios after denoising.
[0054] In the present invention, the digital signal processing (DSP) technology is used for the dubbing audio generated for each segment of dialogue, so as to remove the noise that may be generated when the dubbing audio is generated and improve the clarity of the audio.
[0055] On the basis of the above embodiment, before generating the dubbing file corresponding to the original audio in the target voice according to the plurality of dubbing audios, the method further includes: Obtaining a time length ratio result according to time axis length information of each line time axis in the original audio and corresponding length information of the dubbing audio; According to the duration ratio result, the dubbing audio is subjected to linear interpolation processing, or resampled after reducing the sampling rate, to obtain processed dubbing audio.
[0056] In the present invention, the original timeline information (i.e., timeline length information) of each line in the original audio in the subtitle file and the length information of the generated dubbing audio are extracted, and then, according to the timeline information and the length information, the original audio duration and the generated audio duration are compared under the condition of aligning the start time of the original language audio and the target language audio of the same text content, and the duration ratio is calculated. Based on the calculated duration ratio, a judgment is made. If the duration of the generated dubbing audio is shorter than the duration of the original audio, a linear adjustment function is defined using the librosa library, the audio is linearly interpolated, and the duration of the generated dubbing audio is extended to be close to the duration of the original audio. If the duration of the generated dubbing audio is longer than the duration of the original audio, the generated dubbing audio is resampled by reducing the sampling rate using the librosa or ffmpeg library to appropriately accelerate the playback and shorten the duration of the generated dubbing audio to be close to the duration of the original audio.
[0057] The present invention eliminates common problems such as sound overlap and voice conflict through precise duration matching and track processing, significantly improving the clarity and quality of the final audio, allowing viewers to better understand and enjoy the dialogue content when watching. In the process of adjusting the speech rate, the emotion and tone changes of each character's dubbing can be retained, and even when the dubbing is adjusted in duration, the emotional expression of the character will not be affected, ensuring the naturalness and expressiveness of the dubbing.
[0058] The dubbing system provided by the present invention is described below. The dubbing system described below and the dubbing method described above can be referred to each other.
[0059] Figure 2 A schematic diagram of the structure of the dubbing system provided by the present invention, such as Figure 2As shown, the present invention provides a dubbing system, including an audio acquisition module 201, a processing module 202, a matching module 203, a dubbing subtitle file generation module 204 and a dubbing audio generation module 205, wherein the audio acquisition module 201 is used to acquire a target language subtitle file and the original audio corresponding to the target language subtitle file, and perform feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; the processing module 202 is used to perform clustering processing on the audio feature vectors in the audio feature vector set, and determine the audio role type of each line time axis in the original audio according to the clustering processing result; the matching module 20 3 is used to align the audio role type with the lines on the corresponding time axis in the target language subtitle file to obtain the aligned target language subtitle file; the dubbing subtitle file generation module 204 is used to add the voice audio sequence number of the lines time axis corresponding to the audio role type in the original audio to the position of the corresponding lines in the aligned target language subtitle file to obtain the dubbing language subtitle file; the dubbing audio generation module 205 is used to convert the lines on the time axis of each segment of the lines in the dubbing language subtitle file into corresponding dubbing audio based on the text-to-speech model, and generate the dubbing file corresponding to the original audio under the target voice according to the multiple dubbing audios.
[0060] The dubbing system provided by the present invention can automatically adjust the dubbing speed and accurately match the duration of the lines, and realize audio synchronization processing in multi-role dialogues through intelligent algorithms, thereby eliminating the problems of sound overlap and voice conflict, and greatly improving the efficiency and quality of dubbing processing.
[0061] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.
[0062] Figure 3 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 3As shown, the electronic device may include: a processor (Processor) 301, a communication interface (Communications Interface) 302, a memory (Memory) 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. The processor 301 can call the logic instructions in the memory 303 to execute the dubbing method, which includes: obtaining a target language subtitle file and the original audio corresponding to the target language subtitle file, and extracting features of the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; clustering the audio feature vectors in the audio feature vector set, and determining the audio role type of each line time axis in the original audio according to the clustering result; aligning the audio role type with the line of the corresponding time axis in the target language subtitle file to obtain the aligned target language subtitle file; adding the speech audio sequence number of the line time axis corresponding to the audio role type in the original audio to the position of the corresponding line in the aligned target language subtitle file to obtain a dubbing language subtitle file; based on the text-to-speech model, converting the lines of each line time axis in the dubbing language subtitle file into corresponding dubbing audio, and generating a dubbing file corresponding to the original audio under the target voice according to the multiple dubbing audios.
[0063] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0064] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the dubbing method provided by the above methods, the method comprising: obtaining a target language subtitle file and the original audio corresponding to the target language subtitle file, and performing feature extraction on the speech audio corresponding to the time axis of each segment of the lines in the original audio to obtain an audio feature vector set; performing clustering processing on the audio feature vectors in the audio feature vector set, and determining the audio feature vector according to the clustering processing result. The method comprises the following steps: extracting the audio role type of each line time axis in the original audio; aligning the audio role type with the lines on the corresponding time axis in the target language subtitle file to obtain the aligned target language subtitle file; adding the voice audio sequence number of the line time axis corresponding to the audio role type in the original audio to the position of the corresponding line in the aligned target language subtitle file to obtain the dubbing language subtitle file; based on the text-to-speech model, converting the lines on the time axis of each segment of the line in the dubbing language subtitle file to the corresponding dubbing audio, and generating the dubbing file corresponding to the original audio under the target voice according to the multiple dubbing audios.
[0065] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to perform the dubbing method provided by the above embodiments when the computer program is executed, the method comprising: obtaining a target language subtitle file and an original audio corresponding to the target language subtitle file, and performing feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; clustering the audio feature vectors in the audio feature vector set, and determining the audio role type of each line time axis in the original audio according to the clustering result; aligning the audio role type with the line on the corresponding time axis in the target language subtitle file to obtain an aligned target language subtitle file; adding the speech audio sequence number of the line time axis corresponding to the audio role type in the original audio to the position of the corresponding line in the aligned target language subtitle file to obtain a dubbing language subtitle file; based on a text-to-speech model, converting the lines on each line time axis in the dubbing language subtitle file into corresponding dubbing audio, and generating a dubbing file corresponding to the original audio under the target voice according to the plurality of dubbing audios.
[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A dubbing method, characterized in that: include: Acquire a target language subtitle file and an original audio corresponding to the target language subtitle file, and perform feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; Performing clustering processing on the audio feature vectors in the audio feature vector set, and determining the audio role type of each line time axis in the original audio according to the clustering processing result; Aligning the audio role type with the lines in the corresponding time axis of the target language subtitle file to obtain an aligned target language subtitle file; Add the voice audio sequence number of the audio role type in the dialogue time axis corresponding to the original audio to the position of the corresponding dialogue in the aligned target language subtitle file to obtain a dubbing language subtitle file; Based on the text-to-speech model, the lines of each line timeline in the subtitle file of the dubbing language are converted into corresponding dubbing audios, and based on the multiple dubbing audios, a dubbing file corresponding to the original audio in the target voice is generated.
2. The dubbing method according to claim 1, characterized in that: The feature extraction of the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set includes: Annotate the speech audio of each section of the dialogue time axis in the original audio with the speech audio sequence number corresponding to the speech audio, and obtain the speech audio annotated with the speech audio sequence number; Extracting Mel-frequency cepstral coefficient features of the speech audios marked with the speech audio serial numbers; The Mel-cepstral coefficient features are input into an X-Vector model to obtain the audio feature vector set output by the X-Vector model.
3. The dubbing method according to claim 1, characterized in that: The clustering of the audio feature vectors in the audio feature vector set includes: Based on a hierarchical clustering algorithm, the audio feature vectors in the audio feature vector set are clustered to obtain similarities between the respective audio feature vectors, and the clustering result is obtained according to the similarities and a preset threshold.
4. The dubbing method according to claim 1, characterized in that: The method further comprises: Obtain the voice audio sequence number and the audio role type of each line time axis in the dubbing language subtitle file; Acquire original voice audio, wherein the original voice audio is the voice audio corresponding to the voice audio sequence number in the original audio; Obtaining target dubbing audio, wherein the target dubbing audio is the dubbing audio of the original voice audio corresponding to the line time axis in the dubbing language subtitle file; Performing short-time Fourier transform on the original speech audio and the target dubbing audio to obtain corresponding spectrogram features; The spectrogram features are input into a personalized speech synthesis model to obtain the character dubbing audio output by the personalized speech synthesis model, wherein the personalized speech synthesis model is trained based on a Transformer model.
5. The dubbing method according to claim 1, characterized in that: The method further comprises: Based on digital signal processing technology, multiple dubbing audios are denoised to obtain multiple dubbing audios after denoising.
6. The dubbing method according to claim 1, characterized in that: Before generating a dubbing file corresponding to the original audio in the target voice according to the plurality of dubbing audios, the method further comprises: Obtaining a time length ratio result according to time axis length information of each line time axis in the original audio and corresponding length information of the dubbing audio; According to the duration ratio result, the dubbing audio is subjected to linear interpolation processing, or resampled after reducing the sampling rate, to obtain processed dubbing audio.
7. A dubbing system, characterized in that: include: An audio acquisition module, used to acquire a target language subtitle file and the original audio corresponding to the target language subtitle file, and perform feature extraction on the speech audio corresponding to each line time axis in the original audio to obtain an audio feature vector set; A processing module, configured to perform clustering processing on the audio feature vectors in the audio feature vector set, and determine the audio role type of each line time axis in the original audio according to the clustering processing result; A matching module, used for aligning the audio role type with the lines in the corresponding time axis of the target language subtitle file to obtain an aligned target language subtitle file; A dubbing subtitle file generation module, used for adding the voice audio sequence number of the audio role type in the line time axis corresponding to the original audio to the position of the corresponding line in the aligned target language subtitle file to obtain a dubbing language subtitle file; The dubbing audio generation module is used to convert the lines of each line time axis in the dubbing language subtitle file into corresponding dubbing audio based on the text-to-speech model, and generate a dubbing file corresponding to the original audio under the target voice based on multiple dubbing audios.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the dubbing method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the dubbing method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the dubbing method according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Dubbing method and device, electronic equipment and storage medium
CN119815149A
A dubbing method, apparatus, electronic device and storage medium
CN119815149B
