Automatic dubbing method and system based on emotion mining

Abstract

The invention discloses an automatic dubbing method and system based on emotion mining, and the method comprises the steps: obtaining original video information, uniformly cutting the original video information into a plurality of pieces of sub-segment video information, and extracting the voice audio, background sound audio and silent video of each piece of sub-segment video information; based on the voice audio of each sub-segment video information, classifying and combining subtitle information to obtain new subtitle information, and segmenting the voice audio to obtain a plurality of sub-segment audios; utilizing an emotion recognition model to recognize emotion feature information of the sub-segmented audios and adding the emotion feature information to the corresponding subtitle information; generating voiceprint features of all the sub-segmented audios and classifying the voiceprint features to obtain a basic reference sound of each sub-segmented audio; generating sub-segment dubbing audios with emotional expressions; based on all sub-segment dubbing videos, performing sound effect adjustment on the sub-segment dubbing videos and synthesizing dubbing audios; and synthesizing a dubbing video based on all the dubbing audios. Therefore, high-efficiency, low-cost and high-quality automatic dubbing is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer speech processing, and in particular to an automatic dubbing method and system based on emotion mining. Background Art

[0002] As short videos and short dramas continue to be popular in the domestic and international video markets, video localization is a challenge that many media companies and video platforms need to face. The export and import of a large number of excellent video works cannot avoid the dubbing process, and different companies have adopted completely different strategies for different application scenarios.

[0003] Traditional dubbing uses real-person translation and real-person dubbing, which is suitable for large-scale, high-cost movie dubbing scenes. This dubbing method is natural, smooth, and high-quality, but the participation of a large number of experienced dubbing personnel in the dubbing process results in high costs, low flexibility, and a long production cycle. The new dubbing method uses deep learning technologies such as machine translation, language synthesis, and timbre conversion to achieve automated AI dubbing. This dubbing method is suitable for a large number of low-cost dubbing application scenarios, such as personal short videos, TV series, short dramas, etc.

[0004] The AI ​​dubbing method involves the training and reasoning process of large models, and it also faces many problems at the application level. Individual users can only use software tools to call ready-made dubbing models, which are highly homogeneous in timbre, have a single emotional expression, and are time-consuming and inefficient to operate manually; dubbing companies develop their own large dubbing models, focus on automated processes, and try to solve the efficiency problem of dubbing massive videos, but high-quality and efficient application scenarios are still limited; some application scenarios with fewer characters and simple emotional expressions can already fully adopt automated AI dubbing methods, such as automated dubbing of e-books and promotional educational films; while application scenarios with multiple characters and complex emotional expressions, such as movies, TV series, and short plays, still have problems such as uneven timbre quality and insufficient emotional expression. Summary of the invention

[0005] This application achieves efficient, low-cost, and high-quality automated dubbing by providing an automated dubbing method based on emotion mining.

[0006] This application provides an automated dubbing method based on emotion mining, including:

[0007] S101, obtaining original video information and evenly cutting it into a plurality of sub-segment video information, and respectively extracting the human voice audio, background sound audio, and silent video of each sub-segment video information;

[0008] S102, based on the human voice audio of each sub-segment video information, classify and merge its subtitle information to obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios;

[0009] S103, based on each sub-segment audio, using a pre-trained emotion recognition model, identifying the emotion feature information of the sub-segment audio and adding it to the corresponding subtitle information;

[0010] S104, obtaining character annotations and emotional feature information of all sub-segmented audios, generating voiceprint features of all sub-segmented audios and classifying the voiceprint features, and obtaining a basic reference sound of each sub-segmented audio;

[0011] S105, based on each sub-segment audio, generating a sub-segment dubbing audio with emotional expression;

[0012] S106, based on all the sub-segment dubbing videos, adjusting the sound effects and synthesizing the dubbing audio;

[0013] S107, synthesize the dubbing video based on all the dubbing audios.

[0014] Preferably, the original video information includes video information and subtitle information, the video information includes an audio part and a video part, based on each sub-segment video information, the audio part and the video part thereof are separated to obtain an audio part, a human voice part and a background sound part are extracted from the audio part, and human voice audio and background sound audio of the sub-segment video information are generated;

[0015] The subtitle information of the original video information includes multiple lines of subtitle information, which is expressed as: {[starttm, endtm, subtitle, character], [starttm, endtm, subtitle, character], ...}, where starttm is the start timestamp of each line of subtitles, endtm is the end timestamp, subtitle is the line, and character is the character label.

[0016] Preferably, the S101 specifically includes:

[0017] A1. Using a preset video editing software to cut the original video information according to the timestamp in the subtitle information, a plurality of sub-segment video information and its corresponding subtitle information are obtained;

[0018] A2, separating the audio part and the video part of each sub-segment video information;

[0019] A3. Based on each sub-segment video information, the pre-trained acoustic recognition model is used to separate the background sound part from the human voice part in the audio, and generate the human voice audio and background sound audio of the sub-segment video information.

[0020] Preferably, the S102 specifically includes:

[0021] B1. Using a preset VAD model to monitor the start and end time periods [starttm_r, endtm_r] of all the voiced parts in the human voice audio, a number of voiced audio segments and their corresponding start and end time periods are generated; the start and end timestamps of all the voiced parts in the human voice audio are represented as starttm_r and endtm_r;

[0022] B2. Establishing a corresponding relationship between audio segment and subtitle by determining the intersection and separation relationship between the subtitle time period [starttm, endtm] of the subsegment video information and the start and end time period [starttm_r, endtm_r] of each audio segment, wherein the corresponding relationship includes a one-to-one corresponding relationship or a many-to-one corresponding relationship;

[0023] B3. Based on the correspondence between the audio segment and the subtitle line, the adjacent subtitle lines marked with the same character are merged, and the corresponding lines are connected. The start and end timestamps starttm_r and endtm_r of the audio segment are used to correct the start and end timestamps starttm and endtm of the corresponding subtitle line to achieve subtitle merging and update the subtitle information of the human voice audio in the sub-segment video information;

[0024] B4. Based on the updated subtitle information of the human voice audio, the human voice audio is divided to obtain a plurality of sub-segmented audios: the human voice audio is divided into a plurality of sub-segmented audios according to all the start and end timestamps in the updated subtitle information and the corresponding subtitle information is generated.

[0025] Preferably, the emotional feature information includes an emotional type and a confidence score, and the result annotation is injected into each corresponding line of subtitles to update the subtitle information;

[0026] The updated subtitle information of each line is expressed as: [starttm, endtm, subtitle, character, emotion, score], where emotion is the emotion type and score is the confidence score.

[0027] Preferably, the S104 specifically includes:

[0028] C1. Generate voiceprint features: All sub-segmented audios are aggregated into two levels according to their character annotations and emotion types, and the emotion level and character level are obtained. The character annotations and emotion types are used as inputs, and the pre-trained voiceprint model is called to output the voiceprint features of all sub-segmented audios.

[0029] C2. Calculate the center of voiceprint features: Take the arithmetic average of all voiceprint features in the same level to obtain the center of voiceprint features at each level;

[0030] C3. Filter reference audio: Based on each level, calculate the cosine similarity between the voiceprint features corresponding to all sub-segment audios at that level and the center of the corresponding voiceprint features, sort them in descending order, filter the sub-segment audios with a preset proportion of the top ranking at each level, and splice them into a new audio segment as the basic reference audio for that level.

[0031] Preferably, the S105 specifically includes:

[0032] D1. Generate basic audio: Generate basic audio using the set BaseSpeakerTTS model. The characteristic parameters of the basic audio include lines, dubbing language, emotion type, and speech speed. Generate the voiceprint features of the basic audio and the characteristic constants pre-set for the basic audio.

[0033] D2. Using the preset timbre conversion algorithm, perform timbre conversion on the base audio to obtain the sub-segment dubbing audio: call the convert function of the ToneColorConverter model to perform the timbre conversion operation. The calling method is convert(baseaudio, SE_base, SE_ref), where SE_base and SE_ref represent the original voiceprint features and the reference voiceprint features, respectively.

[0034] Preferably, the preset timbre conversion algorithm includes:

[0035] S1, the original voiceprint feature is calculated according to the following formula:

[0036] SE_base=score*SE_style+(1-score)*SE_baseaudio, where score represents the intensity of emotion conversion and is set according to the actual situation. SE_baseaudio is the voiceprint feature of the basic audio, SE_style is the feature constant pre-set for the basic audio, and SE_base is the original voiceprint feature.

[0037] S2, calculate the reference voiceprint feature according to the following formula:

[0038] SE_ref=0.5*SE_srcsegaudio+0.5*SE_refaudio_emot, wherein SE_srcsegaudio is the voiceprint feature of the sub-segment audio corresponding to the subtitle information, SE_refaudio_emot is the voiceprint feature of the set basic reference sound corresponding to the emotion type in the subtitle information; SE_ref is the reference sound voiceprint feature.

[0039] Preferably, the S106 includes:

[0040] E1. Loudness adjustment: adjust the loudness of the sub-segment dubbing audio and remove the silent audio part at the beginning and end;

[0041] E2. Sound effect adjustment: add corresponding sound effects to the sub-segment dubbing audio for the preset special sound effects;

[0042] E3. Synthesize dubbing audio: Based on the start and end time periods [starttm, endtm] in the subtitle information corresponding to all sub-segment dubbing videos, a preset greedy strategy is used to connect all sub-segment dubbing audios with blank sounds to obtain the final dubbing audio.

[0043] This application also provides an automatic dubbing system based on emotion mining, including:

[0044] Video preprocessing module, subtitle audio processing module, feature mining module, emotional audio generation module, audio post-processing module, video synthesis module; the video preprocessing module is used to obtain the original video information and evenly cut it into a number of sub-segment video information, and respectively extract the human voice audio, background sound audio, and silent video of each sub-segment video information;

[0045] The subtitle audio processing module is used to classify and merge the subtitle information of each sub-segment video information based on the human voice audio, obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios; based on each sub-segment audio, use the pre-trained emotion recognition model to identify the emotional feature information of the sub-segment audio and add it to the corresponding subtitle information;

[0046] The feature mining module is used to obtain character annotations and emotional feature information of all sub-segmented audios, generate voiceprint features of all sub-segmented audios and classify the voiceprint features to obtain a basic reference sound of each sub-segmented audio;

[0047] The emotional audio generation module is used to generate sub-segment dubbing audio with emotional expression based on each sub-segment audio;

[0048] The audio post-processing module is used to adjust the sound effects of all sub-segment dubbing videos and synthesize the dubbing audio;

[0049] The video synthesis module is used to synthesize the dubbing video based on all the dubbing audios.

[0050] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0051] It uses large-scale model technologies such as speech emotion recognition and emotional speech generation, as well as a variety of audio and video processing techniques: pre-processing to extract human voice audio, applying VAD detection technology to merge subtitle segmented audio, automatically identifying the emotional characteristics of the audio and classifying and analyzing the voiceprint features, further generating segmented dubbing audio with rich emotional expression, adding post-processing effects and building a greedy connection strategy, and finally synthesizing high-quality dubbing video; it improves the emotional expression ability of audio, and is suitable for application scenarios with strong and rich emotional expression. By applying a variety of audio and video processing techniques, it improves the stability of the sound quality and improves the overall dubbing effect.

[0052] The automated dubbing method uses automated AI dubbing technology, which greatly improves work efficiency and saves labor costs compared to traditional real-person dubbing methods. It is based on large model technologies such as speech emotion recognition and emotional speech generation, which improves the emotional expression ability of audio compared to other AI dubbing methods and is suitable for application scenarios with strong and rich emotional expressions. At the same time, it also extensively applies audio and video processing techniques such as background sound and human voice separation, VAD detection, emotion intensity control, speech speed and loudness control, and merging strategy control, which improves the stability of the sound quality and improves the overall dubbing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of an automatic dubbing method based on emotion mining according to an embodiment of the present invention;

[0054] Figure 2 It is a structural block diagram of an automatic dubbing system based on emotion mining according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To facilitate the understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0056] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are only for illustrative purposes and do not represent the only implementation method.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items.

[0058] Embodiment 1: Figure 1 It is a flowchart of an automatic dubbing method based on emotion mining according to an embodiment of the present invention.

[0059] like Figure 1 As shown, an automatic dubbing method based on emotion mining includes the following steps:

[0060] S101, obtaining original video information and evenly cutting it into a number of sub-segment video information, and respectively extracting the human voice audio, background sound audio, and silent video of each sub-segment video information.

[0061] Specifically, the original video information includes video information and subtitle information, the video information includes an audio part and a video part, based on each sub-segment video information, its audio part and video part (silent video) are separated to obtain the audio part, the vocal part and the background sound part are extracted from the audio part, and the vocal audio and background sound audio of the sub-segment video information are generated.

[0062] The format of the subtitle information (including multiple lines of subtitle information) of the original video information is expressed as: {[starttm, endtm, subtitle, character], [starttm, endtm, subtitle, character], ...}, where starttm is the start timestamp of each line of subtitles, endtm is the end timestamp, subtitle is the line, and character is the character label.

[0063] Specifically, the S101 includes:

[0064] A1. Using a preset video editing software, the original video information is cut according to the timestamp in the subtitle information to obtain a number of sub-segment video information and its corresponding subtitle information.

[0065] Since the emotional expression of the same character in different video clip information usually fluctuates greatly, in order to focus on exploring the emotional expression of the character in each clip and match each clip with a unique emotional effect, for example, the original video information with a longer duration (such as a movie) is divided into sub-segment video information shorter than 3 minutes, so that the subsequent focus can be placed on the emotional expression of the character in each sub-segment video; based on the subtitle information, by comparing the interval time between the start timestamps starttm, use video editing software (such as ffmpeg) to split the original video information, and by adjusting the timestamp to match the start and end timestamps of the sub-segment video, synchronously generate corresponding subtitle information for each sub-segment video information (by adjusting starttm and endtm and generating corresponding subtitle and character).

[0066] A2. Separate the audio part and the video part of each sub-segment video information.

[0067] For example, video editing software (such as ffmpeg) is called to separate the audio part and the video part of the sub-segment video information to obtain silent video and audio with sound, where the audio with sound includes the human voice part and the background sound part.

[0068] A3. Based on each sub-segment video information, the pre-trained acoustic recognition model is used to separate the background sound part from the human voice part in the audio, and generate the human voice audio and background sound audio of the sub-segment video information.

[0069] For example, based on a deep learning network, a pre-trained acoustic recognition model is used to separate the human voice part and the background sound part in the audio, and obtain clear human voice audio (vocalaudio) and background sound audio (bgmaudio). Among them, the pre-trained acoustic recognition models of human voice, accompaniment, harmony, etc. can use a large amount of audio data and annotate the audio type (human voice audio, background sound audio) as a training data set, and use the training data set to train the pre-built neural network structure to obtain the final acoustic recognition model.

[0070] S102, based on the human voice audio of each sub-segment video information, classify and merge its subtitle information to obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios.

[0071] It should be noted that the human voice audio contains multiple lines of multiple characters (character annotations). A complete sentence may be divided into multiple lines. It is necessary to merge the neighboring segmented lines and segment the entire human voice audio by timestamp as a reference for subsequent dubbing.

[0072] Specifically, step S102 includes:

[0073] B1. Use a preset VAD model (Voice Activity Detection) to monitor the start and end time periods [starttm_r, endtm_r] of all voiced parts in the human voice audio, and generate several voiced audio segments and their corresponding start and end time periods.

[0074] The start and end timestamps (starttm_r and endtm_r) of all the voiced parts in the human voice audio more accurately represent the start and end points of the voiced parts compared to the timestamps (starttm and endtm) in the subtitle information, and will be used to correct the subtitle timestamps.

[0075] B2. By determining the intersection and separation relationship between the line subtitle time period [starttm, endtm] of the sub-segment video information and the start and end time period [starttm_r, endtm_r] of each audio segment, a corresponding relationship between the audio segment and the subtitle is established. That is, a corresponding relationship is established between each line of subtitle information in the subtitle information of the sub-segment video information and all audio segments. Usually, [starttm, endtm] and [starttm_r, endtm_r] are in a one-to-one correspondence or a many-to-one relationship.

[0076] The many-to-one relationship indicates that the audio segment includes multiple lines of subtitles.

[0077] B3. Based on the correspondence between the audio segment and the subtitle line, the adjacent subtitle lines with the same character are merged and connected, and the lines are connected. The start and end timestamps starttm_r and endtm_r of the audio segment are used to correct the start and end timestamps starttm and endtm of the corresponding subtitle lines to achieve subtitle merging and update the subtitle information of the human voice audio in the sub-segment video information.

[0078] The lines are connected in the following way: English is connected with spaces, and Chinese is connected directly; the start time of the merged subtitle information is the start timestamp starttm of the first subtitle line, and the end time is the end timestamp endtm of the last subtitle line. If the first subtitle line is the first in the corresponding time period [starttm_r,endtm_r], starttm needs to be corrected with starttm_r. Similarly, if the last subtitle line is the last in the time period, endtm needs to be corrected with endtm_r. So far, the merged and updated dubbing subtitle information is obtained, the format is as before: [[starttm,endtm,subtitle,character],...], and the lines and start and end times are adaptively adjusted or corrected.

[0079] It should be noted that merging too many subtitle lines (more than 4 lines) will cause the dubbing audio timeline and subtitle timeline to be misaligned, so the maximum number of merged lines is set to 4. In addition, if the subtitle has accurate punctuation marks, you can also merge multiple subtitle lines according to sentence integrity. Sentence integrity focuses on whether the line ends as a sentence. For example, in English, you can refer to the end-of-sentence punctuation.

[0080] B4. Based on the updated subtitle information of the human voice audio, the human voice audio is segmented to obtain several sub-segmented audios: Pydub's cutting function is called to segment the human voice audio into several sub-segmented audios (srcsegaudio) according to all the start and end timestamps in the updated subtitle information and generate the corresponding subtitle information. Each sub-segmented audio corresponds to one subtitle information.

[0081] The sub-segmented audio will be used for subsequent emotion recognition and reference sound selection.

[0082] S103, based on each sub-segment audio, using a pre-trained emotion recognition model, identifying the emotion feature information of the sub-segment audio and adding it to the corresponding subtitle information.

[0083] Specifically, the pre-trained emotion recognition model (deep learning model emotion2vec) is called to obtain the emotion feature information of all sub-segmented audio srcsegaudio, including the emotion type (emotion) and confidence score (score), and the result annotation is injected into each corresponding line of subtitles to update the subtitle information.

[0084] The updated subtitle information format is as follows: [starttm, endtm, subtitle, character, emotion, score]. The emotion types here include but are not limited to angry, happy, excited, fearful, sad, and neutral. The confidence score is a floating point number between 0.0 and 1.0.

[0085] S104, obtaining character annotations and emotional feature information of all sub-segmented audios, generating voiceprint features of all sub-segmented audios and classifying the voiceprint features, and obtaining a basic reference sound of each sub-segmented audio.

[0086] Specifically, the voiceprint features of the sub-segmented audios marked with characters and emotions are further mined for evaluating the timbre quality. Step S104 specifically includes:

[0087] C1. Generate voiceprint features: All sub-segmented audios are aggregated into two levels according to their character labels and emotion types, and the emotion level and character level are obtained. The character labels and emotion types are used as inputs, and the pre-trained voiceprint model (called from the OpenVoice library, for example, ToneColorConverter) is called to output the voiceprint features of all sub-segmented audios.

[0088] Among them, if the number of sub-segmented audio at the emotion level is greater than a preset threshold (for example, the preset threshold is 6), if it is lower than the preset threshold, the voiceprint features are only analyzed at the character level.

[0089] C2. Calculate the voiceprint feature center: Take the arithmetic average of all voiceprint features in the same level to obtain the voiceprint feature center of each level.

[0090] C3. Filter reference audio: Based on each level, calculate the cosine similarity between the voiceprint features corresponding to all sub-segment audios at that level and the center of the corresponding voiceprint features, sort them in descending order, filter the sub-segment audios with a preset proportion of the top ranking at each level, and splice them into a new audio segment as the basic reference audio for that level.

[0091] For example, based on each level, the four sub-segment audios with the highest cosine similarity are connected into a new audio as the basic reference sound of the level (character or emotion), which is marked as refaudio_char for the character level and refaudio_emot for the emotion level.

[0092] For the selection of reference audio for subtitle information corresponding to the sub-segmented audio marked with characters and emotions, the reference audio refaudio_emot of the emotion under the character is given priority. If there is no reference audio refaudio_char for the character, the voiceprint features of all basic reference sounds can be calculated here and saved together with the voiceprint feature results of all previous sub-segmented audios for later use to avoid repeated calculation in the next step.

[0093] Among them, the cosine similarity can be approximately used as a score of the sound quality of the sub-segmented audio. The larger the value of the segmented audio, the more stable the emotion or timbre.

[0094] S105: Generate sub-segment dubbing audio with emotional expression based on each sub-segment audio.

[0095] Specifically, based on the above steps of subtitle merging, emotion recognition, reference sound analysis, etc., each line of subtitle information [starttm, endtm, subtitle, character, emotion, score] and its basic reference sound refaudio_char or refaudio_emot corresponding to the character or emotion level are obtained, and the sub-segment dubbing audio with emotional expression corresponding to the subtitle line is further generated, corresponding to the sub-segment audio.

[0096] Step S105 includes:

[0097] D1. Generate basic audio: First use the set BaseSpeakerTTS model to generate basic audio, for example, single-tone audio of a certain emotion. The characteristic parameters of the basic audio include lines, dubbing language, emotion type, and speaking speed. The voiceprint features of the basic audio include a pre-set constant marked as SE_style.

[0098] Specifically, the tts function of the BaseSpeakerTTS model is called to generate the audio baseaudio. The calling method is tts(subtitle,lang,emotion,speed), where subtitle represents the content of the lines, lang represents the dubbing language (Chinese or English), emotion represents the emotional type, and speed represents the speaking speed (a floating point number between 0.5 and 2.0).

[0099] It should be noted that the difference between the effective duration of the basic audio baseaudio (using VAD detection to remove the silent parts at the beginning and end of the audio) and the subtitle duration (endtm-starttm) of the corresponding sub-segment audio is compared at the same time, and the speed parameter is adjusted to repeat the generation until the duration of the generated audio is close to the subtitle duration. Note that the upper and lower limits of the speed need to be set here. Even if the audio duration may be much larger or much smaller than the subtitle duration, it will no longer be adjusted and will be left to the next step of connecting the dubbing audio for processing.

[0100] D2. Using a preset timbre conversion algorithm, the basic audio is converted to obtain a sub-segment dubbing audio: the convert function of the ToneColorConverter model is called to perform the timbre conversion operation, and the basic audio baseaudio is converted into a sub-segment dubbing audio emotsegaudio that is similar to the timbre of the speaking character.

[0101] Specifically, the calling method is convert(baseaudio,SE_base,SE_ref), where SE_base and SE_ref represent the original voiceprint features and the reference voiceprint features respectively. The preset timbre conversion algorithms include:

[0102] S1, the original voiceprint feature is calculated according to the following formula:

[0103] SE_base=score*SE_style+(1-score)*SE_baseaudio

[0104] Among them, score represents the intensity of emotional conversion and is set according to the actual situation. SE_baseaudio is the voiceprint feature of the base audio baseaudio. This calculation formula has been greatly adjusted compared to the OpenVoice API (the original formula only has SE_style, which is a pre-set constant). It is to express different degrees of emotion and greatly improve the diversity of dubbing audio.

[0105] S2, calculate the reference voiceprint feature according to the following formula:

[0106] SE_ref=0.5*SE_srcsegaudio+0.5*SE_refaudio_emot

[0107] Among them, SE_srcsegaudio is the voiceprint feature of the sub-segment audio srcsegaudio corresponding to the subtitle information, SE_refaudio_emot is the voiceprint feature of the basic reference sound refaudio_emot corresponding to the emotional type in the subtitle information, if not, the basic reference sound refaudio_char corresponding to the character level is selected, refaudio_emot ensures the stability of the timbre feature, srcsegaudio highlights the change of the current timbre feature. The weighted average of the weight (0.5, 0.5) is used here, and it can also be adjusted to a weight such as (0.7, 0.3). In this way, a sub-segment dubbing audio with rich emotional expression is generated, marked as emotsegaudio, and the dubbing audio will be synthesized based on these audios in the future.

[0108] S106, based on all the sub-segment dubbing videos, adjusting the sound effects and synthesizing the dubbing audio.

[0109] Specifically, step S106 includes:

[0110] E1. Loudness adjustment: adjust the loudness of the sub-segment dubbing audio emotsegaudio and remove the silent audio part at the beginning and end.

[0111] Specifically, adjust the loudness of the sub-segment dubbing audio emotsegaudio and remove the silent audio parts at the beginning and end. First, call the decibel acquisition function and adjustment function of pydub to adjust the average loudness value of the sub-segment dubbing audio emotsegaudio to the average loudness value of the sub-segment audio srcsegaudio, so that the loudness fluctuation of the dubbing audio is consistent with the original audio, which can enhance the emotional expression effect, such as angry, sad, etc. Then use Pyannote's VAD model to detect the sub-segment dubbing audio emotsegaudio and cut off the silent audio parts at the beginning and end, so that the control of the timeline connecting the sub-segment dubbing audio will be more precise.

[0112] E2. Sound effect adjustment: For special sound effects (such as telephone tones, inner monologues, etc.), add corresponding sound effects to the sub-segment dubbing audio.

[0113] Specifically, in order to add similar sound effects to the dubbing audio in response to some special sound effect situations such as telephone tones, inner monologues, etc., we have preset some special sound effects through the SOX function library. If the configuration file is configured with special sound effect options, these effects will be automatically added to the sub-segment dubbing audio emotsegaudio.

[0114] E3. Synthesize dubbing audio: Based on the start and end time periods [starttm, endtm] in the subtitle information corresponding to all sub-segment dubbing videos, a greedy strategy is adopted to connect all sub-segment dubbing audios with blank sounds to obtain the final dubbing audio.

[0115] Specifically, based on the subtitle time period [starttm, endtm], a greedy strategy is adopted to connect all sub-segment dubbing audios with blank sounds. Since the duration of the sub-segment dubbing audio emotsegaudio is not necessarily consistent with the subtitle duration (endtm-starttm), and there may be space between two lines of subtitles, these are the main factors affecting the connection strategy.

[0116] When the duration of the sub-segment dubbing audio emotsegaudio is greater than the subtitle duration, priority is given to dynamically moving the audio head forward beyond starttm or the tail backward beyond endtm. If there is insufficient space, consider compressing the audio duration.

[0117] For the convenience of discussion, the difference between the duration of the ith subtitle line endtm-starttm and the duration of emotsegaudio is marked as redundtm_i (i is the subscript), and the interval between the two previous and next subtitle lines (the starttm of the i+1th subtitle line minus the endtm of the ith subtitle line) is marked as gaptm_i. Starting from i=0, the sub-segment dubbing audio emotsegaudio is connected. The start and end times of the i-th audio must meet the following rules:

[0118] (a) The starting time is no greater than starttm, the forward extension cannot exceed 0.8 seconds, and it cannot intersect with the end point of the i-1th audio. The duration of the forward extension must be less than half of the audio duration;

[0119] (b) The end time cannot be greater than the starttm of the (i+1)th subtitle, and cannot extend backward by more than 0.8 seconds;

[0120] (c) Calculate the sum of redundtm_i+1(subscript i+1), gaptm_i and redundtm_i and mark it as sumtm_i. If it is greater than zero, the starting time is set to starttm. If it is less than zero, the starting point is extended forward by the absolute value of sumtm_i.

[0121] Thus, the starting and ending positions of the sub-segment dubbing audio on the timeline are obtained. If the actual duration of the sub-segment dubbing audio emotsegaudio is still greater than this interval, the audio is compressed proportionally and placed in this position interval, and the two sub-segment dubbing audios are connected by blank sound.

[0122] It should be noted that the 0.8 seconds in the greedy strategy is an empirical value of the preset setting, that is, the difference between the audio sound time and the subtitle time on the video is 0.8 seconds, which basically does not affect the overall audio-visual effect. This strategy can effectively reduce the compression ratio of the dubbing audio, retain the overall sound quality, and ensure the alignment of audio and subtitles.

[0123] S107, synthesize the dubbing video based on all the dubbing audios.

[0124] Specifically, the video editing software ffmpeg is called to synthesize the generated dubbing audio with the background sound audio, the silent video, and the original dubbing subtitle information to obtain the final dubbing video.

[0125] If the original video is not segmented in step 101, then calling ffmpeg to directly synthesize it will complete the dubbing process of a single episode; otherwise, calling ffmpeg is divided into two steps: the first step is to synthesize the dubbing audio with the background sound audio and the silent video (the result generated by each sub-segment video) into a small dubbing video; the second step is to connect all the small dubbing videos and synthesize the dubbing subtitle files before segmentation into the final dubbing video.

[0126] The voiceprint features mentioned above are a vector representation of the sound features generated by the deep learning model in the OpenVoice library, and the data format is a 256-dimensional floating-point vector. Calculating the arithmetic mean of multiple voiceprint features is to calculate the arithmetic mean between multiple multi-dimensional floating-point vectors. The cosine similarity of two voiceprint features is to calculate the cosine similarity of two multi-dimensional floating-point vectors.

[0127] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0128] It uses large-scale model technologies such as speech emotion recognition and emotional speech generation, as well as a variety of audio and video processing techniques: pre-processing to extract human voice audio, applying VAD detection technology to merge subtitle segmented audio, automatically identifying the emotional characteristics of the audio and classifying and analyzing the voiceprint features, further generating segmented dubbing audio with rich emotional expression, adding post-processing effects and building a greedy connection strategy, and finally synthesizing high-quality dubbing video; it improves the emotional expression ability of audio, and is suitable for application scenarios with strong and rich emotional expression. By applying a variety of audio and video processing techniques, it improves the stability of the sound quality and improves the overall dubbing effect.

[0129] The entire process runs completely in the Linux environment and is fully automated. In actual production, the entire dubbing method can greatly save labor costs and produce high-quality dubbing videos quickly and efficiently.

[0130] The automated dubbing method uses automated AI dubbing technology, which greatly improves work efficiency and saves labor costs compared to traditional real-person dubbing methods. It is based on large model technologies such as speech emotion recognition and emotional speech generation, which improves the emotional expression ability of audio compared to other AI dubbing methods and is suitable for application scenarios with strong and rich emotional expressions. At the same time, it also extensively applies audio and video processing techniques such as background sound and human voice separation, VAD detection, emotion intensity control, speech speed and loudness control, and merging strategy control, which improves the stability of the sound quality and improves the overall dubbing effect.

[0131] Furthermore, an embodiment of the present invention also provides an automatic dubbing system based on emotion mining.

[0132] Figure 2 It is a structural block diagram of an automatic dubbing system based on emotion mining according to an embodiment of the present invention.

[0133] like Figure 2 As shown, the automatic dubbing system based on emotion mining includes: video preprocessing module, subtitle audio processing module, feature mining module, emotion audio generation module, audio post-processing module, and video synthesis module.

[0134] Specifically, the video preprocessing module is used to obtain the original video information and evenly cut it into a number of sub-segment video information, and respectively extract the human voice audio, background sound audio, and silent video of each sub-segment video information;

[0135] The subtitle audio processing module is used to classify and merge the subtitle information of each sub-segment video information based on the human voice audio, obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios; based on each sub-segment audio, use the pre-trained emotion recognition model to identify the emotional feature information of the sub-segment audio and add it to the corresponding subtitle information;

[0136] The feature mining module is used to obtain character annotations and emotional feature information of all sub-segmented audios, generate voiceprint features of all sub-segmented audios and classify the voiceprint features to obtain a basic reference sound of each sub-segmented audio;

[0137] The emotional audio generation module is used to generate sub-segment dubbing audio with emotional expression based on each sub-segment audio;

[0138] The audio post-processing module is used to adjust the sound effects of all sub-segment dubbing videos and synthesize the dubbing audio;

[0139] The video synthesis module is used to synthesize the dubbing video based on all the dubbing audios.

[0140] It should be noted that other specific implementation contents of the embodiments of the present invention may refer to the above-mentioned automatic dubbing method based on emotion mining.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An automated dubbing method based on emotion mining, characterized in that: The method comprises: S101, obtaining original video information and evenly cutting it into a number of sub-segment video information, and respectively extracting the human voice audio, background sound audio, and silent video of each sub-segment video information; S102, based on the human voice audio of each sub-segment video information, classify and merge its subtitle information to obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios; S103, based on each sub-segment audio, using a pre-trained emotion recognition model, identifying the emotion feature information of the sub-segment audio and adding it to the corresponding subtitle information; S104, obtaining character annotations and emotional feature information of all sub-segmented audios, generating voiceprint features of all sub-segmented audios and classifying the voiceprint features, and obtaining a basic reference sound of each sub-segmented audio; S105, based on each sub-segment audio, generating a sub-segment dubbing audio with emotional expression; S106, based on all the sub-segment dubbing videos, adjusting the sound effects and synthesizing the dubbing audio; S107, synthesize the dubbing video based on all the dubbing audios.

2. The automatic dubbing method based on emotion mining as claimed in claim 1, characterized in that: The original video information includes video information and subtitle information, the video information includes an audio part and a video part, based on each sub-segment video information, the audio part and the video part are separated to obtain the audio part, the human voice part and the background sound part are extracted from the audio part, and the human voice audio and the background sound audio of the sub-segment video information are generated; The subtitle information of the original video information includes multiple lines of subtitle information, which is expressed as: {[starttm, endtm, subtitle, character], [starttm, endtm, subtitle, character], ...}, where starttm is the start timestamp of each line of subtitles, endtm is the end timestamp, subtitle is the line, and character is the character label.

3. The automatic dubbing method based on emotion mining as claimed in claim 2, characterized in that: The S101 specifically includes: A1. Using a preset video editing software to cut the original video information according to the timestamp in the subtitle information, a plurality of sub-segment video information and its corresponding subtitle information are obtained; A2, separating the audio part and the video part of each sub-segment video information; A3. Based on each sub-segment video information, the pre-trained acoustic recognition model is used to separate the background sound part from the human voice part in the audio, and generate the human voice audio and background sound audio of the sub-segment video information.

4. The automatic dubbing method based on emotion mining as claimed in claim 3, characterized in that: The S102 specifically includes: B1. Using a preset VAD model to monitor the start and end time periods [starttm_r, endtm_r] of all the voiced parts in the human voice audio, a number of voiced audio segments and their corresponding start and end time periods are generated; the start and end timestamps of all the voiced parts in the human voice audio are represented as starttm_r and endtm_r; B2. Establishing a corresponding relationship between audio segment and subtitle by determining the intersection and separation relationship between the subtitle time period [starttm, endtm] of the subsegment video information and the start and end time period [starttm_r, endtm_r] of each audio segment, wherein the corresponding relationship includes a one-to-one corresponding relationship or a many-to-one corresponding relationship; B3. Based on the correspondence between the audio segment and the subtitle line, the adjacent subtitle lines marked with the same character are merged, and the corresponding lines are connected. The start and end timestamps starttm_r and endtm_r of the audio segment are used to correct the start and end timestamps starttm and endtm of the corresponding subtitle line to achieve subtitle merging and update the subtitle information of the human voice audio in the sub-segment video information; B4. Based on the updated subtitle information of the human voice audio, the human voice audio is divided to obtain a plurality of sub-segmented audios: the human voice audio is divided into a plurality of sub-segmented audios according to all the start and end timestamps in the updated subtitle information and the corresponding subtitle information is generated.

5. The automatic dubbing method based on emotion mining as claimed in claim 1, characterized in that: The emotional feature information includes the emotional type and the confidence score, and the result annotation is injected into each corresponding line of subtitles to update the subtitle information; The updated subtitle information of each line is expressed as: [starttm, endtm, subtitle, character, emotion, score], where emotion is the emotion type and score is the confidence score.

6. The automatic dubbing method based on emotion mining as claimed in claim 5, characterized in that: The S104 specifically includes: C1. Generate voiceprint features: All sub-segmented audios are aggregated into two levels according to their character annotations and emotion types, and the emotion level and character level are obtained. The character annotations and emotion types are used as inputs, and the pre-trained voiceprint model is called to output the voiceprint features of all sub-segmented audios. C2. Calculate the voiceprint feature center: Take the arithmetic average of all voiceprint features in the same level to obtain the voiceprint feature center of each level; C3. Filter reference audio: Based on each level, calculate the cosine similarity between the voiceprint features corresponding to all sub-segment audios at that level and the center of the corresponding voiceprint features, sort them in descending order, filter the sub-segment audios with a preset proportion of the top ranking at each level, and splice them into a new audio segment as the basic reference audio for that level.

7. The automatic dubbing method based on emotion mining as claimed in claim 6, characterized in that: The S105 specifically includes: D1. Generate basic audio: Generate basic audio using the set BaseSpeakerTTS model. The characteristic parameters of the basic audio include lines, dubbing language, emotion type, and speech speed. Generate the voiceprint features of the basic audio and the characteristic constants pre-set for the basic audio. D2. Using the preset timbre conversion algorithm, perform timbre conversion on the base audio to obtain the sub-segment dubbing audio: call the convert function of the ToneColorConverter model to perform the timbre conversion operation. The calling method is convert(baseaudio, SE_base, SE_ref), where SE_base and SE_ref represent the original voiceprint features and the reference voiceprint features, respectively.

8. The automatic dubbing method based on emotion mining as claimed in claim 7, characterized in that: The preset timbre conversion algorithm includes: S1, the original voiceprint feature is calculated according to the following formula: SE_base=score*SE_style+(1-score)*SE_baseaudio, where score represents the intensity of emotion conversion and is set according to the actual situation. SE_baseaudio is the voiceprint feature of the basic audio, SE_style is the feature constant pre-set for the basic audio, and SE_base is the original voiceprint feature. S2, calculate the reference voiceprint feature according to the following formula: SE_ref=0.5*SE_srcsegaudio+0.5*SE_refaudio_emot, wherein SE_srcsegaudio is the voiceprint feature of the sub-segment audio corresponding to the subtitle information, SE_refaudio_emot is the voiceprint feature of the set basic reference sound corresponding to the emotion type in the subtitle information; SE_ref is the reference sound voiceprint feature.

9. The automatic dubbing method based on emotion mining as claimed in claim 4, characterized in that: The S106 includes: E1. Loudness adjustment: adjust the loudness of the sub-segment dubbing audio and remove the silent audio part at the beginning and end; E2. Sound effect adjustment: add corresponding sound effects to the sub-segment dubbing audio for the preset special sound effects; E3. Synthesize dubbing audio: Based on the start and end time periods [starttm, endtm] in the subtitle information corresponding to all sub-segment dubbing videos, a preset greedy strategy is used to connect all sub-segment dubbing audios with blank sounds to obtain the final dubbing audio.

10. An automated dubbing system based on emotion mining, applied to an automated dubbing method based on emotion mining as claimed in any one of claims 1 to 9, characterized in that: include: Video preprocessing module, subtitle audio processing module, feature mining module, emotional audio generation module, audio post-processing module, video synthesis module; The video preprocessing module is used to obtain the original video information and evenly cut it into a number of sub-segment video information, and respectively extract the human voice audio, background sound audio, and silent video of each sub-segment video information; The subtitle audio processing module is used to classify and merge the subtitle information of each sub-segment video information based on the human voice audio, obtain new subtitle information, and segment the human voice audio based on the updated subtitle information to obtain a plurality of sub-segment audios; based on each sub-segment audio, use the pre-trained emotion recognition model to identify the emotional feature information of the sub-segment audio and add it to the corresponding subtitle information; The feature mining module is used to obtain character annotations and emotional feature information of all sub-segmented audios, generate voiceprint features of all sub-segmented audios and classify the voiceprint features to obtain a basic reference sound of each sub-segmented audio; The emotional audio generation module is used to generate sub-segment dubbing audio with emotional expression based on each sub-segment audio; the audio post-processing module is used to adjust the sound effects and synthesize the dubbing audio based on all sub-segment dubbing videos; the video synthesis module is used to synthesize the dubbing video based on all the dubbing audios.

Images