A method, device, equipment and storage medium for audio and video lip translation

Abstract

The present application discloses a method, device, equipment and storage medium for audio and video lip translation, which relates to the field of video translation, including: determining the motion trajectory of each character ID to determine the audio to be translated corresponding to each character ID; performing speech recognition on the audio to be translated to obtain the text to be translated and the current translated text to generate the current translated audio, and judging whether the audio duration of the current translated audio meets the preset re-translation condition; if it meets, re-translating to obtain the new current translated text, and jumping to the step of generating the current translated audio; if it does not meet, audio alignment is performed on the current translated audio based on the audio to be translated to obtain the target translated audio; synthesizing each target translated audio to obtain the target audio, generating each lip picture and generating the target video based on the target audio and reference frame, and merging the target video with the target audio to obtain the translated audio and video. The present application realizes lip translation of the audio and video to be translated.

Description

Technical Field

[0001] The present invention relates to the field of video translation, and in particular to an audio and video lip translation method, device, equipment and storage medium. Background Art

[0002] Since different languages ​​require different numbers of notes to express the same content, there will be a difference in the length of the audio before and after the translation. At present, when translating video content into speech, in order to align the content of the original video, the general practice is to perform linear scaling, which leads to differences in speech speed, greatly reducing the user experience. At the same time, after completing the speech translation, the speaker's lip shape in the video needs to be aligned with the new speech. In the traditional digital human solution, the lip synchronization algorithm for a single person in a fixed posture is relatively mature. However, in cross-scene video images for multi-person interaction, the quality of multi-angle lower lip generation is low, and the accuracy of driving target selection in multi-person scenes is also low.

[0003] In summary, how to reduce the speed differences of translated audio, improve the quality of lip generation and the accuracy of driving target selection in multi-scene video images of multi-person interaction, and improve user experience are issues that need to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an audio and video lip translation method, device, equipment and storage medium, which can reduce the speed difference of the translated audio, improve the quality of lip generation and the correct selection rate of driving targets in multi-person interactive cross-scene video images, and improve user experience. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses an audio and video lip translation method, comprising:

[0006] Determine the motion trajectory corresponding to each character ID based on the original video file, and determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model;

[0007] Performing speech recognition on the audio to be translated to obtain a text to be translated, obtaining a current translated text corresponding to the target language based on the text to be translated, generating a corresponding current translated audio based on the current translated text, and determining whether the audio duration of the current translated audio meets a preset re-translation condition;

[0008] If the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated text, and the process jumps to the step of generating the corresponding current translated audio based on the current translated text;

[0009] If the audio duration of the current translated audio does not meet the preset re-translation condition, audio alignment is performed on the current translated audio based on the audio to be translated to obtain an aligned target translated audio;

[0010] The target translated audio files corresponding to the audio files to be translated are synthesized to obtain target audio files, and lip-sync images corresponding to the video frames in the original video file are generated based on the target audio files and the reference frames in the original video file, a target video is generated based on the lip-sync images, and the target video is merged with the target audio files to obtain the translated audio and video files.

[0011] Optionally, before determining the motion trajectory corresponding to each character ID based on the original video file, the method further includes:

[0012] Obtaining an original audio and video file, and using the FFmpeg software package to separate the original audio and video file to obtain the original video file and the original audio file;

[0013] Accordingly, merging the target video with the target audio to obtain translated audio and video includes:

[0014] The target video and the target audio are merged with each other using the FFmpeg software package to obtain the translated audio and video.

[0015] Optionally, determining the motion trajectory corresponding to each character ID based on the original video file includes:

[0016] Using a feature extraction network to compare differences between frames in the original video file, and determining a scene switching timestamp of the original video file based on the differences, so as to divide the original video file into independent storyboard videos based on the scene switching timestamps;

[0017] The DeepFace detection network is used to determine each character ID based on each independent storyboard video, and each character ID is tracked to determine the motion trajectory corresponding to each character ID.

[0018] Optionally, the using a speaker recognition model to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory includes:

[0019] Using a deep neural network to obtain pure human voice audio in the original audio file, and extracting Mel frequency spectrum features of the pure human voice audio to determine audio features corresponding to each frame of the pure human voice audio;

[0020] Determine the utterance time period corresponding to each audio ID based on each of the audio features using the DBSCAN clustering algorithm, and segment the pure human voice audio based on the utterance time period to determine the audio to be translated corresponding to each of the audio IDs;

[0021] Determine a face image set corresponding to each character ID based on each of the motion trajectories and the original video file, and input the clean human voice audio and the face image set into the speaker recognition model to determine the correspondence between each of the audio IDs and each of the character IDs;

[0022] The utterance time period and the audio to be translated corresponding to each character ID are determined based on the audio to be translated corresponding to each audio ID and the corresponding relationship.

[0023] Optionally, generating a corresponding current translated audio based on the current translated text includes:

[0024] The target text-to-speech model is used to generate a current translated audio corresponding to the audio to be translated based on the current translated text; wherein the target text-to-speech model is a text-to-speech model generated based on the audio to be translated using the SoVITS algorithm.

[0025] Optionally, the preset re-translation condition is that the ratio of the target difference to the audio duration of the audio to be translated is greater than a preset percentage threshold; the target difference is the difference between the audio duration of the audio to be translated and the audio duration of the currently translated audio.

[0026] Optionally, the obtaining a current translated text corresponding to the target language based on the text to be translated includes:

[0027] Translating the text to be translated using a translation tool based on a large language model and based on a first translation prompt word to obtain a current translated text corresponding to a target language;

[0028] Correspondingly, if the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated text, including:

[0029] If the audio duration of the currently translated audio meets the preset re-translation condition, the first translation prompt word is optimized accordingly to obtain an optimized second translation prompt word;

[0030] The text to be translated is retranslated using the translation tool based on the second translation prompt word to obtain a new current translated text.

[0031] Optionally, generating each lip-sync picture corresponding to each video frame in the original video file based on the target audio and the reference frame in the original video file includes:

[0032] Extracting speech posterior probability features in the target audio to obtain a corresponding target feature vector;

[0033] The reference frames corresponding to the video frames in the original video file are determined, and the lip-sync images corresponding to the video frames are generated based on the reference frame information of the reference frames, the target feature vectors and the video frames.

[0034] In a second aspect, the present application discloses an audio and video lip translation device, comprising:

[0035] The module for determining the audio to be translated is used to determine the motion trajectory corresponding to each character ID based on the original video file, and to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model;

[0036] An audio translation module is used to perform speech recognition on the audio to be translated to obtain a text to be translated, obtain a current translated text corresponding to the target language based on the text to be translated, generate a corresponding current translated audio based on the current translated text, and determine whether the audio duration of the current translated audio meets a preset re-translation condition;

[0037] An audio retranslation module, for retranslating the text to be translated to obtain a new current translated text if the audio duration of the current translated audio meets a preset retranslation condition, and jumping to the step of generating the corresponding current translated audio based on the current translated text;

[0038] An audio alignment module, configured to perform audio alignment on the current translated audio based on the audio to be translated, if the audio duration of the current translated audio does not meet a preset re-translation condition, so as to obtain an aligned target translated audio;

[0039] The lip translation module is used to synthesize the target translated audios corresponding to the audios to be translated to obtain target audios, generate lip pictures corresponding to the video frames in the original video file based on the target audios and the reference frames in the original video file, generate a target video based on the lip pictures, and merge the target video with the target audio to obtain the translated audio and video.

[0040] In a third aspect, the present application discloses an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] The processor is used to execute the computer program to implement the aforementioned audio and video lip translation method.

[0043] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned audio and video lip translation method when executed by a processor.

[0044] In the present application, when performing audio and video lip translation, the motion trajectory corresponding to each character ID is determined based on the original video file, and the speaker recognition model is used to determine the audio to be translated corresponding to each character ID in the original audio file based on each of the motion trajectories; speech recognition is performed on the audio to be translated to obtain the text to be translated, and the current translated text corresponding to the target language is obtained based on the text to be translated, and the corresponding current translated audio is generated based on the current translated text, and it is determined whether the audio duration of the current translated audio meets the preset re-translation condition; if the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated audio. The method comprises the following steps: obtaining a previously translated text, and jumping to the step of generating a corresponding current translated audio based on the current translated text; if the audio duration of the current translated audio does not meet the preset re-translation condition, performing audio alignment on the current translated audio based on the audio to be translated to obtain an aligned target translated audio; synthesizing each target translated audio corresponding to each audio to be translated to obtain a target audio, and generating each lip shape picture corresponding to each video frame in the original video file based on the target audio and the reference frame in the original video file, generating a target video based on the lip shape picture, and merging the target video with the target audio to obtain the translated audio and video. It can be seen that the present application adjusts the audio duration of the translated audio by re-translating the audio to be translated when the audio duration of the translated audio meets the preset re-translation conditions, thereby avoiding the difference in the speaking speed of the translated audio caused by the translated audio duration being too long or too short; by tracking the movement trajectory of each character ID in the original video file, the audio features of the character ID and the audio to be translated and the target audio can be aligned, thereby improving the accuracy of selecting the driving target when generating the lip shape picture corresponding to the current video frame based on the target audio and the reference frame; at the same time, since some video frames in the original video file are introduced as reference frames in combination with the posture and lighting information of the video frame being processed when generating the lip shape picture, the lip generation quality in the generated lip shape picture is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flow chart of an audio and video lip translation method disclosed in this application;

[0047] Figure 2 A schematic diagram of a specific audio and video preprocessing process disclosed in this application;

[0048] Figure 3 A schematic diagram of an audio time track disclosed in this application;

[0049] Figure 4 A schematic diagram of a specific audio translation process disclosed in this application;

[0050] Figure 5 A schematic diagram of a specific video lip translation process disclosed in this application;

[0051] Figure 6 This is a schematic diagram of the structure of an audio and video lip translation device disclosed in this application;

[0052] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] At present, when voice translation is performed on video content, in order to align the content of the original video, the general practice is to perform linear scaling, which leads to differences in speech speed, greatly reducing the user experience. At the same time, after the voice translation is completed, the speaker's lip shape in the video needs to be aligned with the new voice. In the traditional digital human solution, the lip synchronization algorithm for a single person in a fixed posture is relatively mature. However, in cross-scene video images for multi-person interaction, the quality of multi-angle lower lip generation is low, and the selection accuracy of driving targets in multi-person scenes is also low. In order to solve the above technical problems, the present application discloses an audio and video lip translation method, which can reduce the speed difference of the translated audio, improve the quality of lip generation and the selection accuracy of driving targets in cross-scene video images for multi-person interaction, and improve the user experience.

[0055] See also Figure 1 As shown, the embodiment of the present invention discloses an audio and video lip translation method, comprising:

[0056] Step S11: determining the motion trajectory corresponding to each character ID based on the original video file, and determining the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model.

[0057] like Figure 2 As shown, in this embodiment, in order to determine the corresponding relationship between each character ID (Identity) and the audio to be translated, it is necessary to process the audio signal and the video signal in the original audio and video file separately when processing the original audio and video file. To this end, before determining the motion trajectory corresponding to each character ID, it is necessary to first obtain the original audio and video file, and separate the original audio and video file to obtain the original video file and the original audio file. Specifically, the FFmpeg (a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams) software package can be used to separate the video signal and the audio signal in the original video file to obtain the original video file and the original audio file.

[0058] After obtaining the original video file and the original audio file, the specific process of determining each character ID based on the original video file may include: using a feature extraction network to compare the differences between each frame in the original video file, and determining the scene switching timestamp of the original video file based on the differences, so as to divide the original video file into independent storyboard videos based on the scene switching timestamp; using a DeepFace detection network to determine each character ID based on each independent storyboard video, and tracking each character ID to determine the motion trajectory corresponding to each character ID.

[0059] In this embodiment, the specific process of performing scene detection and storyboarding on the original video file may include: using a feature extraction network to compare the differences in the features of the picture information between the video frames in the original video file, thereby determining the scene switching timestamp in the original video file and outputting [t_sc1, t_sc2, ..., t_scK], and dividing the original video file into independent storyboard videos based on the scene switching timestamp. Wherein, t_sc1 represents the timestamp corresponding to the first scene switching in the original video file, t_sc2 represents the timestamp corresponding to the second scene switching in the original video file, and t_scK represents the timestamp corresponding to the second scene switching in the original video file. The feature extraction network may be a feature extraction network pre-trained using ImageNet (a large visualization database for visual object recognition software research).

[0060] In this embodiment, after obtaining each independent storyboard video, it is necessary to detect and track the motion trajectory of the face corresponding to different character IDs. Specifically, the DeepFace detection network can be used to detect the face corresponding to the character ID, and then the DeepID (Deep hidden IDentity features, a deep learning model for face recognition) and IoU (Intersection over Union, intersection ratio) joint feature matching DeepSort (Deep Simple Online and Realtime Tracking, a multi-target tracking algorithm) is used to track the motion trajectory of the face, and the output format is [frame, id, x, y, w, h, conf] data, where frame is the video frame timestamp, ID is the character ID, x, y, w, h are the rectangular frame coordinates of the face corresponding to the character ID in the video frame, and conf is the target detection confidence.

[0061] In this embodiment, the speaker recognition model is used to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory, which may specifically include: using a deep neural network to obtain the pure human voice audio in the original audio file, and extracting the Mel spectrum features of the pure human voice audio to determine the audio features corresponding to each frame of the pure human voice audio; using the DBSCAN clustering algorithm to determine the utterance time period corresponding to each audio ID based on each audio feature, and segmenting the pure human voice audio based on the utterance time period to determine the audio to be translated corresponding to each audio ID; determining the face image set corresponding to each character ID based on each motion trajectory and the original video file, and inputting the pure human voice audio and the face image set into the speaker recognition model to determine the correspondence between each audio ID and each character ID; determining the utterance time period and the audio to be translated corresponding to each character ID based on the audio to be translated corresponding to each audio ID and the correspondence. That is, when the speaker model is used to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory, it includes processing the original audio file, and using the information obtained in the process of processing the original audio file and the original video file to determine the relationship between the character ID, the utterance time period and the audio to be translated. Figure 3 The figure shows a sample distribution of the speaking audio of two people. For audio ID1, the speaking time periods are [t1, t2], [t5, t6], [t9, t10], and for audio ID2, the speaking time periods are [t3, t4], [t7, t8]. Therefore, the corresponding audio to be translated can be determined based on the speaking time period corresponding to each audio ID, thereby determining the relationship between the character ID, the speaking time period and the audio to be translated.

[0062] In this embodiment, the specific process of processing the original audio file may include: first, inputting the original audio file into a deep neural network, removing the background sound in the original audio file to obtain pure human voice audio, and retaining the audio signal of multi-person conversation in the pure human voice audio; then, extracting the Mel spectrum features of the pure human voice audio to obtain the audio features of each frame of pure human voice audio. Specifically, if the frame rate of the original video file is 25 frames per second, each frame of pure human voice audio is a 40 millisecond audio signal; finally, applying the DBSCAN clustering algorithm to process all frame audio features, so as to obtain the sound time period corresponding to each audio ID, so as to segment the pure human voice audio based on the sound time period to obtain the audio to be translated corresponding to each audio ID, and there is no need to define the number of character ID clusters in advance.

[0063] After the processing of the original audio file and the original video file is completed, it is necessary to establish the corresponding relationship between the character ID and the audio ID, so as to determine the relationship between the character ID, the utterance time period and the audio to be translated. Specifically, the pure human voice audio is input into the speaker matching model, and the face image set of the motion trajectory corresponding to each character ID is obtained according to the face detection and tracking sequence information on each video frame (generally a group of 5 video frames, corresponding to the audio features of 200 milliseconds on the audio time axis); then the face image set is also input into the speaker matching model, and the speaker matching model outputs the confidence probability value of whether the face image set corresponding to the current character ID corresponds to the current audio feature. If the output is 1, it means that the current face image set does not match the current audio feature, that is, the character ID corresponding to the current face image set corresponds to the audio ID corresponding to the current audio feature. If the output is 0, it means that the current face image set does not match the current audio feature, that is, the character ID corresponding to the current face image set does not correspond to the audio ID corresponding to the current audio feature.

[0064] It is understandable that there may be multiple sets of face images corresponding to character IDs on the same group of video frames. If the face image sets corresponding to all character IDs on the current group of video frames do not correspond to the current audio features, it means that the character ID corresponding to the current speaker is not in the video frames of the current group, but in the video frames of other groups, or is a narration. Through the above process, the correspondence between the character ID and the audio ID can be established, so as to determine the utterance time period and the audio to be translated corresponding to each character ID based on the audio to be translated corresponding to each audio ID and the correspondence between the character ID and the audio ID, and determine which character screen in the video is driven by the corresponding audio (including the case where there are people in the screen, but the audio is a narration), so as to improve the selection accuracy of the driving target in the multi-scene video screen of multi-person interaction.

[0065] Step S12: performing speech recognition on the audio to be translated to obtain the text to be translated, obtaining the current translated text corresponding to the target language based on the text to be translated, generating the corresponding current translated audio based on the current translated text, and determining whether the audio duration of the current translated audio meets the preset re-translation condition.

[0066] like Figure 4As shown, when translating pure human voice audio, this embodiment will process each audio to be processed corresponding to each audio ID in the pure human voice audio separately. The specific processing process may include: performing speech recognition on the audio to be translated to obtain the text to be translated; using a translation tool based on a large language model and based on the first translation prompt word to translate the text to be translated to obtain the current translated text corresponding to the target language; generating the corresponding current translated audio based on the current translated text, and judging whether the audio duration of the current translated audio meets the preset re-translation condition. Among them, the preset re-translation condition may be that the ratio of the target difference to the audio duration of the audio to be translated is greater than a preset percentage threshold; the target difference is the difference between the audio duration of the audio to be translated and the audio duration of the current translated audio. Generating the corresponding current translated audio based on the current translated text may include: generating the current translated audio corresponding to the audio to be translated based on the current translated text using a target text-to-speech model; wherein the target text-to-speech model is a text-to-speech model generated based on the audio to be translated using the SoVITS algorithm. That is to say, if Figure 3 The audio segment [t1, t2] to be translated is translated. Taking Chinese to English as an example, this segment of audio to be translated is input into the ASR (Automatic Speech Recognition) network and output as the corresponding Chinese text (i.e., the text to be translated), such as "Hello, my name is Wang Xiaoer". Then, the input of the large language model is defined through the prompt word engineering, and the identity of the large language model as a professional language translator is clarified. After the Chinese text is input into the large language model, the corresponding English text (i.e., the current translated text) is obtained, such as "Hello, my name is Xiaoer Wang".

[0067] Step S13: If the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated text, and the process jumps to the step of generating the corresponding current translated audio based on the current translated text.

[0068] In this embodiment, if the audio duration of the current translated audio meets the preset re-translation condition, that is, the ratio of the target difference to the audio duration of the audio to be translated is greater than a preset percentage threshold (such as 10%), the text to be translated will be re-translated to obtain a new current translated text. Specifically, the first translation prompt word can be optimized according to the size relationship between the audio duration of the audio to be translated and the audio duration of the current translated audio, so as to obtain an optimized second translation prompt word, so as to inform the large model to shorten / increase the number of text characters in the translation result compared to the text content of the first round of translation; then use the translation tool and re-translate the text to be translated based on the second translation prompt word to obtain a new current translated text. For example Figure 3The audio duration of the audio segment to be translated shown in [t1, t2] is 3 seconds. The English text (i.e., the current translated text) obtained by the first translation of the Chinese text "Hello, my name is Xiaoer Wang" is "Hello, my name is Xiaoer Wang". The audio duration of the current translated audio obtained by performing text-to-speech operation on this English text based on TTS at a normal speaking speed is 3.6 seconds, but this 3.6 seconds differs from the audio duration of the audio to be translated by 0.6 seconds, which is about 20% of the audio duration of the audio to be translated, which is greater than the preset percentage threshold. In order to avoid the poor audio-visual experience caused by the fast speaking speed of the translated audio, the first translation prompt word needs to be optimized and adjusted to obtain the second translation prompt word, so as to inform the large model to shorten the number of text characters of the translation result compared with the text content of the first round of translation, that is, if the audio duration of the audio to be translated is 3 seconds, the text translation result of the current translated audio with an audio duration of 3.6 seconds needs to be optimized by 20%. After tuning, the output English text is "Hi, I'm Xiaoer Wang". Under the premise of expressing the same semantic meaning, the number of syllables of the English text characters outputted for the second time is reduced, and the audio duration of the current translated audio synthesized at a normal speaking speed is 2.9s, which is closer to the audio duration of the original audio to be translated.

[0069] Step S14: If the audio duration of the current translated audio does not meet the preset re-translation condition, audio alignment is performed on the current translated audio based on the audio to be translated to obtain an aligned target translated audio.

[0070] In this embodiment, if the audio duration of the current translated audio does not meet the preset re-translation condition, that is, the ratio of the target difference to the audio duration of the audio to be translated is less than or equal to the preset percentage threshold, the current translated text can be used as the final translated text, and the audio duration of the current translated audio can be linearly scaled to be equal to the audio duration of the audio to be translated, completing the audio alignment, thereby obtaining the aligned target translated audio.

[0071] Step S15, synthesizing the target translated audios corresponding to the audios to be translated to obtain target audios, and generating lip-sync images corresponding to the video frames in the original video file based on the target audios and the reference frames in the original video file, generating a target video based on the lip-sync images, and merging the target video with the target audio to obtain the translated audio and video.

[0072] In this embodiment, each audio to be translated needs to be translated to obtain the corresponding target translated audio, that is, the audio to be translated from different people and different time periods in the pure human voice audio needs to be processed by the above steps, so that each current translated audio is aligned with the original audio to be translated in terms of speech speed and time, thereby obtaining all the target translated audios, and synthesizing all the target translated audios to obtain the translated audio signal corresponding to the pure human voice audio, that is, the target audio.

[0073] like Figure 5 As shown, after obtaining the target audio, the lip-sync images corresponding to each video frame in the original video file can be generated based on the reference frame. The input signal is the target audio and the original video signal that are time-aligned after translation, so as to use the target audio to drive the change of the speaker's mouth shape in the original video through the deep learning lip-sync generation network. This process will utilize the audio and video speaker matching information and scene storyboard information in the preprocessing stage. Since the audio and video speaker matching information describes the relationship between the original audio and the corresponding speaker in the picture, it is determined which character ID the current audio corresponds to and is saying the corresponding words. Considering that the corresponding relationship between the audio ID in the original audio file and the character ID in the original audio file has been determined in the aforementioned process, and the time alignment of the translated audio has been completed when generating the target translated audio, the target audio also corresponds one-to-one to the character ID that needs to be driven, that is, the speaker ID driven area matching is realized.

[0074] In this embodiment, the specific process of generating each lip-shaped picture corresponding to each video frame in the original video file based on the target audio and the reference frame in the original video file may include: extracting the speech posterior probability feature in the target audio to obtain the corresponding target feature vector; determining each reference frame corresponding to each video frame in the original video file, and generating each lip-shaped picture corresponding to each video frame based on the reference frame information, the target feature vector and each video frame of each reference frame. When selecting the reference frame, since the facial posture, lighting, expression, etc. of the same character ID in different scene storyboards are very inconsistent, this embodiment uses the scene switching timestamp (i.e., scene detection storyboard information) obtained in the above steps to frame the continuity track of the task ID to maintain stability, and at the same time selects several similar reference frames in the storyboard where the video frame is located as the input of the lip-shaped driving network model in combination with the posture, lighting and other information of the video frame.

[0075] When training a lip-sync driven network model, it is necessary to train a general audio-visual synchronizer, such as SyncNet (a neural network model used to determine whether audio and video are synchronized). When training SyncNet, positive and negative sample pairs are formed based on audio signals and multiple frames of continuous face images, and then self-supervised learning is performed using a contrastive learning loss function. In particular, in the selection of positive and negative samples, positive samples can be defined as a matching pair of the audio features of the current frame and the two frames before and after the current frame (including the current frame, a total of 5 frames). Negative samples can be defined in multiple forms, one is 5 frames of images of the non-current frame adjacent to the speaker of the audio, and the other is 5 frames of images of the non-current speaker (can be samples of the current video file or samples of other video files), thereby increasing the generalization and coverage of model training.

[0076] In addition to training SyncNet, it is also necessary to train a lip generator, such as Unet (U-shapedNetwork, a deep learning network model for image segmentation). When training the lip generator, SyncNet needs to be frozen and used as a lip discriminator to provide corresponding loss value calculations. In a specific embodiment, when training the lip generator, it is also necessary to extract the speech posterior probability features in the target audio to obtain the corresponding target feature vector, and determine the reference frames corresponding to each video frame in the original video file, so as to generate multiple groups of speech feature signals and video reference frame signals based on the reference frame information of each reference frame, the target feature vector and each video frame. This process is supervised by multiple loss functions, which can specifically include SyncNet loss, face perception loss, and image quality losses such as SSIM (Structural Similarity).

[0077] After obtaining the trained lip-driven network model, the model can be used to infer the translated lip-shaped images corresponding to each video frame. The target audio (i.e., the translated audio signal), each video frame, and the reference frame information of the reference frame corresponding to each video frame are input into the lip-driven network model, and the translated lip-shaped images corresponding to each video frame can be obtained. Specifically, the target feature vector of the target audio is extracted using a PPG (phonetic posteriorgrams) network. This PPG network is trained using a large-scale speech data set through self-supervised training, and a 384-dimensional feature vector is output for each frame in the target audio; then the 5 reference frames corresponding to the video frame to be processed are input into the UNet network structure trained in the above process to obtain the translated lip-shaped images corresponding to the video frame.

[0078] In a specific implementation, the original video file can be processed frame by frame based on the timestamp corresponding to each video frame. That is, starting from the first video frame in the original video file, the reference frame corresponding to the video frame (that is, the current video frame) is determined, and then the reference frame information of the video frame and the corresponding reference frame is input into the lip-driven network model obtained through the aforementioned training, and the timestamp corresponding to the current video frame is used to determine the audio frame corresponding to the current video frame in the target audio, and the target feature vector corresponding to the audio frame is extracted using the PPG network, and the target feature vector corresponding to the audio frame and the current video frame are input into the UNet network structure, so as to obtain the translated lip picture corresponding to the current video frame, and then the next video frame of the current video frame is used as the new current video frame, and jump to the step of determining the reference frame corresponding to the current video frame, until the translated lip picture corresponding to the last video frame in the original video file is obtained.

[0079] Through the above process, each lip shape picture corresponding to each video frame can be obtained, thereby generating each character lip shape video corresponding to the target audio, and generating a target video based on each character lip shape video, so as to merge the target video with the target audio to obtain the translated audio and video. Since the original video file and the original audio file in the above steps are obtained by separating the original audio and video files using the FFmpeg software package, correspondingly, the FFmpeg software package can be used to merge the target video with the target audio to obtain the translated audio and video, and finally complete the multi-person lip shape video translation task.

[0080] It can be seen that the present application adjusts the audio duration of the translated audio by re-translating the audio to be translated when the audio duration of the translated audio meets the preset re-translation conditions, thereby avoiding the difference in the speaking speed of the translated audio caused by the translated audio duration being too long or too short; by tracking the movement trajectory of each character ID in the original video file, the audio features of the character ID and the audio to be translated and the target audio can be aligned, thereby improving the accuracy of selecting the driving target when generating the lip shape picture corresponding to the current video frame based on the target audio and the reference frame; at the same time, since some video frames in the original video file are introduced as reference frames in combination with the posture and lighting information of the video frame being processed when generating the lip shape picture, the lip generation quality in the generated lip shape picture is relatively high.

[0081] See also Figure 6 As shown, the present application discloses an audio and video lip translation device, comprising:

[0082] The to-be-translated audio determination module 11 is used to determine the motion trajectory corresponding to each character ID based on the original video file, and determine the to-be-translated audio corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model;

[0083] The audio translation module 12 is used to perform speech recognition on the audio to be translated to obtain the text to be translated, obtain the current translated text corresponding to the target language based on the text to be translated, generate the corresponding current translated audio based on the current translated text, and determine whether the audio duration of the current translated audio meets the preset re-translation condition;

[0084] The audio re-translation module 13 is used to re-translate the text to be translated to obtain a new current translated text if the audio duration of the current translated audio meets the preset re-translation condition, and jump to the step of generating the corresponding current translated audio based on the current translated text;

[0085] The audio alignment module 14 is used to align the current translated audio based on the audio to be translated to obtain an aligned target translated audio if the audio duration of the current translated audio does not meet the preset re-translation condition;

[0086] The lip translation module 15 is used to synthesize the target translated audios corresponding to the audios to be translated to obtain target audios, generate lip pictures corresponding to the video frames in the original video file based on the target audios and the reference frames in the original video file, generate a target video based on the lip pictures, and merge the target video with the target audio to obtain the translated audio and video.

[0087] It can be seen that the present application adjusts the audio duration of the translated audio by re-translating the audio to be translated when the audio duration of the translated audio meets the preset re-translation conditions, thereby avoiding the difference in the speaking speed of the translated audio caused by the translated audio duration being too long or too short; by tracking the movement trajectory of each character ID in the original video file, the audio features of the character ID and the audio to be translated and the target audio can be aligned, thereby improving the accuracy of selecting the driving target when generating the lip shape picture corresponding to the current video frame based on the target audio and the reference frame; at the same time, since some video frames in the original video file are introduced as reference frames in combination with the posture and lighting information of the video frame being processed when generating the lip shape picture, the lip generation quality in the generated lip shape picture is relatively high.

[0088] In a specific embodiment, the device may further include:

[0089] An audio and video separation module is used to obtain the original audio and video files, and use the FFmpeg software package to separate the original audio and video files to obtain the original video file and the original audio file;

[0090] Accordingly, the device may further include:

[0091] The audio and video merging module is used to merge the target video and the target audio using the FFmpeg software package to obtain the translated audio and video.

[0092] In a specific implementation, the to-be-translated audio determination module 11 may specifically include:

[0093] A storyboard division unit, configured to compare differences between frames in the original video file using a feature extraction network, and determine a scene switching timestamp of the original video file based on the differences, so as to divide the original video file into independent storyboard videos based on the scene switching timestamp;

[0094] The trajectory determination unit is used to determine each character ID based on each independent storyboard video using a DeepFace detection network, and track each character ID to determine the motion trajectory corresponding to each character ID.

[0095] In a specific implementation, the to-be-translated audio determination module 11 may specifically include:

[0096] An audio feature extraction unit, configured to obtain pure human voice audio in the original audio file by using a deep neural network, and extract Mel spectrum features of the pure human voice audio to determine audio features corresponding to each frame of the pure human voice audio;

[0097] An audio segmentation unit, configured to determine the utterance time period corresponding to each audio ID based on each of the audio features by using a DBSCAN clustering algorithm, and segment the pure human voice audio based on the utterance time period to determine the audio to be translated corresponding to each of the audio IDs;

[0098] A first correspondence determination unit is used to determine a face image set corresponding to each character ID based on each motion trajectory and the original video file, and input the clean human voice audio and the face image set into the speaker recognition model to determine a correspondence between each audio ID and each character ID;

[0099] The second corresponding relationship determining unit is configured to determine the utterance time period and the audio to be translated corresponding to each character ID based on the audio to be translated corresponding to each audio ID and the corresponding relationship.

[0100] In a specific implementation, the audio translation module 12 may specifically include:

[0101] The translated audio generation unit is used to generate a current translated audio corresponding to the audio to be translated based on the current translated text using a target text-to-speech model; wherein the target text-to-speech model is a text-to-speech model generated based on the audio to be translated using the SoVITS algorithm.

[0102] In a specific implementation, the audio translation module 12 may specifically include:

[0103] A text translation unit, configured to translate the text to be translated using a translation tool based on a large language model and based on a first translation prompt word, so as to obtain a current translated text corresponding to a target language;

[0104] Accordingly, the audio re-translation module 13 may specifically include:

[0105] A translation prompt word optimization unit, configured to optimize the first translation prompt word accordingly to obtain an optimized second translation prompt word if the audio duration of the currently translated audio meets a preset re-translation condition;

[0106] The text re-translation unit is used to re-translate the text to be translated by using the translation tool based on the second translation prompt word to obtain a new current translated text.

[0107] In a specific implementation, the lip translation module 15 may specifically include:

[0108] A feature vector acquisition unit, used to extract the speech posterior probability features in the target audio to obtain a corresponding target feature vector;

[0109] The lip-sync picture generation unit is used to determine each of the reference frames corresponding to each of the video frames in the original video file, and generate each of the lip-sync pictures corresponding to each of the video frames based on the reference frame information of each of the reference frames, the target feature vector and each of the video frames.

[0110] Furthermore, the present application also discloses an electronic device. Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0111] Figure 7The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the audio / video lip translation method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may be a computer.

[0112] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0113] In addition, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the audio and video lip translation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0115] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed audio and video lip translation method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0116] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0120] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for lip translation of audio and video, characterized in that: include: Determine the motion trajectory corresponding to each character ID based on the original video file, and determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model; Performing speech recognition on the audio to be translated to obtain a text to be translated, obtaining a current translated text corresponding to the target language based on the text to be translated, generating a corresponding current translated audio based on the current translated text, and determining whether the audio duration of the current translated audio meets a preset re-translation condition; If the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated text, and the process jumps to the step of generating the corresponding current translated audio based on the current translated text; If the audio duration of the current translated audio does not meet the preset re-translation condition, audio alignment is performed on the current translated audio based on the audio to be translated to obtain an aligned target translated audio; The target translated audio files corresponding to the audio files to be translated are synthesized to obtain target audio files, and lip-sync images corresponding to the video frames in the original video file are generated based on the target audio files and the reference frames in the original video file, a target video is generated based on the lip-sync images, and the target video is merged with the target audio files to obtain the translated audio and video files.

2. The method for lip translation of audio and video according to claim 1, characterized in that: Before determining the motion trajectory corresponding to each character ID based on the original video file, the method further includes: Acquire an original audio and video file, and use the FFmpeg software package to separate the original audio and video file to obtain the original video file and the original audio file; Accordingly, merging the target video with the target audio to obtain translated audio and video includes: The target video and the target audio are merged with each other using the FFmpeg software package to obtain the translated audio and video.

3. The method for lip translation of audio and video according to claim 1, characterized in that: The step of determining the motion trajectory corresponding to each character ID based on the original video file includes: Using a feature extraction network to compare differences between frames in the original video file, and determining a scene switching timestamp of the original video file based on the differences, so as to divide the original video file into independent storyboard videos based on the scene switching timestamps; The DeepFace detection network is used to determine each character ID based on each independent storyboard video, and each character ID is tracked to determine the motion trajectory corresponding to each character ID.

4. The method for lip translation of audio and video according to claim 1, characterized in that: The method of using a speaker recognition model to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory includes: Using a deep neural network to obtain pure human voice audio in the original audio file, and extracting Mel frequency spectrum features of the pure human voice audio to determine audio features corresponding to each frame of the pure human voice audio; Determine the utterance time period corresponding to each audio ID based on each of the audio features using the DBSCAN clustering algorithm, and segment the pure human voice audio based on the utterance time period to determine the audio to be translated corresponding to each of the audio IDs; Determine a face image set corresponding to each character ID based on each of the motion trajectories and the original video file, and input the clean human voice audio and the face image set into the speaker recognition model to determine the correspondence between each of the audio IDs and each of the character IDs; The utterance time period and the audio to be translated corresponding to each character ID are determined based on the audio to be translated corresponding to each audio ID and the corresponding relationship.

5. The method for lip translation of audio and video according to claim 1, characterized in that: The generating corresponding current translated audio based on the current translated text includes: The target text-to-speech model is used to generate a current translated audio corresponding to the audio to be translated based on the current translated text; wherein the target text-to-speech model is a text-to-speech model generated based on the audio to be translated using the SoVITS algorithm.

6. The method for lip translation of audio and video according to claim 1, characterized in that: The preset re-translation condition is that the ratio of the target difference value to the audio duration of the audio to be translated is greater than a preset percentage threshold; the target difference value is the difference between the audio duration of the audio to be translated and the audio duration of the currently translated audio.

7. The method for lip translation of audio and video according to claim 1, characterized in that: The step of obtaining a current translated text corresponding to a target language based on the text to be translated includes: Translating the text to be translated using a translation tool based on a large language model and based on a first translation prompt word to obtain a current translated text corresponding to a target language; Correspondingly, if the audio duration of the current translated audio meets the preset re-translation condition, the text to be translated is re-translated to obtain a new current translated text, including: If the audio duration of the currently translated audio meets the preset re-translation condition, the first translation prompt word is optimized accordingly to obtain an optimized second translation prompt word; The text to be translated is retranslated using the translation tool based on the second translation prompt word to obtain a new current translated text.

8. The method for audio and video lip translation according to any one of claims 1 to 7, characterized in that: The step of generating each lip-sync picture corresponding to each video frame in the original video file based on the target audio and the reference frame in the original video file comprises: Extracting speech posterior probability features in the target audio to obtain a corresponding target feature vector; The reference frames corresponding to the video frames in the original video file are determined, and the lip-sync images corresponding to the video frames are generated based on the reference frame information of the reference frames, the target feature vectors and the video frames.

9. An audio and video lip translation device, characterized in that: include: The module for determining the audio to be translated is used to determine the motion trajectory corresponding to each character ID based on the original video file, and to determine the audio to be translated corresponding to each character ID in the original audio file based on each motion trajectory using a speaker recognition model; An audio translation module is used to perform speech recognition on the audio to be translated to obtain a text to be translated, obtain a current translated text corresponding to the target language based on the text to be translated, generate a corresponding current translated audio based on the current translated text, and determine whether the audio duration of the current translated audio meets a preset re-translation condition; An audio retranslation module, for retranslating the text to be translated to obtain a new current translated text if the audio duration of the current translated audio meets a preset retranslation condition, and jumping to the step of generating the corresponding current translated audio based on the current translated text; An audio alignment module, configured to perform audio alignment on the current translated audio based on the audio to be translated, if the audio duration of the current translated audio does not meet a preset re-translation condition, so as to obtain an aligned target translated audio; The lip translation module is used to synthesize the target translated audios corresponding to the audios to be translated to obtain target audios, generate lip pictures corresponding to the video frames in the original video file based on the target audios and the reference frames in the original video file, generate a target video based on the lip pictures, and merge the target video with the target audio to obtain the translated audio and video.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the audio and video lip translation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the audio and video lip translation method according to any one of claims 1 to 8 is implemented.

Images