Video translation method and system based on artificial intelligence

Abstract

The invention discloses a video translation method and system based on artificial intelligence. The method relates to the technical field of video translation and comprises the following steps of original sound track extraction, target AI speaker adaptation, AI dubbing generation and mouth shape synchronization and video synthesis. According to the method, independent audio and video streams are obtained by adopting an audio and video separation technology, and multiple original sound tracks are extracted through a voice separation model; matching or generating an adaptive target AI speaker module in a preset tone library; converting the original language voice into a text, translating the text into a target language text, and synthesizing an AI dubbing audio track in combination with a target AI speaker module; and finally, the independent video stream and the multi-AI dubbing audio track are input into the mouth shape synchronization model to output a translated video, so that the timbre fitting degree, the voice quality and the voice consistency of the same speaker of AI dubbing are improved, and meanwhile, the resource utilization rate of video translation and the processing efficiency under batch tasks are improved. The problem that in the prior art, video translation is low in quality and efficiency is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video translation, and in particular to a video translation method and system based on artificial intelligence. BACKGROUND

[0002] First, enter the core translation stage. If the video has no source subtitles, use Whisper speech recognition tools to complete speech-to-text, then optimize the source language text after transcription, and then use CAT tools such as Trados to translate the target language with the help of artificial intelligence, such as "subtitle embedded" to "hardcoded subtitle", localization adaptation and length adaptation, while ensuring the uniformity of terms in the same video through the term library.

[0003] In the audio and video processing stage, the subtitle production needs to use Arctime tools to synchronize the timeline, design the style, and export SRT / ASS / VTT formats as required. If dubbing is required, select a dubbing actor with a suitable style, optimize the script to a colloquial version, record it in a quiet environment with a condenser microphone, then use Audacity to reduce noise, standardize the volume, and adjust the volume ratio with the background music, and finally use Premiere Pro to achieve sound and picture synchronization. Multi-language versions need to integrate subtitle tracks or produce independent dubbing videos.

[0004] Enter the quality audit stage. First, the target language native speaker audits the translation accuracy, term consistency, and localization adaptation, then tests frame by frame (such as multi-device verification), checks the sound and picture synchronization of the subtitles, style compliance, and dubbing quality, and finally adjusts according to the feedback from the demand side. When the final delivery is required, package the final video, independent subtitles, dubbing audio, and translation documents as required, and archive source files, transcription texts, and other resources for easy future modification and accumulation of term and memory libraries. Different scenarios have slightly different processes (such as short videos that can simplify dubbing and only do subtitles, and movies that need to strictly match the lip movements), but overall, both language accuracy and user viewing experience need to be considered.

[0005] For example, the Chinese invention patent with publication number CN102265271B discloses a system and method for translating timed text in a web video, which includes receiving a request from a client to identify a web video. In response to the request, retrieve a timed text track of the web video from a timed text database. Each timed text track in the timed text database specifies a specific time in the video at which text is displayed. Automatically translate the text from the timed text track into a target language. Finally, send the translated text to the client to be displayed with the web video.

[0006] For example, the Chinese invention patent with publication number CN110489763A discloses a video translation method and device, which includes: extracting audio from a vehicle maintenance video and performing global translation to obtain a global translation text, recognizing target words such as automobile parts and maintenance device surroundings in the global translation text, constructing and outputting a fixed Chinese format, replacing sentences in the global translation text with key sentences, and generating a translation text corresponding to a preset language.

[0007] The above technology at least has the following technical problems: First, the tone matching degree and quality of AI dubbing are poor. Due to the difference in the friendliness of audio and video files to algorithm models, the accuracy of extracting original speech feature information is difficult to guarantee, and the time length of corpus data available for analyzing and extracting feature information in a single video is limited. These two factors together make it impossible for the prior art to accurately match the speech features of different speakers, ultimately resulting in AI dubbing that is stiff and cannot match the characteristics of the original speaker, and even mixing in noise generated by the model in addition to the voice.

[0008] The existing method often directly uses speech conversion technology to extract original speech feature information in the video sentence by sentence, and then adjusts the synthesized AI dubbing to match the original speech feature as much as possible. This technical solution not only consumes a lot of computing resources, but also occupies a large database space. Especially when different videos contain the same person, unnecessary resource waste will be caused by repeated execution of speech conversion. In addition, algorithm errors in the process of sentence-by-sentence conversion may also cause the similarity and consistency of the overall speech features of the same person to decrease.

[0009] Due to the lack of automated video translation process, the traditional video translation process mainly relies on manual work, which is time-consuming and labor-intensive. The existing translation system formed by simply splicing speech recognition, speech synthesis, speech conversion, and lip synchronization technologies, although it achieves semi-automation to some extent, lacks a smooth intelligent connection mechanism between the technical links, still needs to invest a lot of manpower, and the audio and video results directly output by the model need to be adjusted again through manual editing, parameter modification, etc. Therefore, it is impossible to realize truly efficient full-process automation, and there are problems of low quality and efficiency of video translation. SUMMARY

[0010] In order to solve the problem of low quality and efficiency of video translation in the prior art, the embodiments of the present application provide a video translation method and system based on artificial intelligence. The technical solution is as follows: In an aspect, an artificial intelligence-based video translation method is provided, comprising: processing a video to be translated using an audio-video separation technology to separate an independent audio stream and an independent video stream, inputting the independent audio stream into a speech separation algorithm model to obtain a plurality of original sound tracks; inputting the plurality of original sound tracks into a voiceprint matching algorithm model to detect whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in a timbre library, if yes, determining the AI speaker module as a target AI speaker module adapted to the corresponding original sound track, if no, generating a target AI speaker module adapted to the original sound track; converting a voice signal into a text form to obtain an original language text; inputting the original language text into a machine translation algorithm model to translate the original language into a target language to obtain a target language text with the same semantics but different language; inputting the target language text and the target AI speaker module into a speech synthesis algorithm model to obtain AI dubbing sound tracks with high similarity in voice features but different languages to each original sound track; inputting the independent video stream and the plurality of AI dubbing sound tracks into a lip synchronization algorithm model to track and analyze lip dynamic information of each speaker in a video frame, re-drive and replace the lip animation of the speaker using the voice of the AI dubbing sound track to obtain a target video stream with synchronized lip action and AI dubbing, compressing the target video stream and the plurality of AI dubbing sound tracks to complete audio-video mixing, encoding and packaging to obtain a translated video.

[0011] In another aspect, an artificial intelligence-based video translation system is provided, comprising: an original sound track extraction module, a target AI speaker module adaptation module, an AI dubbing generation module, and a lip synchronization and video synthesis module; the original sound track extraction module is configured to process a video to be translated using an audio-video separation technology to separate an independent audio stream and an independent video stream, input the independent audio stream into a speech separation algorithm model to obtain a plurality of original sound tracks; the target AI speaker module adaptation module is configured to input the plurality of original sound tracks into a voiceprint matching algorithm model to detect whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in a timbre library, if yes, determine the AI speaker module as a target AI speaker module adapted to the corresponding original sound track, and if no, generate a target AI speaker module adapted to the original sound track; the AI dubbing generation module is configured to convert a speech signal into a text form to obtain an original language text, input the original language text into a machine translation algorithm model to translate the original language into a target language to obtain a target language text with the same semantics but different language, and input the target language text and the target AI speaker module into a speech synthesis algorithm model to obtain AI dubbing audio tracks with high similarity in voice characteristics but different language to each original sound track; the lip synchronization and video synthesis module is configured to input the independent video stream and the plurality of AI dubbing audio tracks into a lip synchronization algorithm model to track and analyze lip dynamic information of each speaker in a video frame, re-drive and replace the lip animation of the speaker using the AI dubbing audio track, obtain a target video stream with synchronized lip action and AI dubbing, and compress the target video stream and the plurality of AI dubbing audio tracks to complete audio-video mixing, encoding and packaging to obtain a translated video.

[0012] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The AI dubbing obtained by the method and system has higher timbre fitting degree with the original voice, and the method of ordered combination of the voiceprint matching algorithm model and the speech cloning algorithm model can match the AI speaker module in the timbre library, since the AI speaker module in the timbre library is generated based on a large amount of high-quality corpus data, or is a high-quality module precisely modulated in voice feature parameters, so that the timbre fitting degree of the obtained AI dubbing is significantly higher than the AI dubbing generated by sentence-by-sentence speech conversion based on limited original corpus data, or the dubbing performed by a dubbing actor or other traditional methods; and when the AI speaker module is generated by speech cloning without being matched in the timbre library, since each speaker's original sound track contains all the original speech data set of the speaker, the data length of the original corpus is also significantly longer than that of the traditional technology of sentence-by-sentence speech conversion, so that AI dubbing with higher timbre fitting degree can also be obtained.

[0013] 2、The AI dubbing obtained by the method and system has higher consistency of the same speaker's voice. The method for obtaining AI dubbing is to first extract, classify and organize the voice signals in the video according to different speakers, match or clone a corresponding AI speaker module for each speaker, and then perform voice synthesis based on the AI speaker module. Since the internal voice feature parameters of the AI speaker module are relatively stable, when generating AI dubbing for different voice segments of each speaker in the video, the voice segments of the same speaker have high consistency in key voice features. In comparison, the AI dubbing generated by voice conversion based on original corpus data sentence by sentence is difficult to maintain stability in voice features due to the limitations of short sentence corpus and other factors such as the context of different short sentences, and the voice features of the AI dubbing generated each time fluctuate greatly.

[0014] 3、The AI dubbing obtained by the method and system has higher voice quality. By separating the pure voice audio and international audio contained in the video and preferentially matching the AI speaker module in the timbre library, the method can avoid the situation where the original corpus is mixed with additional noise and other algorithm model-unfriendly situations due to the interference of complex and variable factors such as recording environment and equipment in the video, resulting in problems such as noise mixing or voice distortion in the generated AI dubbing, affecting the overall quality of the dubbing.

[0015] 4、The video translation performed by the method and system has higher overall resource utilization. By using the AI speaker module to generate dubbing, the method preferentially matches the existing AI speaker module in the timbre library, and the cloned new AI speaker module can also be saved in the timbre library and directly reused by subsequent video translation tasks containing the same or similar speaker, thus avoiding the repeated process of voice conversion for short sentence corpus in each video, saving a lot of time and computing resources, and making the overall video translation process more efficient and economical.

[0016] 5、The video translation performed by the method and system has higher dubbing efficiency under batch tasks. The video-to-video translation task can be completely completed based on artificial intelligence, a workflow from the video to be translated to the translated video is established, and the AI speaker resource can be modularly reused to compress the dubbing time as much as possible while ensuring quality. Compared with other traditional methods such as dubbing actors, the efficiency is significantly improved in the context of large-scale video translation. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a video translation method based on artificial intelligence provided by an embodiment of the present application; Figure 2A step one flowchart of a video translation method based on artificial intelligence provided by an embodiment of the application; Figure 3 A step two flowchart of a video translation method based on artificial intelligence provided by an embodiment of the application; Figure 4 A step three flowchart of a video translation method based on artificial intelligence provided by an embodiment of the application; Figure 5 A step four flowchart of a video translation method based on artificial intelligence provided by an embodiment of the application; Figure 6 A structural schematic diagram of a video translation system based on artificial intelligence provided by an embodiment of the application. DETAILED DESCRIPTION

[0018] The technical solutions in the application will be described below with reference to the drawings.

[0019] In the embodiments of the application, the words such as “example”, “for example” and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as “example” in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word “example” is intended to present a concept in a specific manner. In addition, in the embodiments of the application, the meaning expressed by “and / or” can be both, or can be either of the two.

[0020] The embodiments of the application provide a video translation method and system based on artificial intelligence, which solve the problem of low quality and efficiency of video translation in the prior art. Independent audio and video streams are obtained by using audio-video separation technology, and multiple original audio tracks are extracted by using a speech separation model. A target AI speaker module that is matched or generated in a preset timbre library is adapted. Original language speech is converted into text and translated into target language text, and then AI dubbing audio tracks are synthesized in combination with the target AI speaker module. Finally, independent video streams and multiple AI dubbing audio tracks are input into a lip synchronization model to regenerate target video streams with lip synchronization. After audio-video mixing, encoding and encapsulation, the translated video is output, improving the timbre fit of AI dubbing, the quality of speech and the consistency of speech of the same speaker, and improving the resource utilization rate of video translation and the processing efficiency under batch tasks.

[0021] The technical solutions in the embodiments of the application are as follows to solve the problem of low quality and efficiency of video translation: The audio-video separation technology is used to process the video to be translated to separate an independent audio stream and an independent video stream, multiple original audio tracks are input into a voiceprint matching algorithm model, whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in a timbre library is detected, if yes, the AI speaker module is determined as a target AI speaker module adapted to the corresponding original audio track, if no, a target AI speaker module adapted to the original audio track is generated, a voice signal is converted into a text form to obtain original language text, the original language text is input into a machine translation algorithm model to translate the original language into a target language to obtain target language text with the same semantics but different languages, the target language text and the target AI speaker module are input into a speech synthesis algorithm model to obtain AI dubbing audio tracks with high similarity in voice features but different languages to each original audio track, the independent video stream and the multiple AI dubbing audio tracks are input into a lip synchronization algorithm model to track and analyze lip dynamic information of each speaker in a video frame, the AI dubbing audio track is used to re-drive and replace the lip animation of the speaker to obtain a target video stream with synchronized lip action and AI dubbing, the target video stream and the multiple AI dubbing audio tracks are compressed to complete audio-video mixing, encoding and packaging to obtain a translated video, the timbre fitting degree, voice quality and voice consistency of the same speaker of AI dubbing are improved, and the resource utilization rate of video translation and processing efficiency under batch tasks are improved.

[0022] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0023] The embodiment of the application provides a video translation method based on artificial intelligence, as shown in a kind of video translation method flow chart based on artificial intelligence, the flow chart of the method can include: original audio track extraction, target AI speaker adaptation, AI dubbing generation and lip synchronization and video synthesis. Figure 1 As shown in a kind of video translation method flow chart based on artificial intelligence, the flow chart of the method can include: original audio track extraction, target AI speaker adaptation, AI dubbing generation and lip synchronization and video synthesis.

[0024] Step one, as shown in a kind of video translation method flow chart based on artificial intelligence, the flow chart of the method can include: original audio track extraction, target AI speaker adaptation, AI dubbing generation and lip synchronization and video synthesis. Figure 2 As shown in a kind of video translation method flow chart based on artificial intelligence, the flow chart of the method can include: original audio track extraction, target AI speaker adaptation, AI dubbing generation and lip synchronization and video synthesis.

[0025] It should be noted that first, the audio and video of the video to be translated are separated using audio and video separation technology to obtain independent audio stream and independent video stream, the audio part in the originally encapsulated audio and video file is extracted separately to obtain a pure and independent audio file, which can be in common MP3, WAV, etc. Format. Then the independent audio stream is input into the speech separation algorithm model to extract clean speech signals from the mixed audio signals to obtain pure speech audio and international audio. International sound refers to the collection of all sound elements in the independent audio stream except the voices of the characters such as dialogue and narration, which can include but is not limited to environmental sound effects, music sound and other non-verbal sounds. Finally, the pure speech audio is input into the voiceprint clustering algorithm model to distinguish and cluster the speech segments of different speakers, and then the speech segments of each speaker are placed on a separate audio track while retaining the original time stamp of each speech segment to obtain multiple original audio tracks. For example, all speech segments belonging to the "first speaker" are integrated into the "first original audio track".

[0026] Further, the independent audio stream is input into the speech separation algorithm model to obtain multiple original audio tracks, specifically including: the speech separation algorithm model extracts speech signals from the mixed audio signals of the independent audio stream to obtain pure speech audio; the pure speech audio is input into the voiceprint clustering algorithm model to distinguish and cluster the speech segments of different speakers, and the speech segments of each speaker are placed on a separate audio track while retaining the original time stamp of each speech segment, and finally multiple original audio tracks are obtained.

[0027] In this embodiment, after the independent audio stream is input into the speech separation algorithm model, the model first accurately extracts the speech signal from the mixed audio signal through deep signal processing technology, effectively suppresses the background noise, music and other non-speech interference, and outputs high-fidelity pure speech audio. This step significantly improves the purity and recognizability of the speech signal; then, the pure speech audio is input into the voiceprint clustering algorithm model, which performs unsupervised clustering of the speech segments based on the voiceprint features of the speakers (such as tone, fundamental frequency, harmonic structure, etc.), automatically distinguishes and places the speech segments of different speakers into corresponding independent audio tracks through similarity measurement, while strictly retaining the time stamp information of each speech segment in the original audio stream to ensure that each sound track after separation is completely aligned with the original time axis; finally, multiple original audio tracks containing single speaker speech and accurate timing are output, and the technical effect is that not only the complete separation of multiple speech in complex scenes is realized, but also the speech crosstalk and aliasing problems in traditional methods are solved, and the consistency of the speaker's identity is ensured through fine clustering of voiceprint features, combined with the time stamp retention technology, which provides a high-quality, structured audio data basis for subsequent speech transcription, content retrieval, voiceprint recognition and other applications, greatly improving the accuracy and efficiency of speech processing.

[0028] Step two, as Figure 3 The diagram shows the second step of a video translation method based on artificial intelligence provided in this application embodiment. Multiple original audio tracks are input into the voiceprint matching algorithm model. The model checks whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in the timbre library. If so, the AI ​​speaker module is determined as the target AI speaker module that is adapted to the corresponding original audio track. If not, a target AI speaker module that is adapted to the original audio track is generated.

[0029] It should be noted that, based on the multiple original audio tracks obtained in step one, this step yields a suitable target AI speaker module. For example, a "first target AI speaker module" is obtained that matches the "first original audio track." Specifically, multiple original audio tracks are input into the voiceprint matching algorithm model, and the model checks whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched within the voiceprint library. The voiceprint library refers to a database containing multiple high-quality AI speaker modules of different types. These high-quality AI speaker modules can be obtained by adjusting pre-set speech feature parameters or by cloning a large amount of high-quality speech data in advance using a speech cloning model. For an original audio track that matches an AI speaker module, the AI ​​speaker module with a voiceprint similarity higher than the preset threshold is used as the target AI speaker module that matches that original audio track. For original audio tracks that do not match an AI speaker module, the original audio track is input into the speech cloning algorithm model. The model learns, analyzes, and models the speech features to obtain the target AI speaker module. The target AI speaker module refers to the new AI speaker module obtained by cloning, which is synchronously stored in the timbre library and can be directly reused by other video translation tasks that contain the same or similar speakers.

[0030] Furthermore, to determine whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched, the specific steps are as follows: the original audio track is input into a preset voiceprint matching algorithm model to extract the source voiceprint feature vector of the original audio track; the source voiceprint feature vector is compared one by one with the reference voiceprint feature vector of the AI ​​speaker module in a preset timbre library to obtain one or more voiceprint similarity scores; it is determined whether there is a highest score higher than a preset threshold among the one or more voiceprint similarity scores. If the determination result is yes, the AI ​​speaker module corresponding to the highest score in the timbre library is determined as the target AI speaker module that matches the original audio track; if the determination result is no, the original audio track is input into a speech cloning algorithm model, and the speech features of the original audio track are learned, analyzed and modeled by the speech cloning algorithm model to generate a new AI speaker module that matches the timbre of the original audio track, and this new AI speaker module is determined as the target AI speaker module.

[0031] In this embodiment, the original sound track to be processed is input into a preset voiceprint matching algorithm model. The core function of this model is to deeply analyze the audio signal and extract a source voiceprint feature vector that can uniquely identify the speaker's voice identity. It converts complex and unstructured audio waveform data into a high-dimensional, compact and computable mathematical representation, laying the foundation for subsequent quantization comparison and realizing accurate mapping from physical signal to digital identity. The extracted source voiceprint feature vector is compared with the reference voiceprint feature vectors of all AI speaker modules in a preset voice library one by one. The voice library is pre-constructed and contains AI synthesis models of various standard or commonly used voices. The comparison process generates a voiceprint similarity score for each AI speaker module in the voice library by calculating the cosine similarity or Euclidean distance between the vectors, converting the subjective feeling of voice similarity into an objective and quantifiable numerical index, enabling accurate assessment of the matching degree between the original sound track and existing AI voices, and providing reliable data basis for subsequent automated decision-making. Check all generated voiceprint similarity scores to determine if there is any score higher than a preset threshold. This preset threshold is calibrated through a large amount of experimental data and represents the minimum standard of acceptable voice similarity. If the result is yes, i.e. there is a highest score higher than the threshold, it means that there is an AI speaker module in the voice library that is close enough to the original sound track in voice. At this time, the AI speaker module in the voice library corresponding to the highest score is directly determined as the target AI speaker module that matches the original sound track to avoid unnecessary repeated calculation and model generation. With extremely low computational cost and extremely fast response speed, it can directly call existing resources to complete the task, significantly improving the overall processing efficiency, especially suitable for processing a large number of contents with common voice. If the result is no, i.e. all similarity scores are lower than the preset threshold, it means that the voice of the original sound track is unique and cannot be found in the existing voice library. At this time, the on-demand voice cloning process will be started and the original sound track will be input into a voice cloning algorithm model. Through deep learning, fine analysis and modeling of the voice features of the original sound track, the core acoustic characteristics of the unique voice, including voice, pitch, rhythm and pronunciation habits, can be captured and learned. Finally, a new AI speaker module that matches the original sound track in voice is generated, and this newly generated module is determined as the target AI speaker module.

[0032] Step 3, as Figure 4As shown, it is step three of the flowchart of the video translation method based on artificial intelligence provided by the embodiment of the application, converting the voice signal into text form to obtain the original language text; inputting the original language text into the machine translation algorithm model, translating the original language into the target language to obtain the target language text with the same semantics but different languages; inputting the target language text and the target AI speaker module into the speech synthesis algorithm model to obtain the AI dubbing soundtrack with high similarity in voice characteristics but different languages.

[0033] It should be noted that based on the plurality of original sound tracks obtained in step one and the plurality of target AI speaker modules obtained in step two, the corresponding AI dubbing soundtrack is obtained through this step. For example, the "first AI dubbing soundtrack" corresponding to the "first original sound track" is obtained. Specifically, first, input the plurality of original sound tracks into the speech recognition algorithm model to convert the voice signal into text form to obtain the original language text consistent with the voice content. Then input the original language text into the machine translation algorithm model to translate the original language into the target language to obtain the target language text with the same semantics but different languages. Finally, input the target language text and the target AI speaker module into the speech synthesis algorithm model to obtain the AI dubbing soundtrack with high similarity in voice characteristics but different languages.

[0034] Further, the voice signal is converted into text form to obtain the original language text, and the specific steps are as follows: performing signal enhancement processing on the input original sound track and positioning and segmenting the effective voice segment through the voice activity detection algorithm to obtain the pure voice signal; performing frame processing on the pure voice signal, and extracting the mel-frequency cepstrum coefficient of each frame of voice signal to form a feature vector sequence capable of representing the acoustic characteristics of the voice; inputting the feature vector sequence into the speech recognition algorithm model, and the speech recognition algorithm model directly decodes the feature vector sequence into the corresponding text character sequence through the acoustic encoder and the attention decoder; applying the pre-trained neural network language model to the text character sequence for context correction and word sequence optimization, and automatically adding punctuation marks and performing case conversion to obtain the original language text.

[0035] In this embodiment, the original sound track is subjected to signal enhancement processing, background interference is eliminated through noise reduction, echo suppression and other means, and the effective speech segment is accurately positioned and segmented by means of a voice activity detection algorithm, so as to obtain a pure speech signal. This preprocessing stage significantly improves the robustness of subsequent recognition, effectively avoiding the influence of environmental noise on recognition accuracy. Subsequently, the pure speech signal is divided into short-time frames, and the mel-frequency cepstral coefficient of each frame is extracted to construct a feature vector sequence that can fully represent the acoustic characteristics of the speech. This feature extraction method not only compresses the data dimension, but also strengthens the frequency domain information sensitive to human ear perception, providing a more discriminative input for the model. Then, the feature vector sequence is input into a speech recognition model based on an acoustic encoder and an attention decoder. The encoder is responsible for mapping the acoustic features to a high-dimensional semantic representation, and the decoder uses the attention mechanism to dynamically capture the context association and directly decode it into a text character sequence. This end-to-end architecture significantly improves the recognition ability of continuous speech, homophones and complex sentence patterns. Finally, a pre-trained neural network language model is used for post-processing of the text to realize context correction, word sequence optimization, punctuation addition and case conversion, and finally output the original language text with correct grammar and clear semantics. This step greatly improves the naturalness and readability of the text, making the recognition result from "character correct" to "semantic smooth", laying a solid foundation for subsequent translation and synthesis.

[0036] As a further specific description, a target language text with the same semantics but different language is obtained, and the specific steps are: inputting the original language text into a machine translation algorithm model, translating the original language into a target language to obtain a target language text with the same semantics but different language; the machine translation algorithm model loads a preset industry term library and a multilingual dictionary, and accurately matches and translates professional terms and specific expressions in the original language text; through context semantic analysis, the semantic deviation caused by direct translation is corrected to ensure that the semantics of the target language text is consistent with that of the original language text.

[0037] In this embodiment, after the original language text is input into the machine translation algorithm model, the model first loads a preset industry term library and a multilingual dictionary to accurately match and translate professional terms and specific expressions in the original language text, ensuring that the expression of the terms in the target language conforms to the industry specifications and context habits; at the same time, the model uses context semantic analysis technology to intelligently correct semantic deviations, ambiguities or cultural differences that may occur during direct translation, such as identifying and adjusting word order, reference relationships, logical connections, etc., so as to ensure that the target language text is highly consistent with the original language text in semantics and is expressed smoothly and naturally; the technical effect of this process is that not only the accuracy and consistency of professional field text translation are significantly improved, effectively avoiding communication barriers caused by term misinterpretation or semantic understanding deviation, but also the readability of the translation result is enhanced through intelligent semantic correction mechanism.

[0038] As a further specific description, the AI dubbing track highly similar to the voice features of each original sound track but with different languages is obtained, and the specific steps are as follows: The speech synthesis algorithm model first establishes a parameter calling channel corresponding to the target AI speaker module, extracts and loads three groups of core feature parameters from the target AI speaker module, specifically including: timbre parameters, intonation models, and rhythm features; specifically, the timbre parameters include quantifiable acoustic indicators such as fundamental frequency range, harmonic distribution, formant frequency, and bandwidth, which directly determine the basic timbre characteristics of AI dubbing; the second is the intonation model, which covers the rules of sentence rising and falling, stress position labeling, and inter-sentence pause duration setting, to ensure that the AI dubbing conforms to the intonation habits of the corresponding speaker's daily language expression; the third is the rhythm feature, including average speech speed (unit: word / second), speech speed variation coefficient of different semantic paragraphs, and emotional word insertion frequency, to restore the rhythm and rhythm of the original speaker's voice.

[0039] The speech synthesis algorithm model uses the three groups of core feature parameters loaded as the basis, adopts an end-to-end speech synthesis architecture, and converts the target language text into an initial speech signal after being divided by semantics.

[0040] The speech feature alignment technology is started, and the dynamic time warping algorithm is used to compare the timbre, speech speed, and emotional tone features of the initial speech signal with the corresponding reference features of the original sound track at the frame level, and the fundamental frequency, speech speed value, and emotional intensity parameters of the initial speech signal are adjusted in real time. Among them, the timbre adjustment is achieved by correcting the formant frequency deviation, the speech speed adjustment is completed by stretching or compressing the speech frame length, and the emotional tone adjustment is achieved by adjusting the energy fluctuation range of the speech signal, so that the adjusted speech signal meets the matching degree of the corresponding original sound track in terms of timbre similarity, speech speed matching degree, and emotional tone matching degree, and the matching degree is not less than the preset matching degree threshold.

[0041] After completing the speech signal optimization, the speech synthesis algorithm model reads the original timestamp information of the corresponding original sound track (including the starting time point, ending time point, and frame interval data of the speech segment), and accurately binds the optimized speech signal with the timestamp information, to ensure that each segment of the AI dubbing track corresponds to the time position of the original sound track, and finally outputs the AI dubbing track corresponding to each original sound track.

[0042] In this embodiment, the speech synthesis algorithm model first establishes a parameter call channel with the target AI speaker module, extracting and loading three sets of core feature parameters: timbre parameters (including quantifiable acoustic indicators such as fundamental frequency range, harmonic distribution, formant frequency, and bandwidth, which directly determine the basic timbre characteristics of the AI ​​dubbing), intonation model (covering the pitch rise and fall rules of sentences, stress position marking, and pause duration settings between sentences to ensure that the AI ​​dubbing conforms to the original speaker's language expression habits), and rhythm features (including average speech rate, speech rate variation coefficients of different semantic segments, and frequency of interjection insertion, used to restore the rhythm and rhyme of the original speaker). Based on this, the model adopts an end-to-end speech synthesis architecture, converting the target language text into initial speech signals sentence by sentence after semantic segmentation. Subsequently, speech feature alignment technology is activated, using a dynamic time warping algorithm to perform frame-level comparison of the timbre, speech rate, and emotional tone features of the initial speech signal with the baseline features of the original audio track, and to fine-tune the fundamental frequency, speech rate, and emotional intensity parameters in real time—the timbre is adjusted through correction. The peak frequency deviation is optimized, the speech rate is matched by stretching or compressing the speech frame duration, and the emotional tone is adapted by adjusting the energy fluctuation range of the speech signal. This ensures that the adjusted speech signal is not lower than the preset matching threshold in terms of timbre similarity, speech rate matching, and emotional tone fit. Finally, the model reads the original timestamp information of the original audio track (including start time, end time, and frame interval data), and accurately binds the optimized speech signal with the timestamp. This ensures that each segment of the AI ​​dubbing audio track corresponds completely to the time position of the original audio track, and finally outputs the AI ​​dubbing audio track corresponding to each original audio track. The technical effects are reflected in the fact that it not only achieves highly personalized synthesis from text to speech, but also ensures strong consistency between the synthesized speech and the original speaker's voice through multi-dimensional feature modeling and dynamic alignment mechanism. Furthermore, the timestamp binding technology ensures the temporal accuracy of the audio track, providing highly realistic, natural, fluent, and highly personalized speech output capabilities for scenarios such as film and television dubbing, virtual anchors, and cross-language content production.

[0043] Step 4, as follows Figure 5 The diagram shown is a flowchart of step four of an AI-based video translation method provided in this application embodiment. Independent video streams and multiple AI-generated audio tracks are input into a lip-syncing algorithm model. The model tracks and analyzes the lip dynamics of each speaker in the video frames. The AI-generated audio tracks are used to re-drive and replace the speaker's lip animation, resulting in a target video stream synchronized with the AI-generated audio. The target video stream and multiple AI-generated audio tracks are then compressed to complete audio-video mixing, encoding, and encapsulation, yielding the translated video.

[0044] It should be noted that based on the independent video stream obtained in step one, the international sound frequency, and the plurality of AI dubbing audio tracks obtained in step three, through this step, the translated video is obtained. Specifically, first, the independent video stream and the plurality of AI dubbing audio tracks are input into the lip synchronization algorithm model, the lip dynamic information of each speaker in the video frame is tracked and analyzed, the lip animation of the speaker is re-driven and replaced using the new voice, and the target video stream with the lip of the speaker is obtained. Finally, the target video stream, the international sound frequency and the plurality of AI dubbing audio tracks are suppressed together, the audio and video are mixed, encoded, packaged and the timestamps of each part are accurately synchronized to obtain the translated video.

[0045] Further, the target video stream with the lip action synchronized with the AI dubbing is obtained, and the specific steps are as follows: The lip synchronization algorithm model first performs a preprocessing operation on each frame image of the independent video stream, removes image noise through Gaussian filtering, and enhances image contrast using histogram equalization. The face detection sub-module is started, and the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm is used to scan the preprocessed frame image to identify all face-containing regions in the image, and output the boundary box coordinates of each face region.

[0046] Based on the face region boundary box, the lip positioning sub-module further locates and extracts the pixel coordinates of the core key points (such as the upper lip peak, lower lip valley, and corner of the mouth) around the lips through a key point detection algorithm, forming a set of lip key point coordinates corresponding to each frame image. Then, by calculating the difference value of the key point coordinates between adjacent frames, the dynamic change data of the lip key points in the time dimension is obtained, including the key point displacement distance, displacement speed, and angle change value.

[0047] The lip synchronization algorithm model reads the speech timestamp (including the start time and end time of each speech phoneme) and pronunciation features (covering vowel / consonant types, mouth opening degree level, and tongue height parameters) of each AI dubbing audio track, and establishes a "speech pronunciation-lip action" mapping database. The database pre-stores the corresponding relationship between different pronunciation types and lip key point positions (such as the lip key point position corresponding to the upper lip lifting and the mouth opening degree when pronouncing "a", and the lip key point position corresponding to the lip slightly opening and the corner of the mouth pulling back when pronouncing "s").

[0048] Based on the above lip key point dynamic change data, the speech timestamp and pronunciation features of the AI dubbing audio track, the dynamic time warping algorithm is used to align the speech time axis and the video frame time axis, match the pronunciation features of each speech phoneme with the lip key point positions in the corresponding time interval, and establish a one-to-one mapping relationship between the lip action and the speech pronunciation.

[0049] The animation generation submodule adopts an animation generation technology based on a morphable model, calculates target positions of the lip key points according to pronunciation features of the current phonetic syllable based on the established mapping relationship, generates smooth motion trajectories of the lip key points between adjacent frames through a linear interpolation algorithm, drives the lip key points to move from original positions to the target positions, and further replaces corresponding lip animation areas in the original video frames.

[0050] In the animation replacement process, through inter-frame time synchronization verification, it is ensured that the starting time and the duration of the lip action after replacement are completely matched with the time of the corresponding AI dubbing syllable, the time synchronization error is controlled within a preset synchronization error threshold range, and finally the target video stream with precise synchronization of the lip action and the AI dubbing is obtained.

[0051] In the present embodiment, the mouth shape synchronization algorithm model first performs a preprocessing operation on each frame image of the independent video stream, removes image noise through Gaussian filtering, and enhances image contrast using histogram equalization, effectively improving image quality and reducing interference in subsequent processing; then a face detection sub-module is started, and the MTCNN algorithm is used to scan the preprocessed frame image, accurately identify all face regions in the image and output the boundary box coordinates, laying the foundation for subsequent lip positioning; based on the face region boundary box, the lip positioning sub-module further locates and extracts the pixel coordinates of the lip and surrounding key points (such as the upper lip peak, lower lip valley, and corner of the mouth) through a key point detection algorithm, forming a set of lip key point coordinates corresponding to each frame image, and obtaining the dynamic change data of the lip key points in the time dimension through the difference calculation of the key point coordinates between adjacent frames, including displacement distance, speed, and angle change value, thereby accurately capturing the details of lip movement; the model then reads the speech timestamps and pronunciation features (such as vowel / consonant types, mouth opening degree levels, and tongue height parameters) of each AI dubbing soundtrack, establishes a "speech pronunciation-lip action" mapping database, which pre-stores the corresponding relationship between different pronunciation types and lip key point position combinations, providing a basis for accurate matching of phonemes and lip actions; based on the above data, the model aligns the speech time axis and the video frame time axis through a dynamic time warping algorithm, matches the pronunciation features of each speech syllable with the lip key point positions in the corresponding time interval, and establishes a one-to-one mapping relationship between lip action and speech pronunciation; the animation generation sub-module uses a morphing model-based animation generation technique to calculate the target position of the lip key points based on the mapping relationship according to the pronunciation features of the current speech syllable, and generates smooth motion trajectories for the lip key points between adjacent frames through a linear interpolation algorithm, driving the lip key points to move from the original position to the target position, and then replacing the corresponding lip animation area in the original video frame; during the animation replacement process, the time synchronization is checked between frames to ensure that the start time and duration of the replaced lip action completely match the time of the corresponding AI dubbing syllable, with a time synchronization error controlled within a preset synchronization error threshold range, and finally outputting a target video stream with accurate synchronization between lip action and AI dubbing; the technical effect lies in not only achieving high-precision lip action capture and dynamic modeling, but also ensuring high consistency and natural fluency between AI dubbing and lip action through multi-modal data fusion and accurate time axis alignment, effectively solving the problems of mouth shape asynchronization and stiff action in traditional video dubbing, and providing a realistic, natural, and high-performance visual presentation solution for virtual anchors, film post-production, digital human interaction, and other applications.

[0052] As a further specific description, the translated video is obtained, and the specific steps are as follows: Using audio and video compression tools with multi-track synchronous integration function (such as FFmpeg, Adobe MediaEncoder), first read the global timeline information of the target video stream (including video frame start timestamp, frame interval duration) and the independent timestamp data of each AI dubbing track (including the start / end time of each dubbing track), through the built-in time axis alignment algorithm of the tool, each AI dubbing track is synchronized and integrated with the target video stream according to the timestamp, ensuring that each AI dubbing track is completely matched with the corresponding speaker's picture frame in the target video stream, without time displacement.

[0053] During the synchronous integration process, the audio and video compression tool loads the preset audio gain parameter, which includes the target loudness value of the audio normalization algorithm and the dynamic range compression ratio. By adjusting the gain of each AI dubbing track audio signal segment by segment, the volume of the segment below the target loudness value is linearly amplified, and the volume of the segment above the target loudness value is compressed. At the same time, the noise threshold is filtered to remove the noise in the audio signal. Finally, the volume deviation of each AI dubbing track is controlled within the preset volume deviation threshold range, and the signal-to-noise ratio is not less than the preset signal-to-noise ratio threshold.

[0054] After completing the audio and video synchronous integration and audio gain adjustment, the target video stream is compressed and encoded using the H.265 / AV1 video encoding standard; if the AV1 encoding standard is selected, the intra-frame prediction and inter-frame prediction dual mode is enabled, and the encoding efficiency parameter is set to "medium-high" level to balance the encoding speed and compression effect; at the same time, the integrated multi-AI dubbing track is encoded using the AAC audio encoding standard, the audio sampling rate is set to the preset sampling rate standard, the number of audio channels is consistent with the original video audio channel number, and the audio code rate is set to the preset audio code rate threshold.

[0055] In this embodiment, the audio and video compression tool with multi-track synchronous integration function (such as FFmpeg, Adobe Media Encoder) is used. First, the global timeline information of the target video stream (accurate to millisecond level, including video frame start timestamp and frame interval duration) and the independent timestamp data of each AI dubbing track (including the start / end time of each dubbing track) are read. Through the built-in timeline alignment algorithm of the tool, each AI dubbing track is synchronized and integrated with the target video stream at the frame level according to the timestamp, ensuring that the speech segment of each AI dubbing track completely matches the picture frame of the corresponding speaker in the target video stream without time displacement, thereby realizing accurate synchronization of audio and video. During the synchronous integration process, the tool loads the preset audio gain parameters, including the target loudness value of the audio normalization algorithm and the dynamic range compression ratio, and adjusts the audio signal of each AI dubbing track in segments. The segments with volume lower than the target loudness value are linearly amplified, and the segments with volume higher than the target loudness value are compressed. At the same time, the noise threshold is used to filter and remove the noise in the audio signal. Finally, the volume deviation of each AI dubbing track is controlled within the preset volume deviation threshold range, and the signal-to-noise ratio is not less than the preset signal-to-noise ratio threshold, effectively improving the clarity and consistency of the audio. After completing the audio and video synchronous integration and audio gain adjustment, the target video stream is compressed and encoded using the H.265 / AV1 video encoding standard. If the AV1 encoding standard is selected, the intra-frame prediction and inter-frame prediction dual mode is enabled, and the encoding efficiency parameter is set to "medium-high" level to balance the encoding speed and compression effect. At the same time, the integrated multi-AI dubbing track is encoded using the AAC audio encoding standard, the audio sampling rate is set to the preset sampling rate standard, the number of sound channels is consistent with the original video audio sound channel number, and the audio code rate is set to the preset audio code rate threshold, ensuring that the final output file has low storage and transmission cost while ensuring high-quality audio and video experience. The technical effect is that not only the millisecond-level accurate synchronization of multiple tracks and video frames is realized, effectively avoiding the phenomenon of audio and picture out of sync, but also the consistency and clarity of the overall sound quality are significantly improved through intelligent audio gain and noise reduction processing. Combined with efficient video and audio encoding technology, the compression efficiency and compatibility of the output file are greatly optimized.

[0056] As a further specific description, the translated video also includes: The audio and video compression tool encapsulates the encoded video data and audio data according to a preset encapsulation format (optional MP4, MOV or MKV); when the MP4 format is selected, the ISO Base Media File Format standard is used for encapsulation, and streaming media playback is supported; when the MOV format is selected, the metadata field of the QuickTime file format is retained, facilitating subsequent editing by professional video editing software; when the MKV format is selected, multiple audio track markers are supported, and language identifiers are added to each AI dubbing track; during the encapsulation process, the resolution verification module built into the tool compares the picture resolution of the encoded target video stream with the original video resolution in real time, ensures that the picture resolution of the generated translated video deviates from the original video by no more than a preset deviation threshold, and keeps the video frame rate and color space consistent with the original video, and finally outputs a translated video that can be directly played in mainstream players.

[0057] In the present embodiment, the audio and video compression tool encapsulates the encoded video data and audio data according to a preset encapsulation format (optional MP4, MOV or MKV); when the MP4 format is selected, the ISO Base Media File Format standard is used for encapsulation, ensuring good streaming media playback compatibility of the file, suitable for online on-demand and mobile playback scenarios; when the MOV format is selected, the metadata field of the QuickTime file format is retained, facilitating subsequent editing and secondary processing by professional video editing software (such as Final Cut Pro, Adobe Premiere Pro), meeting professional production requirements; when the MKV format is selected, multiple audio track markers are supported, and language identifiers are added to each AI dubbing track, facilitating users to switch different language tracks according to needs during playback, improving the use flexibility of multilingual content; during the encapsulation process, the resolution verification module built into the tool compares the picture resolution of the encoded target video stream with the original video resolution in real time, ensures that the picture resolution of the generated translated video deviates from the original video by no more than a preset deviation threshold, and keeps the video frame rate and color space consistent with the original video, effectively avoiding quality degradation or display abnormalities caused by format conversion; the technical effect is that not only does the diversified encapsulation format adapt to the needs of different playback and editing scenarios, improving the universality and professional adaptability of the file, but also the strict resolution and picture parameter verification mechanism ensures that the output video is highly consistent with the original video in visual quality.

[0058] As Figure 6As shown, a structural schematic diagram of a video translation system based on artificial intelligence provided by the embodiment of the application, the video translation system based on artificial intelligence provided by the embodiment of the application comprises a original sound track extraction module, a target AI speaker module adaptation module, an AI dubbing generation module and a lip synchronization and video synthesis module: the original sound track extraction module is used for processing the video to be translated by using an audio-video separation technology, and independent audio streams and independent video streams are obtained by separation, the independent audio streams are input into a speech separation algorithm model, and a plurality of original sound tracks are obtained; the target AI speaker module adaptation module is used for inputting the plurality of original sound tracks into a voiceprint matching algorithm model, detecting whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in a timbre library, if yes, the AI speaker module is determined as a target AI speaker module adapted to the corresponding original sound track, and if no, a target AI speaker module adapted to the original sound track is generated; the AI dubbing generation module is used for converting a speech signal into a text form to obtain original language text, inputting the original language text into a machine translation algorithm model to translate the original language into a target language to obtain a target language text with the same semantics but different language, and inputting the target language text and the target AI speaker module into a speech synthesis algorithm model to obtain AI dubbing audio tracks with high similarity in speech characteristics but different language to each original sound track; the lip synchronization and video synthesis module is used for inputting the independent video stream and the plurality of AI dubbing audio tracks into a lip synchronization algorithm model, tracking and analyzing lip dynamic information of each speaker in a video frame, re-driving and replacing the lip animation of the speaker by using the AI dubbing audio tracks, obtaining a target video stream with synchronized lip action and AI dubbing, and compressing the target video stream and the plurality of AI dubbing audio tracks to complete audio-video mixing, encoding and packaging to obtain a translated video.

[0059] In the embodiment, in the process of video translation, the first step is to extract and classify the speech. Specifically, by means of an audio-video separation technology, a speech separation algorithm model and a voiceprint clustering algorithm model, clean speech signals are accurately extracted from the original video. Then, according to different speakers, the speech signals are classified and systematically arranged, thereby forming original speech data sets of different speakers, and the data sets are respectively arranged in a plurality of original sound tracks, thereby laying a foundation for subsequent processing.

[0060] In the second step, the core is to build AI speaker modules. Based on the original sound tracks of different speakers obtained in the first step, a voiceprint matching algorithm model is first used to find AI speaker modules similar to the original voice features in the existing timbre library. If no suitable module is found in the timbre library, a voice cloning algorithm model is enabled. This model will deeply learn, comprehensively analyze and accurately model the features of the original voice, and finally obtain multiple AI speaker modules. It is worth noting that each AI speaker module corresponds to the corresponding original sound track, which contains voice features based on original voice matching or cloning.

[0061] The main task of the third step is to generate AI dubbing audio tracks. Based on the original sound tracks of different speakers, combined with the AI speaker modules obtained in the second step, the original voice is recognized by a speech recognition algorithm model, and the recognized content is translated into the target language by using a machine translation algorithm model. Finally, the voice is synthesized by using a speech synthesis algorithm model, thereby obtaining multiple AI dubbing audio tracks. Each AI dubbing audio track corresponds to the corresponding original sound track, and the AI dubbing data set contained therein has the same semantics as the original sound track, and the voice features are highly similar, except that the language used is different.

[0062] In the fourth step, the integration and adaptation of audio and video need to be completed. Through a lip synchronization algorithm model, the generated AI dubbing audio track is kept in synchronization with the lip shape of the character in the video, and then an audio and video compression technology is used to adapt and integrate the audio and video files obtained in the previous steps, and finally a complete translated video is obtained.

[0063] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0064] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions described in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-usable storage medium that can guide computers or other programmable data processing devices to work.Figure 1 one or more processes and / or functions described in one or more blocks Figure 1 means for carrying out a function specified by the flow or flows and / or blocks

Claims

1. A video translation method based on artificial intelligence, characterized in that, Includes the following steps: The audio-video separation technology is used to process the video to be translated, separating it into independent audio streams and independent video streams. The independent audio streams are then input into the speech separation algorithm model to obtain multiple original audio tracks. Multiple original audio tracks are input into the voiceprint matching algorithm model. The model checks whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in the timbre library. If so, the AI ​​speaker module is determined as the target AI speaker module that is adapted to the corresponding original audio track. If not, a target AI speaker module that is adapted to the original audio track is generated. The speech signal is converted into text to obtain the original language text; the original language text is input into a machine translation algorithm model to translate the original language into the target language, resulting in target language text with the same semantics but different language; the target language text and the target AI speaker module are input into a speech synthesis algorithm model to obtain an AI dubbing track that is highly similar to the speech features of each of the original audio tracks but different language. The independent video stream and multiple AI dubbing audio tracks are input into the lip-sync algorithm model. The algorithm tracks and analyzes the lip dynamics of each speaker in the video frames. The speech of the AI ​​dubbing audio tracks is used to re-drive and replace the speaker's lip animation to obtain a target video stream in which the lip movements are synchronized with the AI ​​dubbing. The target video stream and multiple AI dubbing audio tracks are then compressed to complete the audio-video mixing, encoding, and encapsulation to obtain the translated video.

2. The video translation method based on artificial intelligence according to claim 1, characterized in that, The independent audio stream input speech separation algorithm model yields multiple original audio tracks, specifically including: The speech separation algorithm model extracts speech signals from the mixed audio signals of the independent audio streams to obtain pure speech audio; The pure speech audio is input into the voiceprint clustering algorithm model to distinguish and cluster the speech segments of different speakers. Each speaker's speech segment is placed into a separate audio track, and the original timestamp of each speech segment is retained, ultimately resulting in multiple original audio tracks.

3. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps for determining whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched are as follows: The original audio track is input into a preset voiceprint matching algorithm model to extract the source voiceprint feature vector of the original audio track; the source voiceprint feature vector is compared one by one with the reference voiceprint feature vector of the AI ​​speaker module in a preset timbre library to obtain one or more voiceprint similarity scores. If a maximum score higher than a preset threshold exists among the one or more voiceprint similarity scores, and the result is yes, then the AI ​​speaker module corresponding to the maximum score in the timbre library is identified as the target AI speaker module that matches the original audio track. If the result is no, then the original audio track is input into a speech cloning algorithm model, and the speech features of the original audio track are learned, analyzed, and modeled by the speech cloning algorithm model to generate a new AI speaker module that matches the timbre of the original audio track, and this new AI speaker module is identified as the target AI speaker module.

4. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps for converting speech signals into text to obtain the original language text are as follows: The input original audio track is subjected to signal enhancement processing, and effective speech segments are located and segmented using a speech activity detection algorithm to obtain a clean speech signal; The pure speech signal is processed by framing, and the Mel frequency cepstral coefficients are extracted from each frame of the speech signal to form a feature vector sequence that can characterize the acoustic properties of the speech. The feature vector sequence is input into a speech recognition algorithm model, which directly decodes the feature vector sequence into the corresponding text character sequence through an acoustic encoder and an attention decoder. A pre-trained neural network language model is applied to the text character sequence to perform context correction and word order optimization, and punctuation marks are automatically added and case conversion is performed to obtain the original language text.

5. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps for obtaining target language texts with the same semantics but different languages ​​are as follows: The original language text is input into a machine translation algorithm model to translate the original language into the target language, resulting in target language text with the same semantics but different language. The machine translation algorithm model loads a pre-set industry terminology database and a multilingual dictionary to accurately match and translate the professional terms and specific expressions in the original language text. By analyzing the contextual semantics, semantic deviations caused by literal translation are corrected to ensure semantic consistency between the target language text and the source language text.

6. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps for obtaining an AI dubbing track that is highly similar in speech features to each of the original audio tracks but uses a different language are as follows: The speech synthesis algorithm model first establishes a parameter call channel with the corresponding target AI speaker module, and extracts and loads three sets of core feature parameters from the target AI speaker module, specifically including: timbre parameters, intonation model and rhythm features; The speech synthesis algorithm model is based on three sets of loaded core feature parameters and adopts an end-to-end speech synthesis architecture to convert the target language text into initial speech signals sentence by sentence after semantic segmentation. The speech feature alignment technology is activated. Through the dynamic time warping algorithm, the timbre, speech rate, and emotional tone features of the initial speech signal are compared with the baseline features of the corresponding original audio track at the frame level. The fundamental frequency, speech rate, and emotional intensity parameters of the initial speech signal are then finely adjusted in real time. After optimizing the speech signal, the speech synthesis algorithm model reads the original timestamp information of the corresponding original audio track, accurately binds the optimized speech signal with the timestamp information, and finally outputs the AI ​​dubbing audio track corresponding to each original audio track.

7. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps for obtaining the target video stream synchronized with lip movements and AI dubbing are as follows: The lip-sync algorithm model first performs preprocessing operations on each frame of the independent video stream, removing image noise through Gaussian filtering and enhancing image contrast through histogram equalization. The face detection submodule is started, and the MTCNN algorithm is used to scan the preprocessed frame image to identify all regions containing faces in the image and output the bounding box coordinates of each face region. Based on the face region bounding box, the lip localization submodule further uses a key point detection algorithm to form a set of lip key point coordinates for each frame of the image. Then, by calculating the difference between the key point coordinates between adjacent frames, the dynamic change data of the lip key points in the time dimension is obtained. The lip-sync algorithm model reads the speech timestamps and pronunciation features of each AI dubbing track and establishes a "speech pronunciation - lip movement" mapping database; Based on the dynamic change data of the above lip key points, the speech timestamps and pronunciation features of the AI ​​dubbing audio track, the speech time axis and the video frame time axis are aligned by the dynamic time warping algorithm. The pronunciation features of each speech syllable are matched with the positions of the lip key points in the corresponding time interval, and a one-to-one mapping relationship between lip movements and speech pronunciation is established. Based on the pronunciation characteristics of the current speech syllable, the target position of the lip key point is calculated. A smooth motion trajectory of the lip key point between adjacent frames is generated by a linear interpolation algorithm, which drives the lip key point to move from its original position to the target position, thereby replacing the corresponding lip animation area in the original video frame. During the animation replacement process, the target video stream is obtained by verifying the inter-frame time synchronization to ensure that the lip movements are accurately synchronized with the AI ​​dubbing.

8. The video translation method based on artificial intelligence according to claim 1, characterized in that, The specific steps to obtain the translated video are as follows: An audio and video compression tool with multi-track synchronization integration function is used to read the global timeline information of the target video stream and the independent timestamp data of each AI dubbing track. Through the timeline alignment algorithm built into the tool, each AI dubbing track is synchronized and integrated with the target video stream at the frame level according to the timestamp. During the synchronization and integration process, the audio and video compression tool loads preset audio gain parameters, which include the target loudness value and dynamic range compression ratio of the audio normalization algorithm, and adjusts the gain of each AI dubbing track segment by segment.

9. The video translation method based on artificial intelligence according to claim 1, characterized in that, The obtained translated video also includes: The audio and video compression tool encapsulates the encoded video and audio data according to a preset encapsulation format. When selecting MP4 format, it is packaged using the ISO Base Media File Format standard and supports streaming media playback; when selecting MOV format, it retains the metadata fields of the QuickTime file format; when selecting MKV format, it supports multi-audio track marking and adds language identifiers to each AI dubbing track. During the encapsulation process, the tool's built-in resolution verification module compares the resolution of the encoded target video stream with the resolution of the original video stream in real time, outputting a translated video that can be played directly in mainstream players.

10. An artificial intelligence-based video translation system, wherein the artificial intelligence-based video translation system applies the artificial intelligence-based video translation method as described in any one of claims 1-9, characterized in that, include: Original audio track extraction module, target AI speaker module adaptation module, AI dubbing generation module, and lip-syncing and video synthesis module: The original audio track extraction module is used to process the video to be translated using audio-video separation technology, separate it into independent audio streams and independent video streams, and input the independent audio streams into the speech separation algorithm model to obtain multiple original audio tracks. The target AI speaker module adaptation module is used to input multiple original audio tracks into the voiceprint matching algorithm model, and detect whether an AI speaker module with a voiceprint similarity higher than a preset threshold can be matched in the timbre library. If so, the AI ​​speaker module is determined as the target AI speaker module adapted to the corresponding original audio track. If not, a target AI speaker module adapted to the original audio track is generated. The AI ​​dubbing generation module is used to convert speech signals into text form to obtain the original language text; input the original language text into a machine translation algorithm model to translate the original language into the target language to obtain the target language text with the same semantics but different language; input the target language text and the target AI speaker module into a speech synthesis algorithm model to obtain an AI dubbing audio track that is highly similar to the speech features of each of the original audio tracks but different language. The lip-sync and video synthesis module is used to input the independent video stream and multiple AI dubbing tracks into the lip-sync algorithm model, track and analyze the lip dynamics information of each speaker in the video frame, use the speech of the AI ​​dubbing tracks to re-drive and replace the speaker's lip animation, obtain a target video stream in which the lip movements are synchronized with the AI ​​dubbing, and compress the target video stream and multiple AI dubbing tracks to complete audio and video mixing, encoding and encapsulation to obtain the translated video.

Images