Video translation method and device, and electronic equipment
By separating the video into driver video and audio, and combining light and shadow processing, translation processing and lip-driven model, the problem of insufficient robustness of video translation in the prior art in complex environments is solved, and efficient, accurate and natural Multilingual video translation improves the authenticity and robustness of the translation.
Patent Information
- Application Number
- CN202410930995.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
AI Technical Summary
In the face of complex environments and varied perspectives, existing video translation technologies are difficult to achieve highly robust and natural video translation, especially the fine-tuning and authenticity recovery of specific characters.
By separating the original video into driver video and audio, light and shadow processing and translation processing are performed, and synthetic reasoning is performed using the lip-driven model to combine face recognition, multi-dimensional reference frame construction, color harmony and face Post back algorithm to generate multilingual, high-fidelity, strong and robust lip-driven output.
It realizes efficient, accurate and natural video translation in complex environments, ensures multilingual support and high fidelity, and improves the robustness and authenticity of video translation.
Smart Images

Figure CN118945416A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image processing technology, and more specifically, to a video translation method, device, and electronic device. Background Art
[0002] The widespread dissemination of short videos and the emergence of digital human technology have greatly helped content producers expand their audiences and improve the efficiency of overall content production. They have also helped to attract global localized groups and further enhance the influence and commercial potential of content.
[0003] Video translation is a high-level application of digital human technology, and has great commercial value in cross-border knowledge dissemination, overseas film and television production, real-time video conferencing, etc. In terms of technology, video multi-language translation technology is based on the STT (Speech to Text) algorithm and the TTS (Text to Speech) algorithm, with superimposed sound cloning and lip synchronization functions, supporting the translation of live-action videos and restoring the authenticity of the video. At present, the lip-sync-driven technology has the following main shortcomings: direct video translation, no fine-tuning of specific character IDs; video translation is oriented to an open environment, the user's input video environment is complex, the perspective is changeable, the transitions are rich, and the target is not unique, which requires a high level of algorithm robustness. Summary of the invention
[0004] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide a video translation method, device, and electronic device, which can realize video translation efficiently, accurately and naturally.
[0005] In a first aspect, an embodiment of the present application provides a video translation method, comprising:
[0006] Separate the original video into driving video and driving audio;
[0007] Performing light and shadow processing on the driving video;
[0008] performing translation processing on the driving audio; and
[0009] The driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred through the lip-sync driving model to form a synthetic video.
[0010] Furthermore, before performing light and shadow processing on the driving video, it also includes:
[0011] The human face in the driving video is recognized and processed.
[0012] Furthermore, the identifying and processing of faces in the driving video includes:
[0013] Performing facial key point extraction, facial segmentation and facial detection processing on the face in the driving video; and
[0014] The human face in the driving video is processed by key point balance, face alignment and face pear-shaped region generation.
[0015] Furthermore, after the face in the driving video is recognized and processed, and before the driving video is subjected to light and shadow processing, the method further includes:
[0016] A multi-dimensional reference frame is constructed for the human face in the driving video.
[0017] Furthermore, after synthesizing and reasoning the driving video after light and shadow processing and the driving audio after translation processing by the lip-driven model to form a synthesized video, the method further includes:
[0018] The composite video is processed by a color harmonization algorithm.
[0019] Further, after the composite video is processed by the color harmony algorithm, the method further includes:
[0020] The synthesized video is processed by a face pasting algorithm.
[0021] Furthermore, before separating the original video into the driving video and the driving audio, the method further includes:
[0022] The original video is subjected to scene screening and frame rate conversion processing.
[0023] In a second aspect, the embodiment of the present application further provides a video translation device, including:
[0024] A video separation module, used for separating the original video into a driving video and a driving audio;
[0025] A light and shadow processing module, used for performing light and shadow processing on the driving video;
[0026] An audio translation module, used for translating the driving audio; and
[0027] The video synthesis module is used to perform synthesis reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-driven model to form a synthetic video.
[0028] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the video translation method according to the first aspect when executing the program.
[0029] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the video translation method according to the first aspect above.
[0030] In a fifth aspect, an embodiment of the present application further provides a computer program product on which a computer program is stored, wherein the computer program is used to implement the video translation method according to the first aspect above.
[0031] The embodiments of the present application bring the following beneficial effects:
[0032] The video translation method provided in the embodiment of the present application separates the original video into a driving video and a driving audio, performs light and shadow processing on the driving video, performs translation processing on the driving audio, and performs synthetic reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-driven model to form a synthetic video, thereby ensuring that multi-language, high-fidelity, and highly robust lip-driven output can be generated, and video translation can be achieved efficiently, accurately, and naturally. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.
[0034] Figure 1 A flowchart of a video translation method provided in an embodiment of the present application;
[0035] Figure 2 A schematic diagram of the overall framework of the video transfer method provided in an embodiment of the present application;
[0036] Figure 3 A structural block diagram of a video translation device provided in an embodiment of the present application;
[0037] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0038] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments described in the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0040] In the specification and claims of this application and the above-mentioned drawings, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "multiple" means two or more. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.
[0041] Figure 1 FIG. 1 is a flow chart of a video translation method according to an embodiment of the present application. Figure 1 As shown, the video translation method of the embodiment of the present application includes the following steps:
[0042] S101: separating the original video into a driving video and a driving audio;
[0043] Specifically, the original video usually has embedded audio. Before video translation, the audio and video in the video need to be separated so as to be edited and processed separately.
[0044] S102: performing light and shadow processing on the driving video;
[0045] Specifically, the light and shadow processing mentioned here includes video fade-in and fade-out, black and white transition, and overexposure and overdark processing, and its purpose is to provide the image quality of the driving video, so as to facilitate subsequent translation processing.
[0046] S103: performing translation processing on the driving audio;
[0047] That is, it is necessary to separate the human voice of the driving audio and perform audio translation, which specifically includes: translating the original text of the driving audio into the target language text. After the original text of the driving audio is obtained, it needs to be translated into the target language text. The translation method includes manual translation or intelligent machine translation, and the target language includes languages of various countries or other languages. After the original text of the driving audio is translated into the target language text, it is necessary to synthesize the target language text into the target audio through track splitting operations, etc.
[0048] S104: Performing synthetic reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-sync driving model to form a synthetic video.
[0049] Specifically, the lip-driven model includes the DINET (Deformation Inpainting Network) model, which is used to improve the lip-driven effect. After the driving video completes the light and shadow processing and the driving audio completes the translation processing, it needs to be synthesized and inferred by the lip-driven model to form a synthetic video.
[0050] The DINET model uses spatial deformation on the feature map of the reference image to better preserve high-frequency texture details. Specifically, the DINET model consists of a deformation module and a restoration module. In the deformation module, in order to align with the input driving audio and the head posture in the original image, five reference face images are adaptively spatially deformed, thereby generating a deformed feature map encoding the mouth shape information in each frame. In the restoration module, in order to generate a visual dubbing effect for the face, a feature decoder is responsible for adaptively fusing the deformed feature map and other attributes (head posture and facial expression) from the original feature map to the mouth. Finally, the DINET model achieves a visual dubbing effect for the face with rich texture details.
[0051] Therefore, the video translation method provided in the embodiment of the present application separates the original video into a driving video and a driving audio, performs light and shadow processing on the driving video, performs translation processing on the driving audio, and performs synthetic reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-driven model to form a synthetic video, thereby ensuring that multi-language, high-fidelity, and highly robust lip-driven output can be generated, and video translation can be achieved efficiently, accurately, and naturally.
[0052] Furthermore, in some embodiments of the present application, before performing light and shadow processing on the driving video, the method further includes:
[0053] The human face in the driving video is recognized and processed.
[0054] Specifically, there are problems such as jitter, instability, and face occlusion in the driving video. Therefore, before performing light and shadow processing on the driving video, it is necessary to identify the faces in the driving video and process the faces through corresponding algorithms to solve the problems of jitter, instability, and face occlusion.
[0055] Furthermore, in some embodiments of the present application, the step of recognizing and processing a face in the driving video includes:
[0056] Performing facial key point extraction, facial segmentation and facial detection processing on the face in the driving video; and
[0057] The human face in the driving video is processed by key point balance, face alignment and face pear-shaped region generation.
[0058] That is, the face recognition mentioned above specifically includes face key point extraction, face segmentation and face detection processing, so that the faces in the driving video can be accurately identified, and the face processing includes key point balance, face alignment and face pear-shaped area generation processing, so as to solve the problems of face shaking instability and face occlusion, and improve the quality of the driving video.
[0059] Furthermore, in some embodiments of the present application, after the face in the driving video is recognized and processed, and before the driving video is subjected to light and shadow processing, the method further includes:
[0060] A multi-dimensional reference frame is constructed for the human face in the driving video.
[0061] Specifically, after recognizing and processing the faces in the driving video and before performing light and shadow processing on the driving video, a multi-dimensional reference frame is constructed for the faces in the driving video, which is a reference frame including multi-dimensional information. This can better capture the correlation between the generated frame and the reference frame, thereby fitting the cross-distribution scenes between the generated frame and the reference frame.
[0062] Among them, the construction of the multi-dimensional reference frame includes the judgment of facial occlusion attributes, video lighting attributes and facial angle attributes, so as to construct the reference frame in multiple dimensions.
[0063] Furthermore, in some embodiments of the present application, after synthesizing and reasoning the driving video after light and shadow processing and the driving audio after translation processing by the lip-driven model to form a synthesized video, the method further includes:
[0064] The composite video is processed by a color harmonization algorithm.
[0065] Specifically, after the lip-sync driving model performs synthetic inference on the driving video after light and shadow processing and the driving audio after translation processing to form a synthetic video, the synthetic video is processed by a color harmonization algorithm. The color harmonization algorithm can further slow down the facial colors before and after generation, making it closer to reality, and improving user experience and application effects.
[0066] Furthermore, in some embodiments of the present application, after the composite video is processed by the color harmony algorithm, the method further includes:
[0067] The synthesized video is processed by a face pasting algorithm.
[0068] Specifically, the face pasting algorithm is also called the 3D face pasting algorithm. It uses the three-dimensional facial point information as a medium to distort the face before and after generation in the three-dimensional space. The specific appearance is the overlap of multiple facial key points. The face pasting algorithm specifically includes face analysis, tooth restoration, Gaussian fusion and edge feathering. On the other hand, focusing on the area with large pixel gradient changes at the edge, multi-scale (resolution) sub-pixel feature fusion is performed in the Gaussian feature pyramid, and then each resolution layer is reconstructed and resampled to the original resolution.
[0069] It should be noted that after completing the processing of the face pasting algorithm, the generated results need to be compared, specifically for image quality judgment and light and shadow judgment, to ensure that the generated composite video meets user needs.
[0070] Furthermore, in some embodiments of the present application, before separating the original video into the driving video and the driving audio, the method further includes:
[0071] The original video is subjected to scene screening and frame rate conversion processing.
[0072] Specifically, before separating the original video into the driving video and the driving audio, the original video is also subjected to scene screening and frame rate conversion processing. The scene screening includes screening of face size, number of faces, number of voices and video resolution, so as to ensure that the user's original video can be translated, and the frame rate conversion processing is to convert the original video into a specific frame rate, so as to improve the picture quality of the original video, thereby facilitating subsequent translation operations.
[0073] Figure 3 FIG. 2 is a structural block diagram of a video translation device 200 according to an embodiment of the present application. Figure 3 As shown, the video translation device 200 of the embodiment of the present application includes: a video separation module 210, a light and shadow processing module 220, an audio translation module 230 and a video synthesis module 240, wherein:
[0074] A video separation module 210, used to separate the original video into a driving video and a driving audio;
[0075] A light and shadow processing module 220, used for performing light and shadow processing on the driving video;
[0076] An audio translation module 230, used for translating the driving audio; and
[0077] The video synthesis module 240 is used to perform synthesis reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-driven model to form a synthesized video.
[0078] The video translation device provided in the embodiment of the present application separates the original video into a driving video and a driving audio, performs light and shadow processing on the driving video, performs translation processing on the driving audio, and performs synthetic reasoning on the driving video after the light and shadow processing and the driving audio after the translation processing through a lip-driven model to form a synthetic video, thereby ensuring that multi-language, high-fidelity, and highly robust lip-driven output can be generated, and video translation can be achieved efficiently, accurately, and naturally.
[0079] It should be noted that the specific implementation of the video translation device of the embodiment of the present application is similar to the specific implementation of the video translation method of the embodiment of the present application. Please refer to the description of the method part for details, and no further details will be given here.
[0080] Figure 4 Schematic diagram of the structure of an electronic device 300 according to an embodiment of the present application.
[0081] like Figure 4 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage part 302 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0082] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.
[0083] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, the above-mentioned functions defined in the electronic device of the present application are executed.
[0084] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0085] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction-executing electronic device, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction-executing electronic device, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0086] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based electronic device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0087] The units or modules involved in the embodiments of the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor, and the processor is used to implement the video translation method when executing the program:
[0088] Separate the original video into driving video and driving audio;
[0089] Performing light and shadow processing on the driving video;
[0090] performing translation processing on the driving audio; and
[0091] The driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred through the lip-sync driving model to form a synthetic video.
[0092] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the video translation method described in the present application:
[0093] Separate the original video into driving video and driving audio;
[0094] Performing light and shadow processing on the driving video;
[0095] performing translation processing on the driving audio; and
[0096] The driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred through the lip-sync driving model to form a synthetic video.
[0097] As another aspect, the present application further provides a computer program product, which may be included in the electronic device described in the above embodiment; or may exist independently without being installed in the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the video translation method described in the present application:
[0098] Separate the original video into driving video and driving audio;
[0099] Performing light and shadow processing on the driving video;
[0100] performing translation processing on the driving audio; and
[0101] The driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred through the lip-sync driving model to form a synthetic video.
[0102] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structural changes made by using the contents of the present application specification and drawings under the application concept of the present application, or directly / indirectly used in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A video translation method, characterized in that: include: Separate the original video into driving video and driving audio; Performing light and shadow processing on the driving video; Performing translation processing on the driving audio; and The driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred through the lip-sync driving model to form a synthetic video.
2. The video translation method according to claim 1, characterized in that: Before performing light and shadow processing on the driving video, the method further includes: The human face in the driving video is recognized and processed.
3. The video translation method according to claim 2, characterized in that: The identifying and processing of faces in the driving video includes: Performing facial key point extraction, facial segmentation and facial detection processing on the face in the driving video; and The human face in the driving video is processed by key point balance, face alignment and face pear-shaped region generation.
4. The video translation method according to claim 1, characterized in that: After recognizing and processing the face in the driving video, and before performing light and shadow processing on the driving video, the method further includes: A multi-dimensional reference frame is constructed for the human face in the driving video.
5. The video translation method according to claim 1, characterized in that: After the driving video after light and shadow processing and the driving audio after translation processing are synthesized and inferred by the lip-driven model to form a synthesized video, the method further includes: The composite video is processed by a color harmonization algorithm.
6. The video translation method according to claim 5, characterized in that: After the composite video is processed by the color harmony algorithm, the method further includes: The synthesized video is processed by a face pasting algorithm.
7. The video translation method according to claim 1, characterized in that: Before separating the original video into the driving video and the driving audio, the method further includes: The original video is subjected to scene screening and frame rate conversion processing.
8. A video translation device, characterized in that: include: A video separation module, used for separating the original video into a driving video and a driving audio; A light and shadow processing module, used for performing light and shadow processing on the driving video; An audio translation module, used for translating the driving audio; and The video synthesis module is used to perform synthesis reasoning on the driving video after light and shadow processing and the driving audio after translation processing through a lip-driven model to form a synthetic video.
9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the video translation method according to any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the video translation method according to any one of claims 1-7.
Citation Information
Cited By
Multi-modal time sequence alignment AI video translation method and system
CN120416568A
