Video translation method and related device

Abstract

The invention discloses a video translation method and a related device, and relates to the technical field of video processing, and the method comprises the steps: obtaining a to-be-translated original video, carrying out the voice synthesis through the voiceprint features and emotion labels of a speaker in the original video, and obtaining the voice data of a target language, the method comprises the following steps: acquiring a target language, replacing original voice data in an original video with the voice data of the target language to obtain an initial translated video, performing speaker mouth area image synthesis by using the voice data of the target language to obtain a synthesized image of the speaker mouth area, and finally fusing the synthesized image of the speaker mouth area with the initial translated video to obtain a speaker mouth area image. And obtaining a final translated video. According to the scheme, the original sound of the video can be restored as much as possible, the speaking state of a character in the original video can be fitted as much as possible, and natural transition of a lip picture before and after the lip picture is lost or appears in the final translated video can be ensured, so that the auditory sense and visual sense of a user can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video translation method and related devices. Background Art

[0002] Nowadays, people live in an era of information explosion and can easily obtain videos from all over the world. However, due to different regions, people may not be able to quickly obtain effective information from videos without professional translation. In order to improve people's viewing efficiency and quality of videos, video translation came into being. Video translation refers to translating a video in one language into another language, for example, translating an English video into a Chinese video, translating a dialect video into a Mandarin video, etc.

[0003] With existing video translation technology, the pronunciation in the translated video is inconsistent with that in the original video in terms of tone, timbre, and emotional expression, resulting in a poor auditory experience for users. In addition, in the translated video, the character's lip shape does not match the newly generated pronunciation, and the overall picture is not coordinated and natural enough, resulting in a poor visual experience for users.

[0004] Therefore, how to provide a video translation method to enhance the user's auditory and visual experience has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] In view of the above problems, this application provides a video translation method and related devices to achieve the purpose of improving the user's auditory and visual experience. The specific scheme is as follows:

[0006] The first aspect of the present application provides a video translation method, comprising:

[0007] Get the original video to be translated;

[0008] Performing speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data in the target language;

[0009] Replacing the original speech data in the original video with the speech data in the target language to obtain an initial translated video;

[0010] Using the speech data of the target language to synthesize an image of the speaker's mouth area, to obtain a synthesized image of the speaker's mouth area;

[0011] The synthesized image of the speaker's mouth area is fused with the initial translation video to obtain a final translation video.

[0012] In a possible implementation, the using the voiceprint features and emotion tags of the speaker in the original video to perform speech synthesis to obtain speech data of the target language includes:

[0013] Acquire original voice data in the original video;

[0014] Using the original speech data, determining each clause, each clause comprising original language clause audio data and original language clause text;

[0015] For each clause, the original language clause audio data is speech translated to obtain the target language clause text, and the original language clause audio data is voiceprint detected to obtain the clause voiceprint feature; emotion recognition is performed on the original language clause text to obtain the clause emotion label; based on the clause voiceprint feature and the clause emotion label, the target language clause text is speech synthesized to obtain the target language clause audio data; each of the target language clause audio data and each of the target language clause text is combined to obtain the speech data of the target language.

[0016] In a possible implementation, the using the original speech data to determine each clause includes:

[0017] Performing speech recognition on the original speech data to obtain original language text and outputting phoneme boundaries;

[0018] Determine clause information based on the phoneme boundary, wherein the clause information includes the pronunciation start time of the first word of each clause and the pronunciation end time of the last word;

[0019] According to the clause information, the original speech data and the original language text are segmented to obtain individual clauses; each clause includes original language clause audio data and original language clause text.

[0020] In a possible implementation, replacing original speech data in the original video with speech data in the target language to obtain an initial translated video includes:

[0021] Adjusting each of the target language clause audio data to obtain adjusted target language clause audio data; each adjusted target language clause audio data has the same duration as the corresponding original language clause audio data;

[0022] For each clause, the adjusted target language clause audio data and the target language clause text are used to replace the original language clause audio data and the original language clause text to obtain the initial translation video.

[0023] In a possible implementation, synthesizing an image of a speaker's mouth region by using the speech data of the target language to obtain a synthesized image of the speaker's mouth region includes:

[0024] Acquire original image data in the original video;

[0025] Starting from the starting frame of the original image data, face detection and face key point detection are performed frame by frame to obtain face data of each frame, wherein the face data refers to the position of the rectangular frame of the face of the person, the face key points and the confidence of the face key points;

[0026] For each frame, determine whether the frame is a valid frame based on the confidence of the facial key points of the frame;

[0027] An image synthesis is performed on the mouth region of the speaker in the valid frame to obtain a synthesized image of the mouth region of the speaker.

[0028] In a possible implementation, fusing the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video includes:

[0029] Based on a preset fusion strategy, the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video to obtain a final translation video.

[0030] A second aspect of the present application provides a video translation device, comprising:

[0031] An acquisition unit, used for acquiring an original video to be translated;

[0032] A speech synthesis unit, used to perform speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data in a target language;

[0033] A voice data replacement unit, used to replace the original voice data in the original video with the voice data in the target language to obtain an initial translated video;

[0034] An image synthesis unit, used to synthesize an image of a speaker's mouth region using the speech data of the target language to obtain a synthesized image of the speaker's mouth region;

[0035] The image fusion unit is used to fuse the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video.

[0036] In a possible implementation, the speech synthesis unit includes:

[0037] An original voice data acquisition unit, used to acquire the original voice data in the original video;

[0038] A clause determination unit, used to determine each clause using the original speech data, each clause comprising original language clause audio data and original language clause text;

[0039] The clause processing unit is used to perform speech translation on the original language clause audio data for each clause to obtain a target language clause text, and perform voiceprint detection on the original language clause audio data to obtain a clause voiceprint feature; perform emotion recognition on the original language clause text to obtain a clause emotion label; perform speech synthesis on the target language clause text based on the clause voiceprint feature and the clause emotion label to obtain target language clause audio data; and combine each of the target language clause audio data and each of the target language clause texts to obtain the speech data of the target language.

[0040] In a possible implementation, the clause determination unit is specifically configured to:

[0041] Performing speech recognition on the original speech data to obtain original language text and outputting phoneme boundaries;

[0042] Determine clause information based on the phoneme boundary, wherein the clause information includes the pronunciation start time of the first word of each clause and the pronunciation end time of the last word;

[0043] According to the clause information, the original speech data and the original language text are segmented to obtain individual clauses; each clause includes original language clause audio data and original language clause text.

[0044] In a possible implementation, the voice data replacement unit is specifically used to:

[0045] Adjusting each of the target language clause audio data to obtain adjusted target language clause audio data; each adjusted target language clause audio data has the same duration as the corresponding original language clause audio data;

[0046] For each clause, the adjusted target language clause audio data and the target language clause text are used to replace the original language clause audio data and the original language clause text to obtain the initial translation video.

[0047] In a possible implementation, the image synthesis unit is specifically used to:

[0048] Acquire original image data in the original video;

[0049] Starting from the starting frame of the original image data, face detection and face key point detection are performed frame by frame to obtain face data of each frame, wherein the face data refers to the position of the rectangular frame of the face of the person, the face key points and the confidence of the face key points;

[0050] For each frame, determine whether the frame is a valid frame based on the confidence of the facial key points of the frame;

[0051] An image synthesis is performed on the mouth region of the speaker in the valid frame to obtain a synthesized image of the mouth region of the speaker.

[0052] In a possible implementation, the image fusion unit is specifically used to:

[0053] Based on a preset fusion strategy, the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video to obtain a final translation video.

[0054] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the video translation method of the first aspect or any implementation of the first aspect.

[0055] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0056] The memory is used to store computer programs;

[0057] The processor is used to execute the computer program so that the electronic device can implement the video translation method of the first aspect or any implementation manner of the first aspect.

[0058] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the video translation method of the first aspect or any implementation of the first aspect.

[0059] By means of the above technical scheme, the video translation method and related device provided by the present application, after obtaining the original video to be translated, first use the voiceprint features and emotion tags of the speaker in the original video to perform speech synthesis to obtain the speech data of the target language, then replace the original speech data in the original video with the speech data of the target language to obtain the initial translation video, then use the speech data of the target language to synthesize the image of the speaker's mouth area to obtain the synthesized image of the speaker's mouth area, and finally merge the synthesized image of the speaker's mouth area with the initial translation video to obtain the final translation video. This scheme can restore the original sound of the video as much as possible, and try to fit the speaking state of the characters in the original video as much as possible, and can ensure that in the final translation video, when the lip picture is lost or the natural transition between the front and back pictures appears, it can improve the user's auditory and visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0061] Figure 1 A flowchart of a video translation method provided in an embodiment of the present application;

[0062] Figure 2 A schematic diagram of a process for synthesizing speech using the voiceprint features and emotion tags of a speaker in an original video to obtain speech data in a target language provided in an embodiment of the present application;

[0063] Figure 3 A schematic diagram of a process for replacing original voice data in an original video with voice data in a target language to obtain an initial translated video provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of a process for synthesizing a speaker's mouth area image using speech data of a target language to obtain a synthesized image of the speaker's mouth area provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a video translation device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0068] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0069] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0070] Nowadays, people live in an era of information explosion and can easily obtain videos from all over the world. However, due to different regions, people may not be able to quickly obtain effective information from videos without professional translation. In order to improve people's viewing efficiency and quality of videos, video translation came into being. Video translation refers to translating a video in one language into another language, for example, translating an English video into a Chinese video, translating a dialect video into a Mandarin video, etc.

[0071] Existing video translation technologies are divided into manual dubbing and algorithm implementation. Among them, the manual dubbing method is to directly ask voice actors to dub each character, and then mix and replace the audio track in the original video. This method involves many people and has high labor costs. Algorithm implementation methods are basically based on deep learning technology, which involves key technologies such as speech recognition, speech translation, semantic understanding and lip generation, such as ASR (Automatic Speech Recognition), Chinese NMT, LLM (Large Language Model), wav2lip, etc. These technologies are connected in series to translate a video in one language into a subtitled video in another language. This method requires the participation of professionals with algorithm background to complete the production.

[0072] Moreover, regardless of whether it is a manual dubbing method or an algorithm implementation method, the pronunciation in the translated video is inconsistent with the pronunciation in the original video in terms of tone, timbre, emotional expression, etc., resulting in the translated video voice being abrupt and the user's auditory experience being poor. In addition, in the translated video, the character's lip shape cannot match the newly generated pronunciation, and the overall picture is not coordinated and natural enough, resulting in a poor user visual experience.

[0073] Therefore, how to provide a video translation method to enhance the user's auditory and visual experience has become a technical problem that needs to be solved urgently by those skilled in the art.

[0074] In order to solve the above problems, the embodiment of the present application provides a video translation method. The video translation method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0075] Reference Figure 1 , Figure 1 A flowchart of a video translation method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, a video translation method provided in an embodiment of the present application may include the following steps, which are described in detail below.

[0076] S101: Obtaining an original video to be translated;

[0077] In this application, the format of the original video to be translated can be any format, such as mp4, mov, etc., and this application does not impose any limitation on this. The speech in the original video to be translated is the speech of the first language.

[0078] S102: Performing speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data of the target language;

[0079] In this application, the voiceprint feature can express the timbre and speaking speed of the speaker in the original video. The emotion label can express the speaking state of the speaker in the original video, such as happy, peaceful, angry, etc. The voiceprint feature and emotion label of the speaker in the original video are used for speech synthesis to obtain the speech data of the target language, which can restore the original sound of the video as much as possible and fit the speaking state of the characters in the original video as much as possible. In a possible implementation, the speech data of the target language includes audio data of the target language and text of the target language.

[0080] S103: replacing the original voice data in the original video with the voice data in the target language to obtain an initial translated video;

[0081] In this application, the audio track and subtitles in the original video can be replaced with the speech data in the target language to obtain an initial translated video. The initial translated video is a complete video with subtitles after translation. At this time, the lip shape and voice of the characters in the video screen are inconsistent and need further processing.

[0082] S104: synthesizing an image of the speaker's mouth region using the speech data of the target language to obtain a synthesized image of the speaker's mouth region;

[0083] In this application, the speaker is the person who appears in the video and corresponds to the original voice data. Considering that the speaker does not appear in every frame of the video, in this application, when synthesizing the speaker's mouth area image using the voice data of the target language, only the frame with the speaker is processed. The specific implementation method will be described in detail through the following embodiments and will not be described here.

[0084] S105: Fusing the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video.

[0085] In one possible implementation, the composite image of the speaker's mouth area can be used to replace the original image of the speaker's mouth area in the initial translation video, and re-encoded to obtain the final translation video. However, the direct replacement method cannot guarantee that the lip image is lost or there is a natural transition between the front and back images in the final translation video. Therefore, in another possible implementation, the composite image of the speaker's mouth area can be fused with the initial translation video to obtain the final translation video. The specific implementation method will be described in detail through the following embodiments and will not be described in detail here.

[0086] The video translation method provided in this embodiment obtains the original video to be translated, first uses the voiceprint features and emotion tags of the speaker in the original video to perform speech synthesis to obtain the speech data of the target language, then replaces the original speech data in the original video with the speech data of the target language to obtain the initial translation video, then uses the speech data of the target language to synthesize the speaker's mouth area image to obtain the synthesized image of the speaker's mouth area, and finally merges the synthesized image of the speaker's mouth area with the initial translation video to obtain the final translation video. This solution can restore the original sound of the video as much as possible, and try to fit the speaking state of the characters in the original video as much as possible, and can ensure that in the final translation video, when the lip image is lost or the front and back images are naturally transitioned, it can improve the user's auditory and visual experience.

[0087] In another embodiment of the present application, the specific implementation method of using the voiceprint features and emotion tags of the speaker in the original video to perform speech synthesis to obtain the speech data of the target language is described, referring to Figure 2 , Figure 2 A flow chart of a method for synthesizing speech using the voiceprint features and emotion tags of a speaker in an original video to obtain speech data of a target language is provided in an embodiment of the present application, such as Figure 2 As shown, the specific steps include:

[0088] S201: Acquire original voice data in the original video;

[0089] In the present application, the original voice data in the original video can be obtained by using a video decoding tool.

[0090] S202: using the original speech data, determining each clause, each clause comprising original language clause audio data and original language clause text;

[0091] In a possible implementation, the using the original speech data to determine each clause includes:

[0092] S2021: Performing speech recognition on the original speech data to obtain original language text and outputting phoneme boundaries;

[0093] In this application, the speech recognition is a prior art and will not be described in detail in this application. The phoneme boundary refers to the pronunciation time period of each phoneme in the original speech data.

[0094] S2022: Determine clause information based on the phoneme boundary, wherein the clause information includes a pronunciation start time of the first word of each clause and a pronunciation end time of the last word;

[0095] Based on the phoneme boundary, the pronunciation time period of each word and the pronunciation time interval of adjacent words can be determined. In a possible implementation, assuming that the pronunciation time interval of adjacent words is T1, it is set that when T1 is greater than 300ms, it means that the previous and next speech are two complete sentences, and then the pronunciation end time te of the last word of the previous sentence and the pronunciation start time ts of the first word of the next sentence are recorded, and the recorded times are the clause information.

[0096] S2023: According to the clause information, the original speech data and the original language text are segmented to obtain individual clauses; each clause includes original language clause audio data and original language clause text.

[0097] In the present application, the speech data is cut based on the clause information to obtain all the original language clause audio data. For ease of understanding, the original language clause audio data in the present application can be represented as A1, A2...An. The operation of cutting the original language text based on the clause information to obtain each clause text is the same, and will not be repeated in the present application. For ease of understanding, the clause texts in the present application can be represented as Txt1, Txt2...Txtn.

[0098] S203: For each clause, perform speech translation on the original language clause audio data to obtain a target language clause text, and perform voiceprint detection on the original language clause audio data to obtain a clause voiceprint feature; perform emotion recognition on the original language clause text to obtain a clause emotion label; perform speech synthesis on the target language clause text based on the clause voiceprint feature and the clause emotion label to obtain target language clause audio data; and combine each of the target language clause audio data and each of the target language clause texts to obtain the speech data of the target language.

[0099] In this application, speech translation, voiceprint detection, and emotion recognition are all existing technologies and will not be introduced in detail in this application. In this application, the clause voiceprint feature can control the speaker's timbre and speaking speed, and can restore the original sound of the video as much as possible; the emotion label can control the speaker's speaking emotion, and can achieve the speaking state of the character in the original video as close as possible.

[0100] In another embodiment of the present application, the specific implementation method of replacing the original voice data in the original video with the voice data of the target language to obtain the initial translation video is described, referring to Figure 3 , Figure 3 A flowchart of replacing the original voice data in the original video with the voice data of the target language to obtain the initial translation video is provided in an embodiment of the present application, such as Figure 3 As shown, the specific steps include:

[0101] S301: adjusting each of the target language clause audio data to obtain adjusted target language clause audio data; each adjusted target language clause audio data has the same duration as the corresponding original language clause audio data;

[0102] In the present application, each target language clause audio data can be inserted according to the time position of the original language clause audio data in the original speech data. When the target language clause audio data duration is greater than the corresponding original language clause audio data, the subsequent target language clause audio data insertion position will be delayed. Therefore, when the target language clause audio data duration is less than the corresponding original language clause audio data, the subsequent target language clause audio data insertion position will be advanced. In the present application, each target language clause audio data can be adjusted before being inserted. Among them, the adjustment of each target language clause audio data includes speech speed adjustment or silence adjustment. When the target language clause audio data duration is greater than the corresponding original language clause audio data, assuming that the default speech speed of the target language clause audio data is S, the adjusted speech speed is (S×target language clause audio data duration) / corresponding original language clause audio data duration; when the target language clause audio data duration is less than the corresponding original language clause audio data, the insufficient part is replaced by silence.

[0103] S302: For each clause, the adjusted target language clause audio data and the target language clause text are used to replace the original language clause audio data and the original language clause text to obtain the initial translation video.

[0104] In the present application, replacing the original language clause audio data and the original language clause text with the adjusted target language clause audio data and the target language clause text means inserting the adjusted target language clause audio data and the target language clause text according to the time position of the original language clause audio data in the original speech data.

[0105] In another embodiment of the present application, the specific implementation method of synthesizing the speaker's mouth area image using the speech data of the target language to obtain the synthesized image of the speaker's mouth area is introduced, referring to Figure 4 , Figure 4 A schematic diagram of a process for synthesizing a speaker's mouth area image using speech data of a target language to obtain a synthesized image of the speaker's mouth area is provided in an embodiment of the present application, such as Figure 4 As shown, the specific steps include:

[0106] S401: Acquire original image data in the original video;

[0107] In the present application, a video decoding tool may be used to obtain the original image data in the original video.

[0108] S402: Starting from the starting frame of the original image data, face detection and face key point detection are performed frame by frame to obtain face data of each frame, wherein the face data refers to the position of the face rectangle frame, face key points and the confidence of the face key points;

[0109] S403: for each frame, determining whether the frame is a valid frame based on the confidence of the facial key points of the frame;

[0110] In one possible implementation, if the confidence of the facial key points meets the preset conditions, the frame is determined to be a valid frame. In one possible implementation, a facial key point confidence threshold (such as 0.4) can be preset. If the facial key point confidence of a frame is higher than the threshold, it is considered to meet the preset conditions. In this application, face detection and facial key point detection are prior arts and will not be described in detail in this application.

[0111] S404: Perform image synthesis on the mouth region of the speaker in the valid frame to obtain a synthesized image of the mouth region of the speaker.

[0112] For each valid frame, if there are more than one characters in the current valid frame, only one of the characters is selected as the speaker. The specific selection method can be to perform lip alignment detection on the original language clause audio data of the clause where the valid frame is located and all characters, output the deviation value of each character from the original language clause audio data, and select the character corresponding to the minimum deviation value as the speaker. Lip alignment detection is a prior art and will not be introduced in detail in this application.

[0113] In the present application, the mouth area of ​​the speaker is determined based on the lip key points of the speaker's face key points. In a possible implementation, the face key points are based on the upper left corner of the character's face rectangular frame as the origin coordinates, including 68 facial contour points, and the 49th to 68th facial contour points, a total of 20 facial contour points, are lip key points. In the present application, the circumscribed rectangular frame rt (x, y, width, height) of the 49th to 68th facial contour point coordinates can be selected as the marking frame, and the width and height of the marking frame are respectively expanded by 1.5 times, that is, (x - width / 4, y - height / 4, width * 3 / 2, height * 3 / 2) The area is taken as the mouth area, where x is the x coordinate of the upper left corner of the circumscribed rectangular box of the 20 facial contour point coordinates, y is the y coordinate of the upper left corner of the circumscribed rectangular box of the 20 facial contour point coordinates, width is the width of the circumscribed rectangular box of the 20 facial contour point coordinates, and height is the height of the circumscribed rectangular box of the 20 facial contour point coordinates.

[0114] In the present application, a lip image synthesis algorithm can be used to perform lip image synthesis on the mouth area of ​​the speaker in the valid frame based on the adjusted target language clause audio data to obtain a synthesized image of the mouth area of ​​the speaker in the valid frame, wherein the lip image synthesis algorithm is a prior art and will not be described in detail in the present application.

[0115] In a possible implementation, fusing the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video includes:

[0116] Based on a preset fusion strategy, the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video to obtain a final translation video.

[0117] In the present application, a fusion strategy may be preset, and when the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video, the preset fusion strategy may be used to obtain a fused image of each frame.

[0118] In one feasible way, assuming that the synthesized image is syn, the corresponding original image is src, and the number of consecutive valid frames starting from the start frame is w, then the calculation method for each pixel of the fused image dst of each frame is as follows:

[0119]

[0120] Among them, 8 represents the window length of the smoothing frame, and w is calculated by adding one if the current frame is valid (that is, the confidence of the facial key points meets the requirements), and subtracting one if it does not meet the requirements, and the range of w is fixed between 0 and 8.

[0121] Based on the above fusion strategy, it can be ensured that in the final translated video, there is a natural transition between the previous and next images when the lip image is lost or appears.

[0122] A video translation method provided by an embodiment of the present application is introduced above, and a device for executing the above video translation method will be introduced below.

[0123] See also Figure 5 , Figure 5 The structure diagram of a video translation device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the video translation device comprises:

[0124] An acquisition unit 11 is used to acquire an original video to be translated;

[0125] A speech synthesis unit 12, configured to perform speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data in a target language;

[0126] A voice data replacement unit 13, used to replace the original voice data in the original video with the voice data in the target language to obtain an initial translated video;

[0127] An image synthesis unit 14 is used to synthesize an image of the speaker's mouth region using the speech data of the target language to obtain a synthesized image of the speaker's mouth region;

[0128] The image fusion unit 15 is used to fuse the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video.

[0129] In a possible implementation, the speech synthesis unit includes:

[0130] An original voice data acquisition unit, used to acquire the original voice data in the original video;

[0131] A clause determination unit, used to determine each clause using the original speech data, each clause comprising original language clause audio data and original language clause text;

[0132] The clause processing unit is used to perform speech translation on the original language clause audio data for each clause to obtain a target language clause text, and perform voiceprint detection on the original language clause audio data to obtain a clause voiceprint feature; perform emotion recognition on the original language clause text to obtain a clause emotion label; perform speech synthesis on the target language clause text based on the clause voiceprint feature and the clause emotion label to obtain target language clause audio data; and combine each of the target language clause audio data and each of the target language clause texts to obtain the speech data of the target language.

[0133] In a possible implementation, the clause determination unit is specifically configured to:

[0134] Performing speech recognition on the original speech data to obtain original language text and outputting phoneme boundaries;

[0135] Determine clause information based on the phoneme boundary, wherein the clause information includes the pronunciation start time of the first word of each clause and the pronunciation end time of the last word;

[0136] According to the clause information, the original speech data and the original language text are segmented to obtain individual clauses; each clause includes original language clause audio data and original language clause text.

[0137] In a possible implementation, the voice data replacement unit is specifically used to:

[0138] Adjusting each of the target language clause audio data to obtain adjusted target language clause audio data; each adjusted target language clause audio data has the same duration as the corresponding original language clause audio data;

[0139] For each clause, the adjusted target language clause audio data and the target language clause text are used to replace the original language clause audio data and the original language clause text to obtain the initial translation video.

[0140] In a possible implementation, the image synthesis unit is specifically used to:

[0141] Acquire original image data in the original video;

[0142] Starting from the starting frame of the original image data, face detection and face key point detection are performed frame by frame to obtain face data of each frame, wherein the face data refers to the position of the rectangular frame of the face of the person, the face key points and the confidence of the face key points;

[0143] For each frame, determine whether the frame is a valid frame based on the confidence of the facial key points of the frame;

[0144] An image synthesis is performed on the mouth region of the speaker in the valid frame to obtain a synthesized image of the mouth region of the speaker.

[0145] In a possible implementation, the image fusion unit is specifically used to:

[0146] Based on a preset fusion strategy, the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video to obtain a final translation video.

[0147] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0148] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0149] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0150] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the video translation methods provided in the embodiments of the present application.

[0151] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any video translation method provided in the embodiment of the present application.

[0152] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0153] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0155] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A video translation method, characterized in that: include: Get the original video to be translated; Performing speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data in the target language; Replacing the original speech data in the original video with the speech data in the target language to obtain an initial translated video; Using the speech data of the target language to synthesize an image of the speaker's mouth area, to obtain a synthesized image of the speaker's mouth area; The synthesized image of the speaker's mouth area is fused with the initial translation video to obtain a final translation video.

2. The method according to claim 1, characterized in that The method of synthesizing speech using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data of the target language includes: Acquire original voice data in the original video; Using the original speech data, determining each clause, each clause comprising original language clause audio data and original language clause text; For each clause, the original language clause audio data is speech translated to obtain the target language clause text, and the original language clause audio data is voiceprint detected to obtain the clause voiceprint feature; emotion recognition is performed on the original language clause text to obtain the clause emotion label; based on the clause voiceprint feature and the clause emotion label, the target language clause text is speech synthesized to obtain the target language clause audio data; each of the target language clause audio data and each of the target language clause text is combined to obtain the speech data of the target language.

3. The method according to claim 2, characterized in that The method of using the original speech data to determine each clause includes: Performing speech recognition on the original speech data to obtain original language text and outputting phoneme boundaries; Determine clause information based on the phoneme boundary, wherein the clause information includes the pronunciation start time of the first word of each clause and the pronunciation end time of the last word; According to the clause information, the original speech data and the original language text are segmented to obtain individual clauses; each clause includes original language clause audio data and original language clause text.

4. The method according to claim 2 or 3, characterized in that: The step of replacing the original voice data in the original video with the voice data in the target language to obtain an initial translated video includes: Adjusting each of the target language clause audio data to obtain adjusted target language clause audio data; each adjusted target language clause audio data has the same duration as the corresponding original language clause audio data; For each clause, the adjusted target language clause audio data and the target language clause text are used to replace the original language clause audio data and the original language clause text to obtain the initial translation video.

5. The method according to claim 3, characterized in that: The method of synthesizing a speaker's mouth area image by using the speech data of the target language to obtain a synthesized image of the speaker's mouth area includes: Acquire original image data in the original video; Starting from the starting frame of the original image data, face detection and face key point detection are performed frame by frame to obtain face data of each frame, wherein the face data refers to the position of the rectangular frame of the face of the person, the face key points and the confidence of the face key points; For each frame, determine whether the frame is a valid frame based on the confidence of the facial key points of the frame; An image synthesis is performed on the mouth region of the speaker in the valid frame to obtain a synthesized image of the mouth region of the speaker.

6. The method according to claim 3, characterized in that The step of fusing the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video includes: Based on a preset fusion strategy, the composite image of the speaker's mouth area in each valid frame is fused into the corresponding frame of the initial translation video to obtain a final translation video.

7. A video translation device, characterized in that: include: An acquisition unit, used for acquiring an original video to be translated; A speech synthesis unit, used to perform speech synthesis using the voiceprint features and emotion tags of the speaker in the original video to obtain speech data in a target language; A voice data replacement unit, used to replace the original voice data in the original video with the voice data in the target language to obtain an initial translated video; An image synthesis unit, used to synthesize an image of a speaker's mouth region using the speech data of the target language to obtain a synthesized image of the speaker's mouth region; The image fusion unit is used to fuse the synthesized image of the speaker's mouth area with the initial translation video to obtain a final translation video.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the video translation method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the video translation method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the video translation method as described in any one of claims 1 to 6.

Images