Translation method, device, electronic device and storage medium

Abstract

The present invention provides a translation method, device, electronic device and storage medium, wherein the method comprises: performing audio and video separation on a source language video to obtain a source language audio and source language video frame sequence; performing key frame extraction on the source language video frame sequence to obtain a source language key frame; translating the source language key frame into a target language to obtain a target language key frame; synthesizing a target language video based on the target language key frame and the target language subtitles obtained by translating the source language audio. The translation method, device, electronic device and storage medium provided by the present invention combine the target language key frame with the target language subtitles obtained by translating the source language audio, so as to fully and comprehensively reflect the content of the video translation, and perform key frame extraction on the source language video frame sequence, thereby improving the efficiency of translation while ensuring the integrity of the translated video information.

Description

Technical Field

[0001] The present invention relates to the field of translation technology, and in particular to a translation method, device, electronic equipment and storage medium. Background Art

[0002] With the development of technology, the demand for translation is increasing, especially the demand for video translation. Video translation is the technology of converting a video in one language into another language, usually by adding subtitles to the original video.

[0003] However, adding subtitles directly to the original video for translation will lose a lot of video information, and the translated information is incomplete, causing language barriers in information transmission. Therefore, how to translate videos completely and comprehensively is still a problem that needs to be solved in the field of video translation technology. Summary of the invention

[0004] The present invention provides a translation method, device, electronic equipment and storage medium, which are used to solve the problem of incomplete video translation in the prior art.

[0005] The present invention provides a translation method, comprising:

[0006] Separating the source language video from audio and video to obtain source language audio and source language video frame sequences;

[0007] Extracting key frames from the source language video frame sequence to obtain source language key frames;

[0008] Translating the source language key frame into a target language to obtain a target language key frame;

[0009] A target language video is synthesized based on the target language key frames and the target language subtitles obtained by translating the source language audio.

[0010] According to a translation method provided by the present invention, extracting key frames from the source language video frame sequence to obtain source language key frames includes:

[0011] Determining a candidate frame sequence based on the source language video frame sequence;

[0012] Based on the similarity between the candidate frames in the candidate frame sequence, a source language key frame is extracted from the candidate frame sequence.

[0013] According to a translation method provided by the present invention, determining a candidate frame sequence based on the source language video frame sequence includes:

[0014] The candidate frame sequence is extracted from the source language video frame sequence based on the encoding information of each video frame in the source language video frame sequence.

[0015] According to a translation method provided by the present invention, extracting source language key frames from the candidate frame sequence based on the similarity between the candidate frames in the candidate frame sequence includes:

[0016] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is greater than a preset threshold, deleting the next candidate frame and updating the next next candidate frame as the next candidate frame;

[0017] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is less than or equal to a preset threshold, updating the next candidate frame to the current candidate frame until the current candidate frame is the last candidate frame;

[0018] Each candidate frame in the candidate frame sequence is determined as a source language key frame.

[0019] According to a translation method provided by the present invention, translating the source language key frame into a target language to obtain the target language key frame includes:

[0020] Performing text recognition on the source language key frame to obtain the source language text included in the source language key frame and position information of the source language text;

[0021] Translating the source language text into a target language to obtain a target language text;

[0022] Based on the position information of the source language text, the target language text is synthesized on the source language key frame to obtain the target language key frame.

[0023] According to a translation method provided by the present invention, synthesizing a target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio includes:

[0024] Synthesizing an initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame; the display period is determined based on the playback time of the corresponding source language key frame and the next source language key frame in the source language video;

[0025] The target language subtitles are superimposed on the initial video to obtain the target language video.

[0026] According to a translation method provided by the present invention, synthesizing an initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame includes:

[0027] splicing the target language key frame and the source language key frame corresponding to the target language key frame to obtain a multi-language key frame;

[0028] An initial video is synthesized based on the multilingual key frames and the display time periods of the multilingual key frames corresponding to the source language key frames.

[0029] The present invention also provides a translation device, comprising:

[0030] An audio and video separation unit, used to separate the audio and video of the source language video to obtain a source language audio and source language video frame sequence;

[0031] A key frame determination unit, configured to extract key frames from the source language video frame sequence to obtain source language key frames;

[0032] A key frame translation unit, used for translating the source language key frame into a target language to obtain a target language key frame;

[0033] The video synthesis unit is used to synthesize the target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned translation methods is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the translation method described above is implemented.

[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned translation methods.

[0037] The present invention provides a translation method, device, electronic device and storage medium, which obtains target language key frames by translating key frames in a source language video frame sequence, and combines the target language key frames with target language subtitles and / or target language audio obtained by translating source language audio, so as to fully and comprehensively reflect the content of the video translation. In addition, in this process, key frames are extracted from the source language video frame sequence, which reduces the computing resource requirements of the translation while ensuring the integrity of the translated video information, and improves the translation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is one of the flowcharts of the translation method provided by the present invention;

[0040] Figure 2 is a flow chart of step 120 in the translation method provided by the present invention;

[0041] Figure 3 is a flow chart of step 130 in the translation method provided by the present invention;

[0042] Figure 4 is a flow chart of step 140 in the translation method provided by the present invention;

[0043] Figure 5 It is a schematic diagram of the process of synthesizing the initial video provided by the present invention;

[0044] Figure 6 This is the second flow chart of the translation method provided by the present invention;

[0045] Figure 7 is a schematic diagram of the structure of the translation device provided by the present invention;

[0046] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The current video translation usually translates the voice carried in the video and displays it in the form of subtitles. The above method omits a lot of information in the original video, and the translated information is incomplete. To solve this problem, an embodiment of the present invention provides a translation method. Figure 1 It is one of the flowcharts of the translation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0049] Step 110, separating the audio and video of the source language video to obtain a source language audio and source language video frame sequence.

[0050] Specifically, the source language video is a video in the source language. Here, the source language is the language that needs to be translated. Correspondingly, the target language is the language used after the translation is completed. For example, if English is translated into Mandarin, English is the source language and Mandarin is the target language. The source language video can be a speech video. The video content of the speech video can be a PPT presentation, a recorded blackboard of a teacher's classroom explanation, or a recorded synchronous presentation of a conference whiteboard. The embodiments of the present invention do not specifically limit this.

[0051] Video can display information from two modes, audio and image. The two modes of information in the source language video can be separated by audio and video separation, thereby obtaining the source language audio and source language video frame sequence. The source language audio here refers to the audio in the source language video, and the source language video frame sequence refers to the image frame sequence in the source language video in the form of images that can reflect video change information. The source language video frame sequence may include multiple video frames, and the multiple video frames included therein are arranged in sequence, and each video frame corresponds to the time displayed in the source language video.

[0052] Step 120: extract key frames from the source language video frame sequence to obtain source language key frames.

[0053] Considering that the source language video frame sequence contains a large number of video frames, if each video frame in the source language video frame sequence is translated, a large amount of computing resources will be occupied and the efficiency of video translation will be directly affected. And because multiple consecutive video frames in the source language video frame sequence may have a high degree of similarity, especially when the video content is a PPT presentation, translating each video frame in the source language video frame sequence will lead to a waste of computing resources.

[0054] In view of this situation, an embodiment of the present invention extracts key frames from the source language video frame sequence. The key frame extraction here can be achieved by removing redundancy, such as comparing the video frames in the source language video frame sequence, thereby merging the same video frames and taking the remaining video frames as the source language key frames; or the encoding information of each video frame in the source language video frame sequence can be used to determine the video frames that can fully reflect the content of the picture as the source language key frames, and the embodiment of the present invention does not make any specific limitation to this.

[0055] There may be multiple source language key frames obtained through key frame extraction, and there is a sequence between the multiple source language key frames. The sequence here is consistent with the sequence of the source language key frames in the source language video frame sequence.

[0056] Step 130: translating the source language key frame into a target language to obtain a target language key frame.

[0057] Specifically, to translate the text contained in the source language key frame, firstly, OCR (Optical Character Recognition) can be performed on the source language key frame to obtain the text contained in the source language key frame, then the text contained in the source language key frame can be translated into the target language, and finally the translated text can be synthesized into the source language key frame to obtain the target language key frame.

[0058] The target language keyframes here are relative to the source language keyframes. The source language keyframes correspond to the target language keyframes one by one. The text contained in the source language keyframes and the text contained in the target language keyframes are in different languages ​​but have the same semantics. In addition, when there are multiple target language keyframes, the order of the multiple target language keyframes is consistent with the order of the corresponding source language keyframes.

[0059] Step 140 , synthesizing a target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio.

[0060] Specifically, for the translation of source language audio, the source language audio can be firstly subjected to speech recognition, and then the text obtained by speech recognition can be translated into the target language, and then the text in the target language can be made into target language subtitles, or the text in the target language can be subjected to speech synthesis to obtain the target language audio. Regardless of the target language subtitles or the target language audio, both can reflect the display information of the audio modality in the target language.

[0061] After completing the translation of the information in the image modality, the target language key frames obtained by the image modality translation can be combined with the target language subtitles and / or target language audio obtained by the audio modality translation to construct a target language video that can reflect the display information in both the image and audio modalities in the target language. For example, a video can be synthesized based on the target language key frames, and then the target language subtitles can be superimposed on the video to form a target language video, or a video can be synthesized based on the target language key frames, and then the video and the target language audio can be merged to form a target language video.

[0062] The method provided by the embodiment of the present invention translates the key frames in the source language video frame sequence to obtain the target language key frames, and combines the target language key frames with the target language subtitles and / or target language audio obtained by translating the source language audio, so as to fully and comprehensively reflect the content of the video translation. In addition, in this process, the key frames are extracted from the source language video frame sequence, which reduces the computing resource requirements of the translation while ensuring the integrity of the translated video information, and improves the translation efficiency.

[0063] Based on the above embodiments, Figure 2 is a flow chart of step 120 in the translation method provided by the present invention, such as Figure 2 As shown, step 120 includes:

[0064] Step 121: Determine a candidate frame sequence based on the source language video frame sequence.

[0065] Specifically, the candidate frame sequence is a sequence of video frames that may be used as source language key frames. The process of determining the candidate frame sequence based on the source language video frame sequence can be understood as the preliminary screening stage of the source language key frames. In this process, the source language video frame sequence can be directly used as the candidate frame sequence, or based on the encoding method of the source language video frame sequence, video frames of the encoding type that meet the key frame extraction requirements can be selected from the source language video frame sequence to construct a candidate frame sequence, for example, it can include I frames, B frames, and P frames in the H264 protocol. The encoding type that meets the key frame extraction requirements can be I frames, and the embodiment of the present invention does not specifically limit this.

[0066] Step 122: extracting source language key frames from the candidate frame sequence based on the similarity between the candidate frames in the candidate frame sequence.

[0067] Specifically, the similarity between the candidate frames in the candidate frame sequence can be compared to achieve the extraction of the source language key frame. Here, the similarity between the candidate frames in the candidate frame sequence can be compared based on image similarity, semantic similarity, or a combination of image similarity and semantic similarity, which is not specifically limited in the embodiment of the present invention. In this process, if the similarity between the two candidate frames is very high, it can be determined that the information reflected by the two candidate frames is basically the same, which can be understood as the two candidate frames are the same, and the two candidate frames can be merged to achieve redundancy removal, and the candidate frames after redundancy removal are used as the source language key frames; if the similarity between the two candidate frames is very low, it can be determined that the information reflected by the two candidate frames is inconsistent, which can be understood as the two candidate frames are not the same, and the two candidate frames can be retained, and after completing the redundancy removal, both of the two candidate frames are used as the source language key frames.

[0068] Based on the above embodiment, step 122 includes:

[0069] Based on the similarity between every two adjacent candidate frames in the candidate frame sequence, the source language key frames are extracted from the candidate frame sequence.

[0070] Specifically, the extraction of the source language key frames can be achieved by comparing the similarity between every two adjacent candidate frames in the candidate frame sequence. In this process, if the similarity between two adjacent candidate frames is very high, it can be determined that the information reflected by the two adjacent candidate frames is basically the same, which can be understood as the two adjacent candidate frames being the same. The two adjacent candidate frames can be merged to achieve redundancy removal, and then redundancy removal can be performed based on the similarity between the candidate frames after the de-redundancy merge and the subsequent adjacent candidate frames. After completing the de-redundancy removal of all adjacent candidate frames in the candidate frame sequence, the remaining candidate frames can be used as the source language key frames. The source language key frames obtained by this method, combined with the timing of video playback, reduce the computing resource requirements for translation while ensuring the integrity of the video information.

[0071] Based on the above embodiment, step 121 includes:

[0072] The candidate frame sequence is extracted from the source language video frame sequence based on the encoding information of each video frame in the source language video frame sequence.

[0073] Specifically, in the process of acquiring the source language video frame sequence, it is involved to encode the source language video frame, and each video frame in the source language video frame sequence thus obtained may correspond to different encoding information, and the encoding information here may include an encoding type, and may also include a timestamp, etc., which is not specifically limited in the embodiment of the present invention. For example, in the H264 protocol, the encoding type in the encoding information of each video frame in the source language video frame sequence may be an I frame, a B frame, or a P frame, as shown in Table 1.

[0074] Table 1. Video frame encoding types

[0075]

[0076] The PTS (Presentation Time Stamp) in Table 1 records the number of time scales occupied by the display period of each video frame. The I frame represents a key frame, which can fully reflect the complete picture; the P frame represents the difference between this frame and the previous I frame or P frame; the B frame is a bidirectional difference frame, which represents the difference between this frame and the previous and next frames. Since the I frame can fully reflect the complete picture, all I frames in the source language video frame sequence can be selected under the H264 protocol to construct a candidate frame sequence.

[0077] It is understandable that for speech video scenes, PPT presentations are mainly used, so the recorded video mainly reflects the switching of PPT presentations, and only the I frames are extracted, which narrows the extraction range of key frames. At the same time, it ensures that the core content of the video is retained, which can effectively improve the extraction speed of key frames in the video frame sequence.

[0078] It should be noted that in the H264 protocol, the judgment of the I frame is based on the NALU (Network Abstraction Layer Unit) header. The 4-8 bits of the NALU header are judged to be 5 to determine whether it is an I frame. The conventional single-byte value is 0x65.

[0079] The method provided by the embodiment of the present invention extracts a candidate frame sequence from a source language video frame sequence through the encoding information of each video frame, which helps to narrow the extraction range of key frames and reduce the calculation amount of key frame extraction.

[0080] Based on the above embodiment, step 122 includes:

[0081] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is greater than a preset threshold, deleting the next candidate frame and updating the next next candidate frame as the next candidate frame;

[0082] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is less than or equal to a preset threshold, updating the next candidate frame to the current candidate frame until the current candidate frame is the last candidate frame;

[0083] Each candidate frame in the candidate frame sequence is determined as a source language key frame.

[0084] Specifically, in the process of extracting key frames of the source language from the candidate frame sequence, each candidate frame in the candidate frame sequence can be operated frame by frame. Specifically for the candidate frame currently being operated, that is, the current candidate frame, the similarity between the current candidate frame and the next candidate frame can be calculated first, and then the similarity can be compared with a preset threshold to determine whether the current candidate frame and the next candidate frame are the same.

[0085] If the similarity between the current candidate frame and the next candidate frame is greater than a preset threshold, it means that the information reflected by the current candidate frame and the next candidate frame is basically consistent, and there is redundancy between the current candidate frame and the next candidate frame. The next candidate frame can be deleted, and the next-next candidate frame can be used as the next candidate frame. The similarity between the current candidate frame and the next candidate frame is continued to be calculated, and so on, until the current candidate frame is the last candidate frame, where the last candidate frame refers to the last candidate frame in the candidate frame sequence.

[0086] If the similarity between the current candidate frame and the next candidate frame is less than or equal to the preset threshold, it means that the information reflected by the current candidate frame and the next candidate frame is inconsistent. At this time, the current candidate frame can be retained, the next candidate frame can be used as the current candidate frame, and the next candidate frame can be used as the next candidate frame. Then, the similarity between the current candidate frame and the next candidate frame is calculated, and so on, until the current candidate frame is the last candidate frame.

[0087] By calculating and comparing the similarity frame by frame, it is possible to implement frame-by-frame traversal in the candidate frame sequence. After the traversal, every two candidate frames in the candidate frame sequence are different, that is, there is no duplication or redundancy in the candidate frame sequence. At this time, each candidate frame in the candidate frame sequence after the traversal can be used as a key frame of the source language.

[0088] In this process, the similarity can be calculated by extracting features from two candidate frames respectively, and then calculating the similarity based on the extracted features, or calculating the feature difference. Here, the similarity can be calculated using image similarity, semantic similarity, or a combination of image similarity and semantic similarity, which is not specifically limited in the embodiment of the present invention. For example, when using image similarity for calculation, image feature extraction can be implemented by a perceptual hash algorithm, and specifically, the grayscale of each pixel of the candidate frame can be compared with the grayscale average of all pixels of the candidate frame. If the grayscale of the pixel of the candidate frame is greater than or equal to the grayscale average of all pixels of the candidate frame, it is recorded as 1, and if the grayscale of the pixel of the candidate frame is less than the grayscale average of all pixels of the candidate frame, it is recorded as 0, thereby obtaining the hash value of each pixel of the candidate frame, and combining the hash values ​​of all pixels of the candidate frame together to form a sequence, which can be used as the image pixel hash value of the candidate frame. For example, after scaling, the candidate frame is represented as an image of 64 pixels, and the image pixel hash value reflecting the image feature is a 64-bit sequence.

[0089] For the calculation of similarity or feature difference, for example, when using image similarity for calculation, it can be achieved by comparing the number of bits of the image pixel hash values ​​of the two candidate frames, thereby obtaining the number of bits of the image pixel hash values ​​of the two candidate frames, which can reflect the feature difference of the two candidate frames. It can be understood that the greater the feature difference, the lower the image similarity, and the smaller the feature difference, the higher the image similarity. For example, if the number of bits of the image pixel hash values ​​of the two candidate frames is less than or equal to 5, it is determined that the feature difference of the two candidate frames is very small and the image similarity is very high.

[0090] In addition, when using semantic similarity to calculate two candidate frames, the OCR technology can be used to read the text contained in the two candidate frames respectively, and then the semantic features of the text contained in the two candidate frames are extracted respectively, and then the similarity between the semantic features of the two candidate frames is calculated as the semantic similarity of the two candidate frames.

[0091] Based on the above embodiments, Figure 3 is a flow chart of step 130 in the translation method provided by the present invention, such as Figure 3 As shown, step 130 includes:

[0092] Step 131, performing text recognition on the source language key frame to obtain the source language text included in the source language key frame and the position information of the source language text;

[0093] Step 132, translating the source language text into a target language to obtain a target language text;

[0094] Step 133: Based on the position information of the source language text, synthesize the target language text on the source language key frame to obtain the target language key frame.

[0095] Specifically, in the process of translating the source language key frame, it is first necessary to identify the text contained in the source language key frame in the form of an image and locate the position of the text in the source language key frame. After obtaining the text included in the source language key frame, that is, the source language text, and the position information of the source language text, the source language text can be translated to obtain the target language text. For example, if the source language text is English text and the target language is Russian, the English text is translated into Russian text. When there are one or more target languages, the target language text obtained thereby can also be one or more, and the embodiment of the present invention does not make specific limitations.

[0096] After obtaining the target language text, the target language text can be synthesized onto the source language key frame. Specifically, the target language text can be synthesized onto the corresponding source language text position in the source language key frame to cover the corresponding source language text. Alternatively, based on the position of the source language text in the source language key frame, the blank position next to the source language text can be selected to fill the target language text, thereby achieving multi-language display in the target language key frame.

[0097] Based on the above embodiments, Figure 4 is a flow chart of step 140 in the translation method provided by the present invention, such as Figure 4 As shown, step 140 includes:

[0098] Step 141, synthesizing an initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame; the display period is determined based on the playback time of the corresponding source language key frame and the next source language key frame in the source language video.

[0099] Specifically, after the target language key frames are obtained by translating the image modality, video synthesis can be performed based on the target language key frames. In this process, it is necessary to refer to the order of the source language key frames corresponding to the target language key frames in the source language video frame sequence, as well as the display time period of each source language video frame.

[0100] Here, the target language key frame corresponds to the display period of the source language key frame, which is used to reflect the time period covered by the source language key frame when it is displayed in the source language video. Considering that the video frame redundancy operation is performed during the source language key frame extraction process, in order to ensure the coherence of the synthesized video itself, the starting point of the display period of the source language key frame can be the playback time of the source language key frame in the source language video, and the end point can be the playback time of the source language key frame arranged after the source language key frame in the source language video, that is, the playback time of the next source language key frame in the source language video.

[0101] For example, in the H264 protocol, PTS is used to record the number of time scales occupied by the display period of each video frame, where the time scale unit can be calculated by the following formula: Time scale unit = total time length of the source language video / PTS of the last source language video frame

[0102] The playback time of any video frame in the source language video frame sequence can be calculated using the following formula:

[0103] Source language key frame playback time = time scale unit * PTS

[0104] Based on this, we can deduce:

[0105] The display period of the source language key frame I1 is: 0→PTS(I2)*time scale unit;

[0106] The display period of the source language key frame I2 is: PTS(I2)*time scale unit→PTS(I3)*time scale unit; ...;

[0107] The display period of the source language key frame In (the last key frame) is: PTS (In) * time scale unit → the playback end time of the source language video.

[0108] Step 142: superimpose the target language subtitles on the initial video to obtain the target language video.

[0109] Specifically, after obtaining the initial video, the target language subtitles obtained by translating the source language audio can be superimposed on the initial video to obtain the target language video. The target language video here can reflect the display information of both image and audio modes in the target language.

[0110] Based on the above embodiment, the target language subtitles in step 142 can be obtained based on the following method:

[0111] After separating the audio and video of the source language video, the source language audio can be obtained.

[0112] For the source language audio, the transcribed text of the source language audio and the timestamp of the transcribed text can be obtained by speech recognition. Here, the timestamp of the transcribed text is used to reflect the presentation period of the speech corresponding to the transcribed text in the source language audio. The timestamp of the transcribed text can be obtained by performing VAD (Voice Activity Detection) on the source language audio.

[0113] Subsequently, the transcribed text of the source language audio is translated into the target language through text translation to obtain a translated text, and the translated text is combined with a timestamp to generate subtitles in the target language. The target language subtitles here can be SRT (SubRip Text, text format subtitles), or SMI, SSA, etc., which is not specifically limited in the embodiment of the present invention.

[0114] In addition, after obtaining the target language subtitles, FFmpeg (Fast Forward Mpeg) or other types of multimedia video processing tools are used to superimpose the target language subtitles on the original video to obtain the target language video.

[0115] Based on the above embodiments, Figure 5 is a flow chart of the method for synthesizing an initial video provided by the present invention, such as Figure 5 As shown, synthesizing the initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame includes:

[0116] Step 510, splicing the target language key frame and the source language key frame corresponding to the target language key frame to obtain a multi-language key frame;

[0117] Step 520: synthesize an initial video based on the multilingual key frames and the display time periods of the multilingual key frames corresponding to the source language key frames.

[0118] Specifically, when generating the initial video, only the target language key frames may be applied, or both the target language key frames and the source language key frames corresponding to the target language key frames may be applied. Specifically, the corresponding target language key frames and source language key frames may be spliced ​​to form multilingual key frames that can display both the source language information and the target language information in the image mode.

[0119] It should be noted that the target language here can be one or more languages, and the source language can also be one or more languages. The multilingual key frames obtained by translating from the source language to the target language thus include at least two languages.

[0120] On this basis, the initial video can be synthesized based on the multilingual key frames and the display time periods of the multilingual key frames corresponding to the source language key frames. The synthesized initial video can reflect the display information of the image modality in multiple languages.

[0121] In the process of splicing multi-language key frames, a blank canvas can be generated according to the size of the source language key frames, and then the source language key frames and the target language key frames corresponding to the target language key frames are drawn on the left and right sides of the blank canvas respectively, or the source language key frames and the target language key frames corresponding to the target language key frames are drawn on the upper and lower sides respectively, which is not specifically limited in the embodiment of the present invention.

[0122] Based on the above embodiments, Figure 6 This is the second flow chart of the translation method provided by the present invention. Figure 6 As shown, the method includes:

[0123] First, the audio and video of the source language video are separated to obtain the source language video frame sequence and the source language audio.

[0124] Subsequently, the source language audio can be transcribed and translated to obtain subtitles in the target language;

[0125] For the source language video frame sequence, a candidate frame sequence is first determined, and then the source language key frame is extracted based on the similarity between the candidate frames in the candidate frame sequence. In this process, when the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is greater than a preset threshold, the next candidate frame can be deleted, and the next next candidate frame can be updated as the next candidate frame; when the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is less than or equal to the preset threshold, the next candidate frame is updated as the current candidate frame until the current candidate frame is the last candidate frame; each candidate frame in the candidate frame sequence is determined as the source language key frame.

[0126] After the source language keyframe extraction is completed, the source language keyframe can be translated into the target language to obtain the target language keyframe. Then, based on the target language keyframe and the display period of the target language keyframe corresponding to the source language keyframe, the initial video is synthesized.

[0127] Finally, the target language subtitles obtained by translating the source language audio are synthesized on the original video to obtain the target language video.

[0128] The translation device provided by the present invention is described below. The translation device described below and the translation method described above can be referred to each other.

[0129] Based on any of the above embodiments, an embodiment of the present invention provides a translation device. Figure 7 is a schematic diagram of the structure of the translation device provided by the present invention, such as Figure 7 As shown, the device comprises:

[0130] The audio and video separation unit 710 is used to separate the audio and video of the source language video to obtain the source language audio and source language video frame sequence;

[0131] A key frame determination unit 720, configured to extract key frames from the source language video frame sequence to obtain source language key frames;

[0132] A key frame translation unit 730, configured to translate the source language key frame into a target language to obtain a target language key frame;

[0133] The video synthesis unit 740 is used to synthesize a target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio.

[0134] The device provided by the embodiment of the present invention combines the target language key frames with the target language subtitles obtained by translating the source language audio, so as to fully and comprehensively reflect the content of the video translation, and extracts the key frames for the source language video frame sequence, thereby improving the efficiency of translation while ensuring the integrity of the translated video information.

[0135] Based on any of the above embodiments, extracting key frames from the source language video frame sequence to obtain source language key frames includes:

[0136] Determining a candidate frame sequence based on the source language video frame sequence;

[0137] Based on the similarity between the candidate frames in the candidate frame sequence, a source language key frame is extracted from the candidate frame sequence.

[0138] Based on any of the above embodiments, determining a candidate frame sequence based on the source language video frame sequence includes:

[0139] The candidate frame sequence is extracted from the source language video frame sequence based on the encoding information of each video frame in the source language video frame sequence.

[0140] Based on any of the above embodiments, extracting source language key frames from the candidate frame sequence based on the similarity between the candidate frames in the candidate frame sequence includes:

[0141] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is greater than a preset threshold, deleting the next candidate frame and updating the next next candidate frame as the next candidate frame;

[0142] When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is less than or equal to a preset threshold, updating the next candidate frame to the current candidate frame until the current candidate frame is the last candidate frame;

[0143] Each candidate frame in the candidate frame sequence is determined as a source language key frame.

[0144] Based on any of the above embodiments, translating the source language key frame into the target language to obtain the target language key frame includes:

[0145] Performing text recognition on the source language key frame to obtain the source language text included in the source language key frame and position information of the source language text;

[0146] Translating the source language text into a target language to obtain a target language text;

[0147] Based on the position information of the source language text, the target language text is synthesized on the source language key frame to obtain the target language key frame.

[0148] Based on any of the above embodiments, synthesizing the target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio includes:

[0149] Synthesizing an initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame; the display period is determined based on the playback time of the corresponding source language key frame and the next source language key frame in the source language video;

[0150] The target language subtitles are superimposed on the initial video to obtain the target language video.

[0151] Based on any of the above embodiments, synthesizing the initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame includes:

[0152] splicing the target language key frame and the source language key frame corresponding to the target language key frame to obtain a multi-language key frame;

[0153] An initial video is synthesized based on the multilingual key frames and the display time periods of the multilingual key frames corresponding to the source language key frames.

[0154] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the translation method, which includes: performing audio and video separation on the source language video to obtain a source language audio and source language video frame sequence; performing key frame extraction on the source language video frame sequence to obtain a source language key frame; translating the source language key frame into the target language to obtain a target language key frame; synthesizing the target language video based on the target language key frame and the target language subtitles obtained by translating the source language audio.

[0155] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the translation method provided by the above methods, which includes: separating audio and video from a source language video to obtain a source language audio and a source language video frame sequence; extracting key frames from the source language video frame sequence to obtain source language key frames; translating the source language key frames into a target language to obtain target language key frames; and synthesizing a target language video based on the target language key frames and target language subtitles obtained by translating the source language audio.

[0157] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the translation method provided by the above-mentioned methods, the method comprising: performing audio and video separation on a source language video to obtain a source language audio and a source language video frame sequence; performing key frame extraction on the source language video frame sequence to obtain a source language key frame; translating the source language key frame into a target language to obtain a target language key frame; synthesizing a target language video based on the target language key frame and the target language subtitles obtained by translating the source language audio.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A translation method, characterized in that: include: Separating the source language video from audio and video to obtain source language audio and source language video frame sequences; Extracting key frames from the source language video frame sequence to obtain source language key frames; Translating the source language key frame into a target language to obtain a target language key frame; Based on the target language key frames and the target language subtitles obtained by translating the source language audio, a target language video is synthesized, wherein the target language subtitles are used to reflect the presentation information of the audio modality in the target language, and the target language key frames are used to reflect the presentation information of the image modality in the target language.

2. The translation method according to claim 1, characterized in that: The step of extracting key frames from the source language video frame sequence to obtain source language key frames includes: Determining a candidate frame sequence based on the source language video frame sequence; Based on the similarity between the candidate frames in the candidate frame sequence, a source language key frame is extracted from the candidate frame sequence.

3. The translation method according to claim 2, characterized in that: The step of determining a candidate frame sequence based on the source language video frame sequence includes: The candidate frame sequence is extracted from the source language video frame sequence based on encoding information of each video frame in the source language video frame sequence.

4. The translation method according to claim 2, characterized in that: The extracting source language key frames from the candidate frame sequence based on the similarity between the candidate frames in the candidate frame sequence includes: When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is greater than a preset threshold, deleting the next candidate frame and updating the next next candidate frame as the next candidate frame; When the similarity between the current candidate frame and the next candidate frame in the candidate frame sequence is less than or equal to a preset threshold, updating the next candidate frame to the current candidate frame until the current candidate frame is the last candidate frame; Each candidate frame in the candidate frame sequence is determined as a source language key frame.

5. The translation method according to any one of claims 1 to 4, characterized in that: The translating the source language key frame into the target language to obtain the target language key frame includes: Performing text recognition on the source language key frame to obtain the source language text included in the source language key frame and position information of the source language text; Translating the source language text into a target language to obtain a target language text; Based on the position information of the source language text, the target language text is synthesized on the source language key frame to obtain the target language key frame.

6. The translation method according to any one of claims 1 to 4, characterized in that: The synthesizing the target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio includes: Synthesizing an initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame; the display period is determined based on the playback time of the corresponding source language key frame and the next source language key frame in the source language video; The target language subtitles are superimposed on the initial video to obtain the target language video.

7. The translation method according to claim 6, characterized in that: The synthesizing the initial video based on the target language key frame and the display period of the target language key frame corresponding to the source language key frame includes: splicing the target language key frame and the source language key frame corresponding to the target language key frame to obtain a multi-language key frame; An initial video is synthesized based on the multilingual key frames and the display time periods of the multilingual key frames corresponding to the source language key frames.

8. A translation device, characterized in that: include: An audio and video separation unit, used to separate the audio and video of the source language video to obtain a source language audio and source language video frame sequence; A key frame determination unit, configured to extract key frames from the source language video frame sequence to obtain source language key frames; A key frame translation unit, used for translating the source language key frame into a target language to obtain a target language key frame; The video synthesis unit is used to synthesize a target language video based on the target language key frames and the target language subtitles obtained by translating the source language audio, wherein the target language subtitles are used to reflect the presentation information of the audio mode in the target language, and the target language key frames are used to reflect the presentation information of the image mode in the target language.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the translation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the translation method according to any one of claims 1 to 7 is implemented.

Images