Video processing method, device and storage medium

Abstract

The present application relates to a video processing method, device and storage medium, the method comprising: during video playback, in response to a language conversion instruction, converting the human voice data of the first language in the part of the video located after the current playback point in the video into the human voice data of the second language to obtain the converted video; and continuing to play the converted video from the current playback point. This can achieve a function of converting the human voice in the video from one language to another language in real time or near real time during video playback, so that the user can directly understand the video content when watching the video, and enhance the user's immersion and experience when watching the video.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a video processing method, device and storage medium. Background Art

[0002] With the development of the Internet, more and more users need to watch videos with original foreign language sounds in their daily work and study. However, when watching these videos in foreign languages, users generally face language comprehension barriers.

[0003] In order to overcome the language comprehension barrier, users try to take some measures. For example, they choose to translate the video content by themselves, by pausing the video and using tools such as online translation or dictionary to translate words. However, this method is not only inefficient, but also destroys the coherence and fluency of the video, seriously affecting the viewing experience. For another example, try to use the existing speech-to-text technology for machine translation. Although this method can help users understand the video content to a certain extent, users still need to switch back and forth between the translated text and the video screen, which is not very efficient. In addition, due to the limitations of machine translation, the translation results may be inaccurate or ambiguous, further increasing the difficulty for users to understand the video content. Some users also choose to search for the Chinese version of foreign videos on the Internet. However, this method is not always feasible. Because not all foreign videos have Chinese versions, and even if the Chinese version is found, it may be necessary to pay or face other cost issues. This increases the threshold and cost for users to watch foreign videos.

[0004] It can be seen that no matter which method users use to watch foreign language videos, they face many difficulties and challenges. Therefore, there is an urgent need for a technical solution that can effectively solve these problems to meet the needs of users to watch foreign language videos smoothly and effectively. Summary of the invention

[0005] The present application provides a video processing method, device and storage medium to solve the technical problem that related technologies cannot meet the needs of users to watch foreign language videos smoothly and effectively.

[0006] In a first aspect, the present application provides a video processing method, the method comprising:

[0007] During video playback, in response to a language conversion instruction, converting human voice data of a first language in a portion of the video located after a current playback point in the video into human voice data of a second language to obtain a converted video;

[0008] Continue playing the converted video from the current playback point.

[0009] In a possible implementation, the method further includes:

[0010] During the process of playing the converted video, in response to the original sound recovery instruction, the original video continues to be played from the current playback point.

[0011] In a possible implementation manner, after converting the human voice data in the first language in the portion of the video located after the current playback point in the video into human voice data in the second language, the method further includes:

[0012] Adjusting the converted video;

[0013] The step of continuing to play the converted video from the current play point is performed using the adjusted converted video.

[0014] In a possible implementation manner, adjusting the converted video includes:

[0015] Compounding the timbre data in the first language vocal data with the second language vocal data in the converted video; and / or,

[0016] According to the human voice data of the second language, the vocal action of the target sound-making object in the video frame data in the converted video is matched.

[0017] In a possible implementation manner, converting the human voice data in the first language in a portion of the video located after the current playback point in the video into human voice data in the second language to obtain the converted video includes:

[0018] Acquire text data in the first language of a portion of the video located after a current playback point in the video;

[0019] translating text data in the first language into text data in a second language;

[0020] generating voice data in the second language based on the text data in the second language;

[0021] The human voice data of the second language and the video frame data of the portion of the video located after the current playback point in the video are synthesized to obtain a converted video.

[0022] In a possible implementation manner, the obtaining of text data in the first language of a portion of the video located after the current playback point in the video includes:

[0023] Separating audio data from a portion of the video located after a current playback point in the video, and separating human voice data of a first language from the audio data;

[0024] Speech-to-text recognition is performed on the human voice data in the first language to obtain text data in the first language of a portion of the video located after a current playback point in the video.

[0025] In a possible implementation manner, the replacing the first language human voice data in a portion of the video located after the current playback point in the video with the second language human voice data to obtain a converted video includes:

[0026] Determine a playback time period of each piece of text data in the first language in the video;

[0027] According to the playback time period corresponding to each piece of text data in the first language, cutting a portion of the video located after a current playback point in the video into a plurality of video segments, and recording a corresponding relationship between each of the video segments and the text data in the first language;

[0028] According to the corresponding relationship, the human voice data of the second language corresponding to each of the video segments is determined, and the video frame data in each of the video segments is synthesized with the human voice data of the second language corresponding to the video segment to obtain a converted video.

[0029] In a possible implementation, synthesizing the video frame data in each of the video segments with the human voice data of the second language corresponding to the video segment to obtain a converted video includes:

[0030] The following processing is performed for each of the video segments to obtain a converted video:

[0031] Determine a first playback duration of the video frame data in the video segment, and determine a second playback duration of the human voice data in a second language corresponding to the video segment;

[0032] When the first play time is inconsistent with the second play time, adjusting the play time of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent;

[0033] The video frame data in the video segment adjusted to have a consistent playback duration is synthesized with the corresponding second language human voice data.

[0034] In a possible implementation manner, adjusting the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent includes:

[0035] Determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0036] According to the extension multiple, the video frame data in the video segment is cyclically spliced ​​to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0037] In a possible implementation manner, adjusting the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent includes:

[0038] Determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0039] According to the extension multiple, the video frame data in the video segment is speed-changed to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0040] In a second aspect, the present application provides a video processing device, the device comprising:

[0041] a conversion module, configured to, during video playback, in response to a language conversion instruction, convert human voice data of a first language in a portion of the video located after a current playback point in the video into human voice data of a second language to obtain a converted video;

[0042] The video playing module is used to continue playing the converted video from the current playing point.

[0043] In a possible implementation manner, the video playback module is further used to:

[0044] During the process of playing the converted video, in response to the original sound recovery instruction, the original video continues to be played from the current playback point.

[0045] In a possible implementation, the device further includes:

[0046] A video adjustment module, configured to adjust the converted video after converting the human voice data of the first language in the portion of the video located after the current playback point into human voice data of the second language;

[0047] The video playing module is used to use the adjusted converted video to execute the step of continuing to play the converted video from the current playing point.

[0048] In a possible implementation, the video adjustment module includes:

[0049] A timbre adjustment unit, for compounding the timbre data in the human voice data of the first language into the human voice data of the second language in the converted video;

[0050] The action adjustment unit is used to match the vocal action of the target sound-making object in the video frame data in the converted video according to the human voice data of the second language.

[0051] In a possible implementation, the conversion module includes:

[0052] A text acquisition unit, used to acquire text data in the first language of a portion of the video located after a current playback point in the video;

[0053] a text translation unit, configured to translate text data in the first language into text data in a second language;

[0054] a text-to-speech unit, configured to generate voice data in the second language based on the text data in the second language;

[0055] The synthesis unit is used to synthesize the human voice data of the second language and the video frame data in the part of the video located after the current playback point in the video to obtain a converted video.

[0056] In a possible implementation, the text acquisition unit includes:

[0057] An audio separation subunit, configured to separate audio data from a portion of the video located after a current playback point in the video, and separate human voice data of a first language from the audio data;

[0058] The text recognition subunit is used to perform speech-to-text recognition on the human voice data in the first language to obtain text data in the first language of a portion of the video located after the current playback point in the video.

[0059] In a possible implementation, the synthesis unit includes:

[0060] a time period determination subunit, used to determine a play time period of each piece of text data in the first language in the video;

[0061] a video segmentation subunit, configured to cut a portion of the video located after a current playback point in the video into a plurality of video segments according to the playback time period corresponding to each piece of text data in the first language;

[0062] a recording subunit, configured to record a correspondence between each of the video segments and the text data in the first language;

[0063] The synthesis subunit is used to determine the human voice data of the second language corresponding to each of the video segments according to the corresponding relationship, and synthesize the video frame data in each of the video segments with the human voice data of the second language corresponding to the video segment to obtain a converted video.

[0064] In a possible implementation, the synthesis subunit is specifically used for:

[0065] The following processing is performed for each of the video segments to obtain a converted video:

[0066] Determine a first playback duration of the video frame data in the video segment, and determine a second playback duration of the human voice data in a second language corresponding to the video segment;

[0067] When the first play time is inconsistent with the second play time, adjusting the play time of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent;

[0068] The video frame data in the video segment adjusted to have a consistent playback duration is synthesized with the corresponding second language human voice data.

[0069] In a possible implementation, the synthesis subunit includes:

[0070] A first alignment subunit, configured to determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0071] According to the extension multiple, the video frame data in the video segment is cyclically spliced ​​to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0072] In a possible implementation, the synthesis subunit includes:

[0073] A second alignment subunit, configured to determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0074] According to the extension multiple, the video frame data in the video segment is speed-changed to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0075] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a video processing program stored in the memory to implement the video processing method described in any one of the first aspects.

[0076] In a fourth aspect, the present application provides a storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the video processing method described in any one of the first aspects.

[0077] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, by responding to the language conversion instruction during the video playback, converting the human voice data of the first language in the part of the video located after the current playback point in the video into the human voice data of the second language, obtaining the converted video, and continuing to play the converted video from the current playback point, realizes a function that can convert the human voice in the video from one language to another language in real time or near real time during the video playback, so that the user can directly understand the video content when watching the video, and the user does not need to switch his or her line of sight between subtitles or other translation tools and the video screen, so that the user can focus more on the video content itself, thereby enhancing the user's immersion and experience when watching the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0079] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0080] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0081] Figure 1 A flowchart of a video processing method provided in an embodiment of the present application;

[0082] Figure 2 A flowchart of an embodiment of the present application for converting vocal data of a first language in a portion of a video located after a current playback point in the video to vocal data of a second language to obtain a converted video;

[0083] Figure 3 This is a schematic diagram of the converted video with subtitles;

[0084] Figure 4A block diagram of an embodiment of a video processing device provided in an embodiment of the present application;

[0085] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0087] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0088] In order to solve the technical problem that related technologies cannot meet the needs of users to watch foreign language videos smoothly and effectively, the present application provides a video processing method, device and storage medium, which can realize a function of converting the human voice in the video from one language to another language in real time or near real time during video playback, so that users can directly understand the video content when watching the video, thereby enhancing the user's immersion and experience when watching the video.

[0089] Figure 1 A flow chart of an embodiment of a video processing method provided in an embodiment of the present application. In one embodiment, the method is applied to a video player, where the video player can be a hardware device or software that supports network connection to provide various network services. When the video player is hardware, it can support various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the video player is software, it can be a local video player installed in the above electronic devices, or a streaming media player used in a website, or a browser plug-in. The embodiment of the present application does not limit the type and implementation form of the video player. For example Figure 1 As shown, the method comprises the following steps:

[0090] Step 101 : During video playback, in response to a language conversion instruction, convert human voice data of a first language in a portion of the video after a current playback point into human voice data of a second language to obtain a converted video.

[0091] Step 102: Continue playing the converted video from the current playback point.

[0092] For ease of understanding, step 101 and step 102 are described in a unified manner as follows:

[0093] During video playback, that is, when the user is watching a video, the user can trigger a language conversion instruction for the currently playing video to instruct the human voice in part of the video (referring to the part of the video segment currently playing) to be converted from one language (for the convenience of description, referred to as the first language, that is, the language currently used by the video) to another language (for the convenience of description, referred to as the second language, that is, the target language that the user wants to convert to). Exemplarily, the first language and the second language are different languages, for example, the first language is English and the second language is Chinese; or, the first language and the second language are different dialects, for example, the first language is the dialect of region A and the second language is the dialect of region B; or, the first language is a Chinese dialect and the second language is Mandarin Chinese; and so on. There is no limitation on the specific types of the first language and the second language here. Exemplarily, the above-mentioned language conversion instruction can be triggered in a variety of ways, such as through buttons, shortcut keys, voice commands, etc. on the video player interface.

[0094] The human voice category includes real human voice, cartoon character anthropomorphic voice, etc. Based on this, for example, the first language can be a real human voice, and the second language can be a cartoon character anthropomorphic voice. This enables the technical solution provided in the embodiment of the present application to be applied to a variety of interesting video voice conversion scenarios, such as cartoon character dubbing scenarios, to increase the fun of users watching videos and improve user experience.

[0095] In one embodiment, the current play point refers to the play point to which the video is played when the user triggers a language change instruction on the video during the video playback process. For example, when the video is played to 5 minutes and 6 seconds, the user triggers a language change instruction on the video, and the current play point is 5 minutes and 6 seconds. In this embodiment, the language conversion operation is for the portion of the video after the current play point, that is, the portion of the video that the user has not watched, and the portion of the video that the user has watched no longer participates in the language conversion operation. For example, the vocal data of the first language in the portion of the video located after 5 minutes and 6 seconds in the video is converted into the vocal data of the second language to obtain the converted video.

[0096] In another embodiment, the current play point refers to the play point indicated when the user triggers the language conversion instruction to the video during the video playback process. The play point can be any play point, such as any play point after the play point currently played by the video, or any play point before the play point currently played by the video (including the play start point of the entire video). In this embodiment, the language conversion operation is also aimed at the part of the video after the current play point, but here if the current play point is before the play point currently played by the video, then the part of the video participating in the language conversion operation will include the part of the video that the user has watched. For example, when the video is played to 5 minutes and 6 seconds, the user triggers the language conversion instruction to the video and indicates a specific conversion starting point: 3 minutes and 20 seconds. Then the current play point is 3 minutes and 20 seconds. After that, the human voice data of the first language in the part of the video located after 3 minutes and 20 seconds in the video is converted into the human voice data of the second language to obtain the converted video. For another example, when the video is played to 5 minutes and 6 seconds, the user triggers the language conversion instruction for the video and indicates a specific conversion starting point: 0 seconds (the start point of the entire video). Then the current playback point is the start point of the entire video. After that, the human voice data in the first language of the entire video is converted into human voice data in the second language to obtain the converted video, and then the converted video is played so that the user can watch the complete video after the language conversion from the beginning.

[0097] This method of allowing users to select the language conversion starting point as needed provides greater flexibility and can better meet the needs of users in different scenarios, for example, it is suitable for the situation where users want to review and understand another language version of a part of the content they have watched.

[0098] In one embodiment, the portion of the video after the current playback point in the video refers to: all the videos after the current playback point in the video. For example, if the total length of the video is 10 minutes and the current playback point is 5 minutes and 6 seconds, then the portion of the video after the current playback point in the video refers to: videos within 5 minutes and 6 seconds to 10 minutes. In this embodiment, once the user triggers the language conversion instruction for the video, the language conversion operation will be completed at one time, which in actual applications allows users to watch subsequent videos without worrying about language issues.

[0099] In another embodiment, the portion of the video after the current playback point in the video refers to: a video within a set duration after the current playback point in the video. Here, as the video is played, this portion of the video will also be gradually pushed backwards to achieve language conversion while playing. Finally, until the video is played, or the user triggers the language conversion instruction for the video again. In this embodiment, the language conversion operation is dynamic and is performed step by step as the video is played. That is, as the video is played, the converted content is gradually presented to the user. This method can save computing resources in a scenario where the user only needs to perform language conversion operations on part of the video.

[0100] It can be seen from the above embodiments that the video-to-language function provided in the embodiments of the present application has high flexibility and diversity, and users can choose different application modes according to actual needs, thereby improving user experience.

[0101] As mentioned in the above description, during the process of playing the converted video, the user can also trigger the language conversion instruction again for the video (here refers to the converted video). This can be discussed in two cases:

[0102] One scenario is: the user instructs to convert the human voice of part of the video from the current second language to a third language, and the third language is different from the second language and the first language. It can be seen from this that the technical solution provided by the embodiment of the present application allows the user to switch freely between multiple languages ​​according to their needs and preferences, and can respond to the user's language conversion instructions in real time or near real time to perform language conversion, so that the user can seamlessly switch languages ​​during video viewing without affecting the viewing experience. This greatly improves the flexibility and personalized experience of video viewing.

[0103] Another situation is: the user instructs to restore the human voice of part of the video from the current second language to the first language, that is, to restore the original sound. Specifically, in the process of playing the converted video, in response to the original sound restoration instruction (which can also be regarded as a kind of language conversion instruction), the original video continues to be played from the current playback point. It can be seen from this that the technical solution provided by the embodiment of the present application allows real-time language restoration and presents the restored video content to the user, which can meet the ever-changing needs of users.

[0104] As for the specific implementation of converting the vocal data of the first language in the part of the video located after the current playback point in the video into the vocal data of the second language to obtain the converted video, it is explained in the following through exemplary embodiments and will not be described in detail here.

[0105] The technical solution provided in the embodiment of the present application converts the human voice data in the first language in the part of the video located after the current playback point into the human voice data in the second language in response to the language conversion instruction during the video playback process to obtain the converted video, and continues to play the converted video from the current playback point, thereby realizing a function of converting the human voice in the video from one language to another language in real time or near real time during the video playback process, so that the user can directly understand the video content when watching the video, and the user does not need to switch his or her line of sight between subtitles or other translation tools and the video screen, so that the user can focus more on the video content itself, thereby enhancing the user's immersion and experience when watching the video.

[0106] Figure 2 A flowchart of an embodiment of the present application is provided for converting vocal data of a first language in a portion of a video located after a current playback point in the video to obtain a converted video. Figure 2 exist Figure 1 Based on the process shown, the following steps are included:

[0107] Step 201: Obtain text data in the first language of a portion of the video that is located after the current playback point.

[0108] In one embodiment, the specific implementation of obtaining the text data in the first language of the portion of the video located after the current playback point in the video includes: separating audio data from the portion of the video located after the current playback point in the video, and separating vocal data in the first language from the audio data; performing speech-to-text recognition on the vocal data in the first language to obtain the text data in the first language of the portion of the video located after the current playback point in the video.

[0109] In the above embodiment, the current playback time point is first determined, and then a video editing tool is used, or a corresponding interface is called to cut out a portion of the video starting from the current playback point from the video. Afterwards, an audio and video separation tool is used, or a corresponding interface is called to separate the audio data and the video frame data in the video. This usually produces a separate audio file and a separate video frame file. The audio in the video usually includes human voice and background sound, so human voice data will also be separated from the aforementioned separated audio data, and the human voice data is human voice data in the first language. Finally, a speech-to-text recognition algorithm (such as a machine learning model or other tools) can be used to perform speech-to-text recognition on the human voice data in the first language, thereby obtaining text data in the first language of the portion of the video located after the current playback point in the video.

[0110] Exemplarily, before using a language recognition algorithm to perform speech-to-text recognition on the human voice data of the first language, the extracted human voice data is preprocessed, such as noise reduction, standardization and other processing operations, to improve the accuracy of subsequent processing.

[0111] Similarly, before separating the human voice data from the audio data, the extracted audio data may be preprocessed, such as noise reduction, standardization, and other processing operations, to improve the accuracy of subsequent processing.

[0112] In another embodiment, the subtitle file of the video (if any) may also be obtained, and the subtitle data of the portion of the video after the current playback point may be determined from the subtitle file, which subtitle data is the text data in the first language.

[0113] Here, the large model can be used to proofread the subtitle data to make the content in the subtitle data more accurate. Furthermore, in response to the user's request to view the subtitle data, the subtitle content can be displayed multiple times or once, and the displayed content can be the subtitles corresponding to the content that has not been played in the video.

[0114] Step 202: Translate the text data in the first language into text data in the second language.

[0115] In one embodiment, text data in a first language and a second language are input into a text translation model, so that the text data in the first language is translated into text data in the second language by the text translation model.

[0116] Exemplarily, the text translation model is implemented using a neural network, such as a model of a Transformer architecture.

[0117] For example, after translating text data in a first language into text data in a second language using a text translation model, a series of post-processing operations may be performed on the text data in the second language, such as text formatting, punctuation adjustment, etc., to ensure that it complies with the norms and habits of the second language and to improve the accuracy of subsequent processing.

[0118] Step 203: Generate voice data in the second language based on the text data in the second language.

[0119] In one embodiment, the text data in the second language is input into a text-to-speech model, so that the text-to-speech model generates voice-like data in the second language from the text data in the second language.

[0120] Exemplarily, the text-to-speech model is implemented using a deep learning-based method (such as WaveNet, Tacotron, etc.).

[0121] Exemplarily, after using the text-to-speech model to generate voice data in the second language from text data in the second language, a series of post-processing operations, such as volume adjustment, noise removal, etc., can be performed on the generated voice data to ensure quality and improve the accuracy of subsequent processing.

[0122] Step 204 : synthesize the human voice data of the second language and the video frame data in the portion of the video after the current playback point in the video to obtain a converted video.

[0123] In step 204, the human voice data of the second language is synthesized with the video frame data separated in the aforementioned step 201 to generate a converted video file. In addition, in the case where the background sound data is separated in the aforementioned step, the human voice data of the second language, the background sound data and the video frame data separated in the aforementioned step 201 are synthesized together to generate a converted video file.

[0124] In this step, ensuring that the second language voice data is synchronized with the video frame data in time is an important operation. By ensuring that the second language voice data is synchronized with the video frame data in time, the problem of sound delay during video playback can be avoided and the quality of the converted video can be improved.

[0125] In this regard, in one embodiment, the human voice data in the second language and the video frame data in the part of the video located after the current playback point in the video are synthesized and processed to obtain the converted video. The specific implementation includes: determining the playback time period of each text data in the first language in the video; according to the playback time period corresponding to each text data in the first language, the part of the video located after the current playback point in the video is cut into multiple video segments, and the correspondence between each video segment and the text data in the first language is recorded; according to the correspondence, the human voice data in the second language corresponding to each video segment is determined, and the video frame data in each video segment is synthesized with the human voice data in the second language corresponding to the video segment to obtain the converted video.

[0126] If the video has a subtitle file, the text data in the first language and its corresponding timestamp can be directly extracted from the subtitle file. The timestamp includes the subtitle data, that is, the start time and end time of the text data, and the start time and end time form the playback time period corresponding to the text data. Here, the start time and end time of each subtitle data are converted into a unified time format (such as using milliseconds as the basic unit), and in the parsing process, possible encoding problems are handled, and the validity of the subtitle timeline is verified, so as to ensure the continuity and rationality of the timestamp and improve the accuracy of subsequent processing.

[0127] If the video does not have a subtitle file, each piece of text data in the first language and its corresponding timestamp can be identified through embedded subtitles in the video frame data or by using optical character recognition technology to identify available text in the video frame data. The specific implementation method is not described here.

[0128] When the part of the video located after the current playback point in the video is cut into multiple video segments according to the playback time period corresponding to each piece of text data in the first language, the image group structure of the video is first analyzed to find the key frame position closest to the subtitle time point, and the cutting is performed from the key frame position, so as to ensure the integrity of the picture at the cutting point. Exemplarily, the FFmpeg tool is used during cutting to achieve lossless cutting as much as possible to avoid quality loss caused by re-encoding. In addition, in order to deal with possible audio and video synchronization problems, a small amount of buffer time (usually 0.1-0.2 seconds) can be reserved at each cutting point, and a smooth transition can be achieved later through a cross-fade effect.

[0129] In addition, in terms of specific implementation, cutting can be done in batch mode, that is, all cutting tasks are organized into queues and multi-threaded to improve efficiency. At the same time, a temporary file management mechanism needs to be established to clean up intermediate files in a timely manner, and the quality of the cut segments needs to be verified after processing to ensure the integrity and continuity of the output video. A log recording system is established throughout the process to record key operation nodes and possible abnormal situations, which is convenient for subsequent problem investigation and optimization and improvement.

[0130] Finally, the correspondence between each video segment and the text data in the first language is recorded. Based on the correspondence, the correspondence between each video segment and the text data in the second language can be constructed, and then the human voice data in the second language corresponding to each video segment is determined according to the correspondence, and the video frame data in each video segment is synthesized with the human voice data in the second language corresponding to the video segment to obtain the converted video.

[0131] Generally speaking, different languages ​​have their own characteristics in terms of speaking speed, pronunciation, intonation, etc. These characteristics will directly affect the duration of the same text data in different languages. For example, some languages ​​tend to be spoken at a faster speed, while others tend to be spoken at a relatively slower speed. Therefore, when performing multi-language conversion and processing, it is also necessary to consider the impact of these language characteristics on the duration of speech, and take corresponding measures to ensure that the converted human voice data is aligned with the video frame data in time, so as to ensure the synchronization of human voice data with the video screen and avoid problems such as sound delay.

[0132] In this regard, in one embodiment, the video frame data in each video segment is synthesized with the vocal data of the second language corresponding to the video segment to obtain the converted video. The specific implementation includes: for each video segment, determining a first playback duration of the video frame data in the video segment, and determining a second playback duration of the vocal data of the second language corresponding to the video segment; when the first playback duration and the second playback duration are inconsistent, adjusting the playback durations of the video frame data in the video segment and the vocal data of the second language corresponding to the video segment to be consistent; synthesizing the video frame data in the video segment whose playback durations are adjusted to be consistent with the corresponding vocal data of the second language.

[0133] The inconsistency between the first playback duration and the second playback duration mentioned above usually means that the second playback duration is longer than the first playback duration, that is, the duration of the vocal data of the second language is longer than the playback duration of the corresponding video clip. In this case, there will be a synchronization problem between the audio and video images. For example, the duration of the vocal data exceeds the duration of the video image, and the audience may notice that the speaker's mouth shape does not match the sound heard. For another example, in certain parts of the video image, the vocal data may lag behind the image content, resulting in a time difference between the sound and the image. For another example, the vocal data is too long, and the video image has ended, and the sound continues to play on the empty image.

[0134] In one embodiment, the specific implementation of adjusting the playback duration of video frame data in a video segment and the human voice data in a second language corresponding to the video segment to be consistent includes: determining an extension multiple of the video playback duration based on the second playback duration and the first playback duration; and according to the extension multiple, cyclically splicing the video frame data in the video segment to adjust the playback duration of the video frame data in the video segment and the human voice data in the second language corresponding to the video segment to be consistent.

[0135] The above-mentioned loop splicing refers to repeatedly splicing the video frame data in the video segment to achieve the purpose of looping the video segment. The number of loop splicing depends on the extension multiple of the video playback time. Among them, the extension multiple of the video playback time can be calculated by dividing the second playback time by the first playback time.

[0136] For example, a different playback rate is used for each loop to avoid mechanical repetition. In addition, a smooth transition is performed at the key frame position at the splicing point. For example, an optical flow algorithm can be used to calculate the inter-frame motion, and a cross-dissolve effect can be used at the transition point to achieve a natural connection.

[0137] In another embodiment, the specific implementation of adjusting the playback duration of video frame data in a video segment and the vocal data of a second language corresponding to the video segment to be consistent includes: determining an extension multiple of the video playback duration based on the second playback duration and the first playback duration; and performing speed control on the video frame data in the video segment based on the extension multiple to adjust the playback duration of the video frame data in the video segment and the vocal data of a second language corresponding to the video segment to be consistent.

[0138] Exemplarily, the video frame data in the video segment is processed at a variable speed by frame interpolation. Among them, a suitable frame interpolation algorithm can be automatically selected according to the video content characteristics. For example, for scenes with intense motion, the RIFE (Real-Time Intermediate Flow Estimation) algorithm is used, which can better handle fast-moving objects. For static scenes, the DAIN (Depth-Aware Video Frame Interpolation) algorithm is used to maintain picture details. For close-ups of people, a special interpolation algorithm based on facial key points is used to ensure natural facial details.

[0139] For example, while using frame interpolation to process the speed of video frame data in the video segment, a quality evaluation system is established to dynamically adjust the interpolation parameters by calculating the peak signal-to-noise ratio and structural similarity index to ensure the smoothness and image quality of the output video. During the processing, the audio pitch is kept unchanged and only the playback speed is adjusted. The rubberband filter of FFmpeg can be used to achieve high-quality audio time stretching.

[0140] For example, in the process of synthesizing the video frame data in the video segment whose playback duration is adjusted to be consistent with the corresponding second language vocal data, the audio timestamp can be automatically corrected and appropriate audio delay compensation can be set to solve possible audio and video synchronization problems. At the same time, high-quality encoders and parameters are used to ensure the quality of the output video, and the audio and video synchronization is verified after the merger is completed to ensure the quality of the final converted video.

[0141] In addition, during the merging process, an error handling mechanism can also be established. If any abnormality is found during the merging process (such as serious asynchrony between audio and video, encoding errors, etc.), it will be interrupted in time and detailed error information will be provided to facilitate subsequent debugging and correction.

[0142] Finally, after the video frame data and the corresponding second language human voice data are synthesized for each video segment, the synthesized multiple video segments are sequentially spliced ​​to obtain a converted video.

[0143] For example, when multiple video segments after synthesis processing are spliced ​​in sequence, the following series of operations can also be performed: Ensure that all video segments have the same encoding parameters, resolution, and frame rate to ensure that the converted video can be played normally. Add smooth transition effects (such as cross-fade) at the transition between two adjacent video segments, and use key frame alignment technology to ensure the smoothness of the segment connection. Maintain the continuity of the audio to avoid popping or interruption at the splicing. Verify the quality of the final output, including transition effects and audio and video synchronization. And so on.

[0144] In addition, in specific applications, if there are many video segments, a batch merging strategy can be adopted to first merge adjacent segments into larger blocks, and then gradually merge these blocks to improve processing efficiency and stability. At the same time, if memory allows, memory cache can be used to speed up the merging process.

[0145] Figure 2 The process shown in the figure realizes converting the human voice data of the first language in the part of the video after the current playback point into the human voice data of the second language to obtain the converted video. Figure 2 In the process shown, by cutting the part of the video after the current playback point into multiple video segments according to the playback time period corresponding to each piece of text data in the first language, it is possible to accurately match the second language vocal data with the video segment. This accurate matching is a key step to ensure the synchronization of sound and picture after language conversion. When the playback time of the video segment is inconsistent with the playback time of the vocal data in the second language, the video frame data in the video segment and the playback time of the vocal data in the second language corresponding to the video segment are adjusted to be consistent, and the video frame data in the video segment whose playback time is adjusted to be consistent is synthesized with the corresponding vocal data in the second language, so that the synthesized video can be ensured to maintain a high degree of synchronization in sound and picture. This improves the quality of the video and enhances the user's viewing experience.

[0146] Based on the above embodiments, the present application also provides the following embodiments:

[0147] In one embodiment, after converting vocal data of a first language in a portion of the video located after a current playback point into vocal data of a second language, the converted video is adjusted; and the step of continuing to play the converted video from the current playback point is performed using the adjusted converted video.

[0148] Exemplarily, the above adjustment includes compression, format conversion and other adjustment operations to improve the quality of the converted video and minimize quality loss. The format conversion can also improve the compatibility of the converted video, so that the generated converted video can be played normally on a variety of devices and players.

[0149] Exemplarily, adjusting the converted video includes: compounding the timbre data in the vocal data of the first language into the vocal data of the second language in the converted video; and / or, matching the vocal action of the target sound-making object in the video frame data in the converted video based on the vocal data of the second language.

[0150] The target sound-generating object may be a person or a cartoon character. The sound-generating action of the target sound-generating object may include facial expressions, lip movements, head movements, etc.

[0151] When the vocal movements of the target sound-making object in the video frame data in the converted video are matched according to the human voice data of the second language, a deep learning model can be used to achieve more accurate synchronization of vocal movements. For example, when synchronizing the lip shape and mouth shape, the contextual relationship of the phonemes is considered to ensure that the lip shape and mouth shape transition are natural. When matching the facial expressions of the target sound-making object, the coordination of facial expressions and head movements is achieved, and small random changes are added to avoid a mechanical feel, and the natural movements of other parts of the face are maintained. At the same time, attention is paid to maintaining the physical rationality of facial muscle movements to avoid unnatural deformations.

[0152] Here, when synchronization is performed, the lip shape and mouth shape are adjusted according to the matching situation so that the vocal data can be coordinated, or the vocal data is adjusted to match the lip shape and mouth shape to achieve a coordinated effect.

[0153] By combining the timbre data in the first language's vocal data with the second language's vocal data in the converted video, the voice characteristics of the original speaker can be retained in the converted video, making the converted voice sound more like the original speaker speaking in the second language. By matching the vocal movements of the target sound-making object in the video frame data in the converted video according to the second language's vocal data, the mouth shape, expression, etc. of the target sound-making object (such as a person) in the converted video can be made more consistent with the pronunciation characteristics of the second language. This can significantly improve the realism and naturalness of the converted video, and enhance the viewing and credibility of the converted video.

[0154] Exemplarily, the adjustment in the above matching may also include: adding subtitles to the converted video according to the text data in the second language. For example, the text data in the second language is encapsulated as a subtitle file and burned into the converted video. In the burning process, the subtitle style (including font, size, color, stroke, etc.) can be set by a custom style to achieve a richer display effect. At the same time, the position and time axis of the subtitles are fine-tuned before burning to ensure the rationality and readability of the subtitle display.

[0155] For example, see Figure 3 , which is a schematic diagram of the converted video with subtitles.

[0156] Figure 4 This is a block diagram of an embodiment of a video processing device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0157] The conversion module 41 is used to convert the human voice data of the first language in the part of the video located after the current playback point in the video into the human voice data of the second language in response to the language conversion instruction during the video playback process to obtain the converted video;

[0158] The video playing module 42 is used to continue playing the converted video from the current playing point.

[0159] In a possible implementation manner, the video playback module 42 is further configured to:

[0160] During the process of playing the converted video, in response to the original sound recovery instruction, the original video continues to be played from the current playback point.

[0161] In a possible implementation, the device further includes:

[0162] A video adjustment module, configured to adjust the converted video after converting the human voice data of the first language in the portion of the video located after the current playback point into human voice data of the second language;

[0163] The video playing module is used to use the adjusted converted video to execute the step of continuing to play the converted video from the current playing point.

[0164] In a possible implementation, the video adjustment module includes:

[0165] A timbre adjustment unit, for compounding the timbre data in the human voice data of the first language into the human voice data of the second language in the converted video;

[0166] The action adjustment unit is used to match the vocal action of the target sound-making object in the video frame data in the converted video according to the human voice data of the second language.

[0167] In a possible implementation, the conversion module 41 includes:

[0168] A text acquisition unit, used to acquire text data in the first language of a portion of the video located after a current playback point in the video;

[0169] a text translation unit, configured to translate text data in the first language into text data in a second language;

[0170] a text-to-speech unit, configured to generate voice data in the second language based on the text data in the second language;

[0171] The synthesis unit is used to synthesize the human voice data of the second language and the video frame data in the part of the video located after the current playback point in the video to obtain a converted video.

[0172] In a possible implementation, the text acquisition unit includes:

[0173] An audio separation subunit, configured to separate audio data from a portion of the video located after a current playback point in the video, and separate human voice data of a first language from the audio data;

[0174] The text recognition subunit is used to perform speech-to-text recognition on the human voice data in the first language to obtain text data in the first language of a portion of the video located after the current playback point in the video.

[0175] In a possible implementation, the synthesis unit includes:

[0176] a time period determination subunit, used to determine a play time period of each piece of text data in the first language in the video;

[0177] a video segmentation subunit, configured to cut a portion of the video located after a current playback point in the video into a plurality of video segments according to the playback time period corresponding to each piece of text data in the first language;

[0178] a recording subunit, configured to record a correspondence between each of the video segments and the text data in the first language;

[0179] The synthesis subunit is used to determine the human voice data of the second language corresponding to each of the video segments according to the corresponding relationship, and synthesize the video frame data in each of the video segments with the human voice data of the second language corresponding to the video segment to obtain a converted video.

[0180] In a possible implementation, the synthesis subunit is specifically used for:

[0181] The following processing is performed for each of the video segments to obtain a converted video:

[0182] Determine a first playback duration of the video frame data in the video segment, and determine a second playback duration of the human voice data in a second language corresponding to the video segment;

[0183] When the first play time is inconsistent with the second play time, adjusting the play time of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent;

[0184] The video frame data in the video segment adjusted to have a consistent playback duration is synthesized with the corresponding second language human voice data.

[0185] In a possible implementation, the synthesis subunit includes:

[0186] A first alignment subunit, configured to determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0187] According to the extension multiple, the video frame data in the video segment is cyclically spliced ​​to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0188] In a possible implementation, the synthesis subunit includes:

[0189] A second alignment subunit, configured to determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration;

[0190] According to the extension multiple, the video frame data in the video segment is speed-changed to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

[0191] like Figure 5 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0192] Memory 113, used for storing computer programs;

[0193] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement the video processing method provided by any of the above method embodiments, including:

[0194] During video playback, in response to a language conversion instruction, converting human voice data of a first language in a portion of the video located after a current playback point in the video into human voice data of a second language to obtain a converted video;

[0195] Continue playing the converted video from the current playback point.

[0196] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video processing method provided in any of the aforementioned method embodiments are implemented.

[0197] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0199] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0200] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A video processing method, characterized in that: The method comprises: During video playback, in response to a language conversion instruction, converting human voice data of a first language in a portion of the video located after a current playback point in the video into human voice data of a second language to obtain a converted video; Continue playing the converted video from the current playback point.

2. The method according to claim 1, characterized in that The method further comprises: During the process of playing the converted video, in response to the original sound recovery instruction, the original video continues to be played from the current playback point.

3. The method according to claim 1, characterized in that After converting the human voice data of the first language in the portion of the video located after the current playback point in the video into human voice data of the second language, the method further includes: Adjusting the converted video; The step of continuing to play the converted video from the current play point is performed using the adjusted converted video.

4. The method according to claim 3, characterized in that The step of adjusting the converted video includes: Compounding the timbre data in the first language vocal data with the second language vocal data in the converted video; and / or, According to the human voice data of the second language, the vocal action of the target sound-making object in the video frame data in the converted video is matched.

5. The method according to claim 1, characterized in that The step of converting the human voice data of the first language in the part of the video located after the current playback point in the video into the human voice data of the second language to obtain the converted video includes: Acquire text data in the first language of a portion of the video located after a current playback point in the video; translating the text data in the first language into text data in a second language; generating voice data in the second language based on the text data in the second language; The human voice data of the second language and the video frame data of the portion of the video located after the current playback point in the video are synthesized to obtain a converted video.

6. The method according to claim 5, characterized in that The step of obtaining text data in the first language of a portion of the video located after the current playback point in the video includes: Separating audio data from a portion of the video located after a current playback point in the video, and separating human voice data of a first language from the audio data; Speech-to-text recognition is performed on the human voice data in the first language to obtain text data in the first language of a portion of the video located after a current playback point in the video.

7. The method according to claim 5, characterized in that The synthesizing process of the human voice data of the second language and the video frame data of the part of the video located after the current playback point in the video to obtain the converted video includes: Determine a playback time period of each piece of text data in the first language in the video; According to the playback time period corresponding to each piece of text data in the first language, cutting a portion of the video located after a current playback point in the video into a plurality of video segments, and recording a corresponding relationship between each of the video segments and the text data in the first language; According to the corresponding relationship, the human voice data of the second language corresponding to each of the video segments is determined, and the video frame data in each of the video segments is synthesized with the human voice data of the second language corresponding to the video segment to obtain a converted video.

8. The method according to claim 7, characterized in that The synthesizing the video frame data in each of the video segments with the human voice data of the second language corresponding to the video segment to obtain a converted video includes: The following processing is performed for each of the video segments to obtain a converted video: Determine a first playback duration of the video frame data in the video segment, and determine a second playback duration of the human voice data in a second language corresponding to the video segment; When the first play time is inconsistent with the second play time, adjusting the play time of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent; The video frame data in the video segment adjusted to have a consistent playback duration is synthesized with the corresponding second language human voice data.

9. The method according to claim 8, characterized in that The step of adjusting the playing duration of the video frame data in the video segment and the playing duration of the second language human voice data corresponding to the video segment to be consistent includes: Determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration; According to the extension multiple, the video frame data in the video segment is cyclically spliced ​​to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

10. The method according to claim 8, characterized in that The step of adjusting the playing duration of the video frame data in the video segment and the playing duration of the second language human voice data corresponding to the video segment to be consistent includes: Determine an extension multiple of the video playback duration according to the second playback duration and the first playback duration; According to the extension multiple, the video frame data in the video segment is speed-changed to adjust the playing duration of the video frame data in the video segment and the second language human voice data corresponding to the video segment to be consistent.

11. A video processing device, characterized in that: The device comprises: a conversion module, configured to, during video playback, in response to a language conversion instruction, convert human voice data of a first language in a portion of the video located after a current playback point in the video into human voice data of a second language to obtain a converted video; The video playing module is used to continue playing the converted video from the current playing point.

12. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the video processing method according to any one of claims 1 to 10.

Images