Film and television dubbing method, terminal and storage medium

Abstract

The present invention discloses a film and television dubbing method, terminal, and storage medium, comprising: entering a film and television dubbing editing mode according to a user operation instruction; selecting a video clip to be edited, determining the audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip; performing dubbing processing on the separated video clip to obtain a dubbed video clip, and replacing the dubbed video clip with the position corresponding to the original video clip; splicing all replaced video clips according to an audio splicing operation to obtain a new dubbed video. The present invention distinguishes the audio of different characters through voiceprint recognition, and realizes the operation of audio editing of the audio in the original video according to the character, which is more in line with the user's usage logic, better helps users to sort out creative ideas and stimulates the user's interest in secondary creation.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a film and television dubbing method, a terminal and a storage medium. Background Art

[0002] With the development of internet technology, users have access to a wide variety of media data. Video, as a common data format, is widely used in our daily lives. More and more users are uploading their own videos online for others to watch. To improve the quality of the dialogue in their videos, users often perform post-production dubbing.

[0003] Currently, there are many software on the market that provide the function of directly dubbing videos, so that the videos can better meet the personalized needs of users. Users usually use manual dubbing to post-dub their self-made videos.

[0004] However, existing TV dubbing methods can only combine the user's voiceover with background sound, but cannot support editing the audio in the original video. Furthermore, existing technologies cannot provide users with real-time interaction with film and television works, failing to stimulate their interest in secondary creation.

[0005] Therefore, the existing technology needs to be improved. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that, in response to the defects of the existing technology, the present invention provides a film and television dubbing method, terminal and storage medium to solve the technical problem that the current existing technology cannot provide users with the function of interacting with film and television works in real time and cannot stimulate users' interest in secondary creation.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] In a first aspect, the present invention provides a method for dubbing a film or television, comprising:

[0009] Enter the film and television dubbing editing mode according to user operation instructions;

[0010] Selecting a video clip to be edited, determining an audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip;

[0011] Performing dubbing on the separated video segments to obtain dubbed video segments, and replacing the dubbed video segments with corresponding positions of the original video segments;

[0012] All replaced video clips are spliced ​​together according to the audio splicing operation to obtain a newly dubbed video.

[0013] In one implementation of the present invention, determining the audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain the separated video clip includes:

[0014] Determining whether the audio track type of the video clip to be edited is multi-track or mixed track;

[0015] If the audio track type is the multi-track type, use the multimedia framework to perform audio separation to obtain the separated video clips;

[0016] If the audio track type is the mixed audio track, AI technology is used to perform audio separation processing on the video clip to be edited to obtain the separated video clip.

[0017] In one implementation of the present invention, the step of using a multimedia framework to perform audio separation to obtain the separated video clips includes:

[0018] Separating the audio track and the visual track of the video clip to be edited by a multimedia separation acquirer to obtain corresponding information and buffer data;

[0019] The buffered data is decoded using a multimedia decoding tool.

[0020] In one implementation of the present invention, performing dubbing on the separated video segments to obtain dubbed video segments includes:

[0021] The separated video segments are dubbed according to the audio sequence replacement operation and the audio masking operation to obtain dubbed video segments.

[0022] In one implementation of the present invention, the dubbing process of the separated video segments according to the audio sequence replacement operation and the audio masking operation includes:

[0023] shielding the original audio in the separated video clip according to the audio shielding operation, and performing a dubbing operation on the shielded original audio;

[0024] The separated video segments are sorted according to the audio sequence replacement operation.

[0025] In one implementation of the present invention, the dubbing process of the separated video segments according to the audio sequence replacement operation and the audio masking operation further includes:

[0026] Subtitles and time reminder functions are provided during the dubbing process.

[0027] In one implementation of the present invention, the step of splicing all replaced video clips according to the audio splicing operation to obtain a newly dubbed video includes:

[0028] All replaced video clips are spliced ​​together according to the editing order to obtain the newly dubbed video;

[0029] The newly dubbed video is stored locally in the user according to a preset storage path.

[0030] In one implementation of the present invention, the step of splicing all replaced video clips according to the audio splicing operation to obtain a newly dubbed video may further include:

[0031] Determine whether the current mode is the film and television dubbing editing mode;

[0032] If it is the video dubbing editing mode, then play the newly dubbed video;

[0033] If it is not the film and television dubbing editing mode, the original video is played.

[0034] In a second aspect, the present invention further provides a terminal comprising: a processor and a memory, wherein the memory stores an execution program for film and television dubbing, and when the execution program for film and television dubbing is executed by the processor, it is used to implement the steps of the film and television dubbing method as described in the first aspect.

[0035] In a third aspect, the present invention further provides a storage medium, which is a computer-readable storage medium, and which stores an execution program for film and television dubbing. When the execution program for film and television dubbing is executed by a processor, it is used to implement the steps of the film and television dubbing method as described in the first aspect.

[0036] The present invention adopts the above technical solution to achieve the following effects:

[0037] The present invention discloses a film and television dubbing method, terminal, and storage medium, comprising: entering a film and television dubbing editing mode according to a user operation instruction; selecting a video clip to be edited, determining the audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip; performing dubbing processing on the separated video clip to obtain a dubbed video clip, and replacing the dubbed video clip with the position corresponding to the original video clip; splicing all replaced video clips according to an audio splicing operation to obtain a new dubbed video. The present invention distinguishes the audio of different characters through voiceprint recognition, and realizes the operation of audio editing of the audio in the original video according to the character, which is more in line with the user's usage logic, better helps users to sort out creative ideas and stimulates the user's interest in secondary creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of film and television dubbing in one implementation of the present invention.

[0040] Figure 2 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0041] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0042] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0044] Exemplary Methods

[0045] With the development of internet technology, users have access to a wide variety of media data. Video, as a common data format, is widely used in our daily lives. More and more users are uploading their own videos online for others to watch. To improve the quality of the dialogue in their videos, users often perform post-production dubbing.

[0046] Currently, there are many software on the market that provide the function of directly dubbing videos, so that the videos can better meet the personalized needs of users. Users usually use manual dubbing to post-dub their self-made videos.

[0047] However, the existing TV dubbing method can only support the combination of the user's dubbing and background sound, and cannot support the editing of the audio in the original video, and cannot provide a film and television dubbing function for editing the audio in the original video.

[0048] In response to the above technical problems, an embodiment of the present invention provides a film and television dubbing method, which distinguishes the audio of different characters through voiceprint recognition, and realizes the operation of audio editing of the audio in the original video according to the character. The present invention distinguishes the audio of different characters through voiceprint recognition, and realizes the operation of audio editing according to the character. At the same time, the configured audio will be stored in a local specified location. When the original video is played in editing mode, the configured audio is read when the video is loaded to realize the real-time playback function. This invention can give users more playability and stimulate users' secondary creation. At the same time, performing audio editing operations on the TV side is more in line with the user's usage logic. When you see the film and television content that interests you, you can directly perform secondary creation.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for dubbing a film and television, comprising the following steps:

[0050] Step S100: Entering the film and television dubbing editing mode according to the user's operation instruction.

[0051] In this embodiment, in order to achieve the effect of editing film and television dubbing, it is necessary to enter the dubbing editing mode, and process it separately according to the type of audio track of the film and television dubbing. The user needs to send a request editing instruction to enter the film and television dubbing editing mode. This embodiment provides two modes: dubbing editing mode and playback mode, and the user can switch between the two modes at any time. In other words, when the user is watching the film and television content of interest, as long as there is an operation demand, he can directly switch to the dubbing editing mode. The advantage of this is that flexible mode switching can greatly improve the user experience and also allow users to record their creative inspiration at the first time.

[0052] like Figure 1As shown, in one implementation of the embodiment of the present invention, the film and television dubbing method further includes the following steps:

[0053] Step S200 : selecting a video clip to be edited, determining the audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip.

[0054] In this embodiment, after selecting the video clip to be edited, the system will separate the human voice audio and the background sound audio, and separate the audio of different characters through voiceprint recognition. The user selects the film and television video clip to be edited. At this time, the audio track type of the video clip selected by the user is judged, and different processing schemes are set according to different audio track types. In this embodiment, the audio track types are roughly divided into two types: multi-audio track type and mixed audio track type, and audio separation operations are performed on them.

[0055] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0056] Step S201 : determining whether the audio track type of the video clip to be edited is multiple audio tracks or mixed audio tracks.

[0057] Step S202: If the audio track type is the multi-track audio track, the multimedia framework is used to perform audio separation to obtain the separated video clips.

[0058] Specifically, in one implementation of this embodiment, step S202 includes the following steps:

[0059] Step S202a: Separate the audio track and the video track of the video clip to be edited through a multimedia separation acquirer to obtain corresponding information and buffer data.

[0060] In this example, the multimedia framework uses MediaExtractor as a multimedia separation and acquisition tool. MediaExtractor acts as a parser within the multimedia framework, parsing file encapsulation. MediaExtractor parses video files into audio and video streams, and audio files into audio streams, thereby separating the audio and video tracks of a film and television program and obtaining the required information and buffered data.

[0061] Step S202b: decoding the buffered data using a multimedia decoding tool.

[0062] In this embodiment, a multimedia framework is used and Mediacodec is used as a multimedia decoding tool. Mediacodec is a codec component in the Android system that is used to access the codec provided by the underlying layer. It is usually used in conjunction with MediaExtractor, AudioTrack, etc. Mediacodec first obtains an empty input buffer, fills it with data to be encoded or decoded, and then sends the input buffer filled with data to Mediacodec for processing. After processing the data, the input buffer filled with data is released. Finally, the encoded or decoded output buffer is obtained and released after use, thereby completing the decoding and encoding operations of the buffer data, combining it with the buffer recorded using AudioRecord into a PCM audio file, and playing it using AudioTrack, where AudioTrack can play the decoded PCM stream. In this embodiment, AudioTrack is used for playback, but sound can be played using MediaPlayer and AudioTrack or other players, and is not limited to using AudioTrack for playback.

[0063] Step S203: If the audio track type is the mixed audio track, AI technology is used to perform audio separation processing on the video clip to be edited to obtain the separated video clip.

[0064] At present, the application of AI technology in the processing of mixed audio tracks has become relatively mature. There are many AI artificial intelligence software on the market that have rich experience in mixing applications in various technical fields. AI sound separation is a technology that can remove unnecessary noise from audio data and extract only the sound of human voices or other specific instruments. For film and television works with mixed audio tracks, this embodiment will use AI technology to intelligently separate the human voice, background sound and different human voices in the mixed audio track into corresponding files through training and learning, edit the corresponding files according to the user's operation, and then encode and synthesize them according to the protocol required for TV broadcasting to form a new audio track for playback.

[0065] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the film and television dubbing method further includes the following steps:

[0066] Step S300 , dubbing is performed on the separated video segments to obtain dubbed video segments, and the dubbed video segments are replaced with corresponding positions of the original video segments.

[0067] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0068] Step S301 : dubbing the separated video clips according to the audio sequence replacement operation and the audio masking operation to obtain dubbed video clips.

[0069] When watching a video, if a user wants to participate in the dubbing of the video and realize their own dialogue with the characters in the video, but does not want to be limited to the original background sound in the video and wants to change the background sound for secondary creation, the user can operate the audio track according to the method in this embodiment and selectively splice it. For example, if the user wants to realize a dialogue with a character in a film or television work in a scene, or the user wants to realize the dialogue between the characters in the film or television work in different scenes, or the user wants to realize the audio of the characters in the film or television work to be interchangeable, then the original video audio needs to be processed accordingly.

[0070] Users can change the original video and audio as needed, and can change the order or block the audio of different characters as needed. In this embodiment, users can directly divide the original video into blocks based on the film and television characters, then block the video and audio of the corresponding modules by character, and then use the user-configured film and television clips to overwrite the original clips to achieve video and audio configuration operations. Alternatively, users can select a clip from the original video, block the audio in the original video clip, and combine it with the locally uploaded configured audio to complete the editing operation, achieving the effect of secondary creation of the user's dubbing for the character.

[0071] Specifically, in one implementation of this embodiment, step S301 includes the following steps:

[0072] Step S301a: shielding the original audio in the separated video segment according to the audio shielding operation, and performing a dubbing operation on the shielded original audio.

[0073] In this embodiment, the selected video clips that have undergone audio separation are segmented. Users can change the order of the selected clips according to their needs or preferences to change the original content of the video and achieve the effect of secondary creation. At the same time, they can also perform a masking operation, selecting a specific clip for masking, and hiding the content of the clip in the original video audio.

[0074] When the user chooses to dub by themselves, the voice driver is started to collect the user's audio to complete the dubbing. The completed dubbing will replace the content in the original video audio.

[0075] Step S301b: sort the separated video segments according to the audio sequence changing operation.

[0076] Users can replace the selected masked audio or video clips with other prepared audio or video files or self-recorded audio or video. If users want to have a conversation with a character in a scene, or want to have characters in a film or TV show have a conversation in different scenes, or want to swap the audio of characters in a film or TV show, they can achieve the above effects by simply selecting the original audio clip and replacing it.

[0077] Step S301c: providing subtitles and time reminder functions during the dubbing process.

[0078] In dubbing mode, in order to enable users to better create video and audio content that meets their original intentions based on their own needs, a subtitle function is set in this embodiment. Users can understand the approximate content of a certain segment in the video and audio based on the subtitles. By combining the subtitle text with the video and audio, the content can be clearer, avoiding problems such as difficulty in understanding caused by homophones or pronunciation problems. Providing a subtitle function can also better help users sort out their creative ideas and assist users in secondary creation.

[0079] This embodiment also provides a time reminder function. With the help of this function, users can better control the speaking speed and the length of the video clip to be operated when editing video audio content and dubbing, which better assists users in creation.

[0080] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the film and television dubbing method further includes the following steps:

[0081] Step S400: splice all replaced video clips together according to the audio splicing operation to obtain a new dubbed video.

[0082] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0083] Step S401: splice all replaced video clips according to the editing order to obtain a new dubbed video.

[0084] After repeating the dubbing process, the video and audio are combined to form a new work, completing the dubbing process. The user-configured audio is stored locally. When playing the video in dubbing mode, the newly edited audio will play. After exiting editing mode, the original audio will play again.

[0085] Step S402: The newly dubbed video is stored locally in the user according to a preset storage path.

[0086] In this embodiment, the video and audio are stored in a local preset storage path instead of directly replacing the original video, which does not affect the user's operation of dubbing the original video and audio again. By controlling the editing and playback of video and audio in two modes, users can use the dubbing and video and audio functions more conveniently, thereby improving the user experience.

[0087] Specifically, in one implementation of this embodiment, the following steps are further included after step S400:

[0088] Step S501, determining whether the current mode is the film and television dubbing editing mode.

[0089] Step S502: If it is the film and television dubbing editing mode, play the newly dubbed video.

[0090] After entering the dubbing mode, users can process and modify the film and television works according to their needs. If they remain in the dubbing editing mode, when playing the video and audio, what is played is the latest film and television work obtained by the user after editing operations.

[0091] Step S503: If the video dubbing editing mode is not selected, the original video is played.

[0092] After the user completes editing a video, the latest video is saved in a local directory. At this point, the user exits dubbing editing mode and plays the video and audio, using the original audio. This embodiment prioritizes preserving the user's creative inspiration. The edited video is played only in dubbing editing mode. Dubbing editing does not affect normal video playback; upon exiting dubbing editing mode, the original video resumes.

[0093] This embodiment achieves the following technical effects through the above technical solution:

[0094] This invention uses voiceprint recognition to distinguish the audio of different characters, enabling audio editing of the original video based on the character. Providing a film and television dubbing function that allows editing of the audio in the original video is more in line with user logic, better helping users clarify their creative ideas, stimulating their interest in secondary creation and assisting them in better secondary creation.

[0095] Exemplary devices

[0096] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein the processor is used to provide computing and control capabilities; the memory includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and computer program in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0097] When the computer program is executed by the processor, it is used to implement the steps of a film and television dubbing method.

[0098] It will be understood by those skilled in the art that Figure 2 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0099] In one embodiment, a terminal is provided, comprising: a processor and a memory, wherein the memory stores an execution program for film and television dubbing, and when the execution program for film and television dubbing is executed by the processor, it is used to implement the steps of the film and television dubbing method described above.

[0100] In one embodiment, a storage medium is provided, wherein the storage medium stores an execution program for film and television dubbing, and when the execution program for film and television dubbing is executed by the processor, it is used to implement the steps of the film and television dubbing method described above.

[0101] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory.

[0102] In summary, the present invention discloses a film and television dubbing method, terminal and storage medium, including: entering a film and television dubbing editing mode according to a user operation instruction; selecting a video clip to be edited, determining the audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip; performing dubbing processing on the separated video clip to obtain a dubbed video clip, and replacing the dubbed video clip to the position corresponding to the original video clip; splicing all replaced video clips according to an audio splicing operation to obtain a new dubbed video. The present invention distinguishes the audio of different characters through voiceprint recognition, and realizes the operation of audio editing of the audio in the original video according to the character, which is more in line with the user's usage logic, better helps users sort out creative ideas and stimulates users' interest in secondary creation.

[0103] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A film and television dubbing method, characterized in that: include: Enter the film and television dubbing editing mode according to user operation instructions; Selecting a video clip to be edited, determining an audio track type of the video clip to be edited, and performing audio separation processing on the video clip to be edited according to the audio track type to obtain a separated video clip; The step of determining the audio track type of the video clip to be edited and performing audio separation processing on the video clip to be edited according to the audio track type to obtain the separated video clip includes: Separate human voice audio from background audio, and separate different character audio through voiceprint recognition, setting different processing solutions according to different audio track types; Determining whether the audio track type of the video clip to be edited is multi-track or mixed track; If the audio track type is the multi-track type, use the multimedia framework to perform audio separation to obtain the separated video clips; If the audio track type is the mixed audio track, performing audio separation processing on the video clip to be edited using AI technology to obtain the separated video clip; Performing dubbing on the separated video segments to obtain dubbed video segments, and replacing the dubbed video segments with corresponding positions of the original video segments; The step of performing dubbing on the separated video segments to obtain dubbed video segments includes: performing dubbing processing on the separated video clips according to the audio sequence replacement operation and the audio masking operation to obtain dubbed video clips; The original video is divided into blocks according to the film and television roles, and the video and audio of the corresponding module are shielded by the role. The original clips are covered by the user-configured film and television clips to realize the configuration operation of video and audio; All replaced video clips are spliced ​​together according to the audio splicing operation to obtain a newly dubbed video.

2. The film and television dubbing method according to claim 1, characterized in that: The method of using a multimedia framework to separate the audio and obtain the separated video clips includes: Separating the audio track and the visual track of the video clip to be edited by a multimedia separation acquirer to obtain corresponding information and buffer data; The buffered data is decoded using a multimedia decoding tool.

3. The film and television dubbing method according to claim 1, characterized in that: The dubbing process of the separated video segments according to the audio sequence replacement operation and the audio masking operation includes: shielding the original audio in the separated video clip according to the audio shielding operation, and performing a dubbing operation on the shielded original audio; The separated video segments are sorted according to the audio sequence replacement operation.

4. The film and television dubbing method according to claim 1, characterized in that: The dubbing process of the separated video segments according to the audio sequence replacement operation and the audio shielding operation further includes: Subtitles and time reminder functions are provided during the dubbing process.

5. The film and television dubbing method according to claim 1, characterized in that: The step of splicing all replaced video clips according to the audio splicing operation to obtain a newly dubbed video includes: All replaced video clips are spliced ​​together according to the editing order to obtain the newly dubbed video; The newly dubbed video is stored locally in the user according to a preset storage path.

6. The film and television dubbing method according to claim 1, characterized in that: The method splices all replaced video clips according to the audio splicing operation to obtain a new dubbed video, and then includes: Determine whether the current mode is the film and television dubbing editing mode; If it is the video dubbing editing mode, then play the newly dubbed video; If it is not the film and television dubbing editing mode, the original video is played.

7. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores an execution program for film and television dubbing, and when the execution program for film and television dubbing is executed by the processor, it is used to implement the steps of the film and television dubbing method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an execution program for film and television dubbing, and when the execution program for film and television dubbing is executed by a processor, it is used to implement the steps of the film and television dubbing method as described in any one of claims 1 to 6.

Images