Video processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a video processing method and apparatus, an electronic device, and a storage medium. The method comprises: displaying text data to be processed corresponding to a video to be processed, wherein said video comprises a media video and media audio, a first mapping relationship is present between text time code information corresponding to said text data and first timeline information corresponding to the media video, and a second mapping relationship is present between the text time code information and second timeline information corresponding to the media audio; in response to a first trigger operation for target text data in said text data, on the basis of the first mapping relationship, determining from the media video a target video clip corresponding to the target text data; on the basis of the second mapping relationship, determining from the media audio a target audio clip corresponding to the target text data; and on the basis of the first trigger operation, performing corresponding processing on the target text data, the target video clip, and the target audio clip.

Description

Video processing method, device, electronic device, and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 31, 2023, with application number 202311120475.X and invention name “Video processing method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of video technology, and in particular to a video processing method, device, electronic device, storage medium, and program product. Background Art

[0003] With the increasing abundance of information on the Internet, watching videos has become an entertainment activity in people's daily lives. In order to improve users' viewing experience, video editing is an important step before publishing various videos.

[0004] When creating, creators can use video editing tools to edit the filmed video footage. Currently, most video editing tools focus on editing video frames, including dragging, sorting, switching materials, adding subtitles, and other operations to complete the video editing process. However, editing video frames requires creators to have certain video editing knowledge, and creators cannot complete the video editing work quickly. Therefore, editing video frames is relatively complex and time-consuming, affecting the creator's user experience.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a video processing method, device, electronic device, storage medium and program product.

[0007] A first aspect of an embodiment of the present disclosure provides a video processing method, which includes: displaying text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and text time code information corresponding to the text data to be processed has a first mapping relationship with first timeline information corresponding to the media video; a second mapping relationship exists between the text time code information and second timeline information corresponding to the media audio; in response to a first trigger operation for target text data in the text data to be processed, based on the first mapping relationship, determining a target video segment corresponding to the target text data from the media video; based on the second mapping relationship, determining a target audio segment corresponding to the target text data from the media audio; and based on the first trigger operation, performing corresponding processing on the target text data, the target video segment and the target audio segment.

[0008] In some embodiments of the present application, the video to be processed also includes at least one packaging object, and there is a mapping relationship between the text time code information and the timeline information corresponding to each packaging object; wherein the object type of each packaging object includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packages, and background music.

[0009] In some embodiments of the present application, the first trigger operation includes any one of the following: a delete operation, a drag operation, a copy operation, and a cut operation; the method also includes: in response to the first trigger operation on the target text data in the text data to be processed, determining the target packaging object corresponding to the target text data from at least one packaging object; based on the first trigger operation, when the object type of the target packaging object is any one of the first types, performing corresponding processing on the target packaging object; when the object type of the target packaging object is any one of the second types, prohibiting corresponding processing on the target packaging object; wherein, the first type includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages; the second type includes: background music.

[0010] In some embodiments of the present application, the first type includes a third type and a fourth type; when the object type of the target packaging object is any one of the first types, the target packaging object is processed accordingly, including: when the object type of the target packaging object is any one of the third types, the target packaging fragment in the target packaging object is processed accordingly, and the target packaging fragment is the part of the target packaging object corresponding to the target text data; or, when the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is within the time code range corresponding to the target text data, the target packaging object as a whole is processed accordingly; wherein the third type includes at least one of the following: subtitle text, picture-in-picture; the fourth type includes at least one of the following: non-subtitle text, stickers, special effects, filters, and material packages.

[0011] In some embodiments of the present application, the method further includes: prohibiting corresponding processing of the target packaging object when the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is not within the time code range corresponding to the target text data.

[0012] In some embodiments of the present application, the method also includes: in response to a modification operation on first text data in the text data to be processed, obtaining modified second text data corresponding to the modification operation; based on a second mapping relationship, determining a first audio segment corresponding to the first text data from the media audio; generating a second audio segment based on the second text data and the timbre information of the first audio segment; and replacing the first audio segment in the media audio with the second audio segment.

[0013] In some embodiments of the present application, the video to be processed also includes subtitle text, and a third mapping relationship exists between the text time code information and the third timeline information corresponding to the subtitle text; the method also includes: in response to a modification operation on the first text data in the text data to be processed, based on the third mapping relationship, determining a first subtitle segment corresponding to the first text data from the subtitle text; based on the second text data, modifying the first subtitle segment to obtain a second subtitle segment; and replacing the first subtitle segment in the subtitle text with the second subtitle segment.

[0014] In some embodiments of the present application, the first trigger operation also includes a selection operation; in response to the first trigger operation for the target text data in the text data to be processed, before determining the target video segment corresponding to the target text data from the media video based on the first mapping relationship, the method also includes: displaying the first timeline track and the second timeline track, the first timeline track being the timeline track corresponding to the media video, and the second timeline track being the timeline track corresponding to the media audio; based on the first trigger operation, performing corresponding processing on the target text data, the target video segment and the target audio segment, including: displaying the target text data according to the first display style; displaying the part of the first timeline track corresponding to the target video segment according to the second display style; and displaying the part of the second timeline track corresponding to the target audio segment according to the third display style.

[0015] In some embodiments of the present application, the method also includes: in response to a second trigger operation on selected text data in the text data to be processed, displaying a tool menu in the area corresponding to the selected text data, the tool menu including at least one of the following categories: clipping controls and adding controls, the clipping controls including at least one of the following: a delete control corresponding to a delete operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut control, a modification control corresponding to a modification operation, an intelligent long and short sentence control, a search control, and a replacement control; the adding controls including at least one of the following: adding a picture-in-picture control, adding a non-subtitle text control, adding a sticker control, adding a special effect control, adding a sound effect control, adding a filter control, adding a material package control, and adding a background music control; in response to a third trigger operation on a target control in the tool menu, performing corresponding processing on at least one of the media video, the text data to be processed, and the media audio.

[0016] In some embodiments of the present application, after displaying the text data to be processed corresponding to the video to be processed, the method further includes: in response to a trigger operation on a material library control, displaying a plurality of preset video materials; in response to a fifth trigger operation on a target video material among the plurality of preset video materials, displaying a text rough-cut floating window, the text rough-cut floating window including material text data corresponding to the target video material, the target video material including a target material video and a target material audio, the material time code information of the material text data and the timeline information of the target material video having a fourth mapping relationship, and the material time code information and the timeline information of the target material audio having a fifth mapping relationship; in response to a selected pixel in the material text data The text rough cutting operation of the material text synthesizes the selected material text with the text data to be processed to obtain the text data to be edited, synthesizes the material video clip and the media video into the media video to be edited, and synthesizes the material audio clip and the media audio into the media audio to be edited, so that the text editing operation based on the text data to be edited can realize the editing of the video to be edited; wherein, the material video clip is a video clip corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship; the material audio clip is an audio clip corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship; the video to be edited includes the media video to be edited and the media audio to be edited.

[0017] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, which includes: a display module for displaying text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first timeline information corresponding to the media video; the text time code information has a second mapping relationship with the second timeline information corresponding to the media audio; a determination module for determining, in response to a first trigger operation for target text data in the text data to be processed, a target video segment corresponding to the target text data from the media video based on the first mapping relationship; the determination module is also used to determine, from the media audio based on the second mapping relationship, a target audio segment corresponding to the target text data; and a processing module for performing corresponding processing on the target text data, the target video segment and the target audio segment based on the first trigger operation.

[0018] In some embodiments of the present application, the video to be processed also includes at least one packaging object, and there is a mapping relationship between the text time code information and the timeline information corresponding to each packaging object; wherein the object type of each packaging object includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packages, and background music.

[0019] In some embodiments of the present application, the first trigger operation includes any one of the following: a delete operation, a drag operation, a copy operation, and a cut operation; the determination module is also used to determine, in response to the first trigger operation on the target text data in the text data to be processed, a target packaging object corresponding to the target text data from at least one packaging object; the processing module is also used to, based on the first trigger operation, perform corresponding processing on the target packaging object when the object type of the target packaging object is any one of the first types; and prohibit corresponding processing on the target packaging object when the object type of the target packaging object is any one of the second types; wherein the first type includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages; the second type includes: background music.

[0020] In some embodiments of the present application, the first type includes a third type and a fourth type; the processing module is specifically used to perform corresponding processing on a target packaging segment in the target packaging object when the object type of the target packaging object is any one of the third types, and the target packaging segment is the part of the target packaging object corresponding to the target text data; or, when the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is within the time code range corresponding to the target text data, perform corresponding processing on the entire target packaging object; wherein the third type includes at least one of the following: subtitle text, picture-in-picture; the fourth type includes at least one of the following: non-subtitle text, stickers, special effects, filters, and material packages.

[0021] In some embodiments of the present application, the processing module is also used to prohibit corresponding processing of the target packaging object when the object type of the target packaging object is any of the fourth types and the start timestamp of the target packaging object is not within the time code range corresponding to the target text data.

[0022] In some embodiments of the present application, the device also includes: an acquisition module, a generation module and a replacement module, the acquisition module is used to respond to a modification operation on the first text data in the text data to be processed, and obtain the modified second text data corresponding to the modification operation; the determination module is also used to determine the first audio segment corresponding to the first text data from the media audio based on the second mapping relationship; the generation module is used to generate a second audio segment based on the second text data and the timbre information of the first audio segment; the replacement module is used to replace the first audio segment in the media audio with the second audio segment.

[0023] In some embodiments of the present application, the video to be processed also includes subtitle text, and a third mapping relationship exists between the text time code information and the third timeline information corresponding to the subtitle text; the device also includes: a modification module and a replacement module; the determination module is also used to respond to a modification operation on the first text data in the text data to be processed, and based on the third mapping relationship, determine the first subtitle segment corresponding to the first text data from the subtitle text; the modification module is used to modify the first subtitle segment based on the second text data to obtain a second subtitle segment; the replacement module is used to replace the first subtitle segment in the subtitle text with the second subtitle segment.

[0024] In some embodiments of the present application, the first trigger operation also includes a selection operation; the display module is also used to display the first timeline track and the second timeline track before determining the target video segment corresponding to the target text data from the media video based on the first mapping relationship in response to the first trigger operation for the target text data in the text data to be processed, the first timeline track is the timeline track corresponding to the media video, and the second timeline track is the timeline track corresponding to the media audio; the processing module is specifically used to display the target text data according to the first display style; display the part of the first timeline track corresponding to the target video segment according to the second display style; and display the part of the second timeline track corresponding to the target audio segment according to the third display style.

[0025] In some embodiments of the present application, the display module is also used to display a tool menu in the area corresponding to the selected text data in response to a second trigger operation on the selected text data in the text data to be processed, and the tool menu includes at least one of the following categories: clipping controls and adding controls. The clipping controls include at least one of the following: a delete control corresponding to the delete operation, a copy control corresponding to the copy operation, a cut control corresponding to the cut control, a modification control corresponding to the modification operation, an intelligent long and short sentence control, a search control, and a replacement control; the adding controls include at least one of the following: adding a picture-in-picture control, adding a non-subtitle text control, adding a sticker control, adding a special effect control, adding a sound effect control, adding a filter control, adding a material package control, and adding a background music control; the processing module is also used to perform corresponding processing on at least one of the media video, the text data to be processed, and the media audio in response to a third trigger operation on the target control in the tool menu.

[0026] In some embodiments of the present application, the display module is used to display multiple preset video materials in response to a trigger operation on a material library control after displaying the text data to be processed corresponding to the video to be processed; in response to a fifth trigger operation on a target video material among the multiple preset video materials, display a text rough cut floating window, the text rough cut floating window includes material text data corresponding to the target video material, the target video material includes a target material video and a target material audio, the material time code information of the material text data and the timeline information of the target material video have a fourth mapping relationship, and the material time code information and the timeline information of the target material audio have a fifth mapping relationship; in response to a selected pixel in the material text data The text rough cutting operation of the material text synthesizes the selected material text with the text data to be processed to obtain the text data to be edited, synthesizes the material video clip and the media video into the media video to be edited, and synthesizes the material audio clip and the media audio into the media audio to be edited, so that the text editing operation based on the text data to be edited can realize the editing of the video to be edited; wherein, the material video clip is a video clip corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship; the material audio clip is an audio clip corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship; the video to be edited includes the media video to be edited and the media audio to be edited.

[0027] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the video processing method as described in the first aspect.

[0028] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video processing method as described in the first aspect is implemented.

[0029] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program. When the computer program product runs on a processor, the processor executes the computer program to implement the video processing method as described in the first aspect.

[0030] According to a sixth aspect of the embodiments of the present disclosure, a chip is provided, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run program instructions to implement the video processing method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] FIG1 is a flow chart of a video processing method according to an embodiment of the present disclosure;

[0034] FIG2 is a second flow chart of a video processing method according to an embodiment of the present disclosure;

[0035] FIG3 is a schematic diagram of one interface of the video processing method provided by an embodiment of the present disclosure;

[0036] FIG4 is a second schematic diagram of an interface of a video processing method provided by an embodiment of the present disclosure;

[0037] FIG5 is a third schematic diagram of an interface of a video processing method provided by an embodiment of the present disclosure;

[0038] FIG6 is a structural block diagram of a video processing device provided by an embodiment of the present disclosure;

[0039] FIG7 is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0041] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0042] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects related to each other are in an "or" relationship.

[0043] The electronic devices in the embodiments of the present disclosure may be mobile electronic devices or non-mobile electronic devices. Mobile electronic devices may include mobile phones, tablet computers, laptop computers, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs); non-mobile electronic devices may include personal computers (PCs), televisions (TVs), ATMs, or self-service kiosks; and the embodiments of the present disclosure do not specifically limit these.

[0044] The execution subject of the video processing method provided in the embodiments of the present disclosure may be the above-mentioned electronic device (including mobile electronic devices and non-mobile electronic devices), or may be a functional module and / or functional entity in the electronic device that can implement the video processing method. The specific method may be determined based on actual usage requirements and is not limited by the embodiments of the present disclosure.

[0045] The video processing method provided by the embodiment of the present disclosure is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0046] As shown in FIG1 , an embodiment of the present disclosure provides a video processing method, which may include the following steps 101 to 104 .

[0047] 101. Display the text data to be processed corresponding to the video to be processed.

[0048] Among them, the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first timeline information corresponding to the media video; the text time code information has a second mapping relationship with the second timeline information corresponding to the media audio.

[0049] Among them, the text time code information includes the timestamp corresponding to each semantic character in the text data to be processed, and the first timeline information includes the timestamp of each video frame of the media video. The first mapping relationship is the correspondence between the timestamp of each semantic character in the text data to be processed and the timestamp of each video frame of the media video frame. Exemplarily, the timestamp of a semantic character in the text data to be processed can correspond to the timestamp of a video frame (that is, the timestamp of the semantic character in the text data to be processed corresponds to the timestamp of the video frame in the media video one by one), or the timestamp of a semantic character in the text data to be processed can correspond to the timestamps of multiple video frames.

[0050] Here, a semantic character refers to a character with a single semantic meaning. For example, in Chinese, a semantic character refers to a Chinese character, and in English, a semantic character refers to a word.

[0051] Among them, the second timeline information includes the timestamp of each audio frame in the media audio. The second mapping relationship is the correspondence between the timestamp of each semantic character in the text data to be processed and the timestamp of each audio frame of the media audio. Exemplarily, the timestamp of a semantic character in the text data to be processed can correspond to the timestamp of an audio frame (that is, the timestamp of the semantic character in the text data to be processed corresponds to the timestamp of the audio frame in the media audio one-to-one), or the timestamp of a semantic character in the text data to be processed can correspond to the timestamps of multiple audio frames.

[0052] It can be understood that based on the first mapping relationship, at least one video frame corresponding to any semantic character in the text data to be processed can be determined from the media video, or multiple video frames corresponding to any multiple semantic characters in the text data to be processed can be determined from the media video.

[0053] It can be understood that based on the second mapping relationship, at least one audio frame corresponding to any semantic character in the text data to be processed can be determined from the media audio, or multiple audio frames corresponding to any multiple semantic characters in the text data to be processed can be determined from the media audio.

[0054] In the embodiment of the present application, the video to be processed includes but is not limited to a recorded video, a video obtained based on a script, etc. The text data to be processed can be obtained by performing speech recognition on media audio or the video to be processed, or it can be a script. In the case where the text data to be processed is a script, the text data to be processed can be matched with the media video to obtain a first mapping relationship between the video timestamp of the text data to be processed and the media video, and the speech recognition method includes but is not limited to Automatic Speech Recognition (ASR) technology.

[0055] In some embodiments of the present application, when the text data to be processed is a script, the text data to be processed can be matched with the media audio to obtain a second mapping relationship between the text data to be processed and the audio timestamp of the media audio; the second mapping relationship can also be obtained based on the mapping relationship between the video timestamp of the media video and the audio timestamp of the media audio, as well as the first mapping relationship; the specific mapping relationship can be determined based on actual conditions and is not limited here.

[0056] In some embodiments of the present application, in response to a triggering operation on a text clipping control in an interface displaying a video to be processed, a text clipping panel can be called up, and the media video and media audio of the video to be processed can be extracted simultaneously. Then, speech recognition is performed on the media audio to obtain text data to be processed, and the text data to be processed is displayed in the text clipping panel, and a first mapping relationship and a second mapping relationship are obtained simultaneously.

[0057] In some embodiments of the present application, while the text clipping panel displays the text data to be processed, it can also display at least one of the video to be processed, the first timeline track corresponding to the media video, the second timeline track corresponding to the media audio, the text clipping menu, and the global function menu.

[0058] Among them, the text clipping menu can include at least one of the following: a delete control corresponding to the delete operation, a copy control corresponding to the copy operation, a cut control corresponding to the cut control, a modify control corresponding to the modify operation, an intelligent long and short sentence control (intelligent long and short sentence replacement control), a replace control, an add picture-in-picture control, an add non-subtitle text control, an add sticker control, an add special effect control, an add sound effect control, an add filter control, an add material package control, and an add background music control.

[0059] Among them, the global function menu includes at least one of the following: search control, watermark removal control, and subtitle switch control.

[0060] Among them, the dewatering control is used to indicate the dewatering process, which refers to removing invalid words or fragments such as meaningless words, modal particles, and silent fragments in the text data to be processed to ensure the semantic coherence of the text data to be processed after the dewatering process.

[0061] In some embodiments of the present application, the text clipping panel also includes an exit control, and the text clipping panel is collapsed (exited) in response to a triggering operation of the exit control.

[0062] In some embodiments of the present application, if ASR recognition is still being performed on the media audio when exiting the text clipping interface, the recognition is not interrupted and continues in the background.

[0063] In this embodiment, the text data to be processed can be displayed on the interface, and then the editing process of the video to be processed can be achieved by editing the text data to be processed.

[0064] 102 : In response to a first triggering operation on target text data in the text data to be processed, determine a target video segment corresponding to the target text data from the media video based on a first mapping relationship.

[0065] The first trigger operation can be a click operation, a slide operation, a drag operation, etc., which can be determined according to actual conditions and is not limited here.

[0066] It can be understood that the timestamp of the target video segment corresponds to the timestamp of the target text data.

[0067] 103. Based on the second mapping relationship, determine a target audio segment corresponding to the target text data from the media audio.

[0068] It can be understood that the timestamp of the target audio segment corresponds to the timestamp of the target text data.

[0069] 104. Based on the first trigger operation, perform corresponding processing on the target text data, the target video segment, and the target audio segment.

[0070] It can be understood that, based on the first trigger operation, processing corresponding to the first trigger operation is performed on the target text data, the target video segment, and the target audio segment.

[0071] In the embodiment of the present application, based on the first mapping relationship and the second mapping relationship, the first trigger operation is used to perform corresponding processing on the text data to be processed, thereby achieving corresponding processing on the media video and media audio, and then achieving editing of the video to be processed. Compared with the editing operation of video frames, the editing operation of text data is relatively simple and requires less professionalism from the creator, which can improve the editing efficiency of the video to be processed and enhance the creator's user experience.

[0072] In some embodiments of the present application, the video to be processed also includes at least one packaging object, and there is a mapping relationship between the text time code information and the timeline information corresponding to each packaging object; wherein the object type of each packaging object includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packages, and background music.

[0073] The packaging object can correspond to the complete text data to be processed (such as the packaging object is background music, subtitle text, picture-in-picture, etc.), or the packaging object can correspond to part of the text data in the text data to be processed (such as the packaging object is stickers, special effects, filters, etc.); the specific details can be determined based on actual conditions and are not limited here.

[0074] Taking packaging object 1 and packaging object 2 included in at least one packaging object as an example, there is a mapping relationship 1 between the text time code information and the time line information corresponding to packaging object 1, and there is a mapping relationship 2 between the text time code information and the time line information corresponding to packaging object 2.

[0075] The subtitle text is the text corresponding to the content of the video to be processed or the content of the media audio, and the non-subtitle text is the text other than the subtitle text.

[0076] Among them, picture-in-picture is a way of presenting video content. It means that while one video is broadcast in full screen, another video is broadcast on a small area of ​​the screen at the same time. It is widely used in television, video recording, monitoring, demonstration equipment, etc.

[0077] Among them, stickers can include text, pictures, images, etc.

[0078] Among them, special effects may include visual effects, sound effects (sound effects for short), etc.

[0079] Among them, filters are mainly used to achieve various special effects of images. For details, please refer to existing related technologies.

[0080] Among them, the material package is like a series of small templates that are combined together. It contains elements such as pictures, subtitles, special effects, etc. It supports one-click addition, and after adding, the elements can be changed as needed.

[0081] Background music, also known as accompaniment or soundtrack, generally refers to music used to set the mood in television dramas, movies, animations, video games, and websites. It is inserted into dialogues to enhance emotional expression and create an immersive experience for the audience. In the embodiments of this application, background music is audio content that is distinct from media audio and sound effects.

[0082] In an embodiment of the present application, the video to be processed includes at least one packaging object, and the timeline information of each packaging object is mapped to the text time code information of the text data to be processed. Therefore, in the process of performing a clipping operation on part of the text data to be processed, it may be necessary to perform corresponding processing on the packaging object that matches the part of the text data. The embodiment of the present application extracts the media video, media music, at least one packaging object, etc. in the video to be processed respectively, and then performs corresponding operations on at least one of the media video, media music, and at least one packaging object according to the clipping operation on the text data to be processed, so as to realize the clipping of the video to be processed. Compared with the clipping based on the text data to be processed, the video to be processed is clipped as a whole, so that the clipping of the video to be processed is more detailed and rational, and is simple and easy to understand without the need for user operation, which can improve the user experience.

[0083] In some embodiments of the present application, the first trigger operation includes any one of the following: a delete operation, a drag operation, a copy operation, and a cut operation; the first trigger operation may also include other operations, which are not limited here. In the embodiments of the present application, the first trigger operation can support multiple editing operations, which are simple and easy for users to understand, and can improve the user experience.

[0084] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: displaying the text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first timeline information corresponding to the media video; the text time code information has a second mapping relationship with the second timeline information corresponding to the media audio; in response to a first trigger operation for the target text data in the text data to be processed, based on the first mapping relationship, the target video segment corresponding to the target text data is determined from the media video; based on the second mapping relationship, the target audio segment corresponding to the target text data is determined from the media audio; based on the first trigger operation, the target text data, the target video segment and the target audio segment are processed accordingly. In the embodiment of the present application, based on the first mapping relationship and the second mapping relationship, the text data to be processed is processed accordingly through the first trigger operation, and the media video and media audio are processed accordingly, thereby realizing the editing of the video to be processed. Compared with the editing operation of the video frame, the editing operation of the text data is relatively simple and has lower professional requirements for the creator, which can improve the editing efficiency of the video to be processed and improve the user experience of the creator.

[0085] In some embodiments of the present application, in combination with FIG1 , as shown in FIG2 , after the above step 101 , the video processing method provided in the embodiment of the present application may further include the following steps 105 to 107 .

[0086] 105 . In response to a first triggering operation on target text data in the text data to be processed, determine a target packaging object corresponding to the target text data from at least one packaging object.

[0087] It can be understood that the target packaging object is a packaging object in which the timestamp of at least one packaging object overlaps with the timestamp of the target text data.

[0088] 106. Based on the first trigger operation, when the object type of the target packaging object is any one of the first types, perform corresponding processing on the target packaging object.

[0089] Among them, the first type includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages.

[0090] 107. When the object type of the target package object is any of the second types, prohibiting corresponding processing of the target package object.

[0091] Among them, the second type includes: background music.

[0092] It is understood that in the embodiment of the present application, the types of packaging objects are differentiated. Some types of packaging objects are linked to the clipping operation of the target text data, while some types of packaging objects do not respond to the clipping operation of the target text data. In this way, it can be determined whether to link the packaging object with clipping based on the type of the packaging object.

[0093] It is understandable that when the type of packaging object is background music, if it is edited in conjunction with the target text data, the background music may become incoherent, repetitive, etc., and will no longer be able to enhance emotions and make the audience feel as if they are in the scene.

[0094] In some embodiments of the present application, the first type includes the third type and the fourth type; the above-mentioned step 106 can be specifically implemented through the following step 106a or step 106b.

[0095] 106a. When the object type of the target packaging object is any one of the third types, perform corresponding processing on the target packaging segment in the target packaging object, where the target packaging segment is a portion of the target packaging object corresponding to the target text data.

[0096] The third type includes at least one of the following: subtitle text, picture-in-picture.

[0097] In the embodiment of the present application, the packaging objects of the subtitle text and the picture-in-picture type (third type) generally correspond one-to-one to the media video, and therefore can be linked to the target text data for editing.

[0098] 106b. When the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is within the time code range corresponding to the target text data, perform corresponding processing on the entire target packaging object.

[0099] The time code is divided into a time range between the time code (timestamp) corresponding to the first semantic character in the target text data and the time code corresponding to the last semantic character.

[0100] It can be understood that the start timestamp of the target packaging object is within the time code range corresponding to the target text data, that is, the start timestamp of the target packaging object is the same as a timestamp in the time code range.

[0101] Among them, the fourth type includes at least one of the following: non-subtitle text, stickers, special effects, filters, and material packages.

[0102] In the embodiment of the present application, the packaging objects of non-subtitle text, stickers, special effects, filters, and material package types (the fourth type) usually only correspond to a part of the media video. If a judgment is not made to determine whether to perform linkage editing with the target text data, and the target text data is directly linked, it may cause the packaging objects to be incoherent, and one packaging object to become multiple, etc., thereby losing the original effect of the packaging object and increasing the bad effect. Therefore, in the embodiment of the present application, when the object type of the target packaging object is any one of the fourth types and the starting timestamp of the target packaging object is within the time code range corresponding to the target text data, the entire target packaging object is processed accordingly, that is, when it is determined to perform linkage editing with the target text data, the packaging object as a whole is processed accordingly. In this way, it can be ensured that when it is determined to perform linkage editing with the target text data, the packaging object as a whole is processed accordingly, and no bad effect is increased.

[0103] In some embodiments of the present application, after the above step 105, the video processing method provided by the embodiment of the present application may further include the following step 108.

[0104] 108. When the object type of the target packaging object is any of the fourth types and the start timestamp of the target packaging object is not within the time code range corresponding to the target text data, corresponding processing of the target packaging object is prohibited.

[0105] It can be understood that when the object type of the target packaging object is any of the fourth types and the starting timestamp of the target packaging object is not within the time code range corresponding to the target text data, it is prohibited to perform corresponding processing on the target packaging object, that is, it is determined not to respond to the linked editing with the target text data (not to perform linked editing with the target text data). In this way, it can be ensured that the packaging object still has the original effect.

[0106] In the embodiments of the present application, the modification operation acts on the text data to be processed and the media audio, and if there is subtitle text, it will also be linked together, but it does not act on the media video. That is, under normal circumstances, for the modification operation, the media audio and the text data to be processed are linked together for editing. If the video to be processed includes subtitle text, the subtitle text and the text data to be processed are linked together for editing, while the media video is not linked together for editing. In this way, when the text data to be processed is modified, the media audio and subtitle text are modified accordingly, but there is no need to modify the media video. The logic is simple and fits the actual situation, and the user experience is good.

[0107] In some embodiments of the present application, after the above step 101, the video processing method provided by the embodiment of the present application may further include the following steps 109 to 112.

[0108] 109 . In response to a modification operation on first text data in the text data to be processed, obtain modified second text data corresponding to the modification operation.

[0109] 110. Determine, based on the second mapping relationship, a first audio segment corresponding to the first text data from the media audio.

[0110] 111. Generate a second audio segment based on the second text data and the timbre information of the first audio segment.

[0111] In an embodiment of the present application, a second audio segment is generated based on the second text data and the timbre information of the first audio segment. On the one hand, it can ensure that the content of the second audio segment corresponds to the second text data, and on the other hand, it can ensure that the timbre information of the second audio segment is the same as the timbre information of the first audio segment. In this way, not only is the media audio modified accordingly with respect to the modification of the processed text data, but it is also ensured that the timbre information of the modified position in the media audio does not change, which can improve the user experience.

[0112] 112. Replace the first audio segment in the media audio with the second audio segment.

[0113] It can be understood that by replacing the first audio segment in the media audio with the second audio segment, the media audio and the first text data are linked to each other.

[0114] In some embodiments of the present application, the video to be processed also includes subtitle text, and there is a third mapping relationship between the text time code information and the third timeline information corresponding to the subtitle text; after the above step 109, the video processing method provided in the embodiment of the present application may also include the following steps 113 to 115.

[0115] 113. In response to the modification operation on the first text data in the text data to be processed, determine a first subtitle segment corresponding to the first text data from the subtitle text based on the third mapping relationship.

[0116] 114. Modify the first subtitle segment based on the second text data to obtain a second subtitle segment.

[0117] It can be understood that the second subtitle segment corresponds to the second text data, and the second audio segment also corresponds to the second text data. Therefore, the second subtitle segment also corresponds to the second audio segment. In this way, when the audience watches, the subtitles will not be inconsistent with the audio.

[0118] 115. Replace the first subtitle segment in the subtitle text with the second subtitle segment.

[0119] It can be understood that by replacing the first subtitle segment in the subtitle text with the second subtitle segment, the subtitle text and the first text data are modified in a linked manner.

[0120] In some embodiments of the present application, when the media video timeline track and the media audio timeline track are not displayed, the selection operation is only for text data, and the selected text data is selected and displayed.

[0121] In some embodiments of the present application, when displaying a media video timeline track or a media audio timeline track, a selection operation selects and displays the selected text data, selects and displays the portion of the track in the media video timeline track corresponding to the selected text data, and selects and displays the portion of the track in the media audio timeline track corresponding to the selected text data. This provides a better visual experience for the user and allows the user to understand how the corresponding editing process is implemented.

[0122] In some embodiments of the present application, the first trigger operation also includes a selection operation; before the above step 102, the video processing method provided in the embodiment of the present application may further include the following step 116, and the above step 104 can be specifically implemented through the following steps 104a to 104c.

[0123] 116. Display the first timeline track and the second timeline track.

[0124] The first timeline track is the timeline track corresponding to the media video, and the second timeline track is the timeline track corresponding to the media audio.

[0125] 104a. Display the target text data according to the first display style.

[0126] 104b. Display the portion of the first timeline track corresponding to the target video clip in a second display style.

[0127] 104c. Display the portion of the second timeline track corresponding to the target audio clip in a third display style.

[0128] The first display style, the second display style, and the third display style may be partially the same, partially different, or all the same, or all different, which is not limited here.

[0129] In some embodiments of the present application, when the timeline track of each packaged object is displayed in addition to the media video timeline track and the media audio timeline track, whether the timeline track of the packaged object is selected for display is determined based on the type of the packaged object and whether the starting timestamp of the packaged object is within the time code range corresponding to the selected text data.

[0130] The track linkage logic during editing in the embodiment of the present application (specifically corresponding to whether the above-mentioned media video, media audio, and packaging object are linked to the target text data for editing) includes:

[0131] First, determine the start time code and end time code of the selected text, and determine the target time code range of the selected text;

[0132] The track portions corresponding to the target time code range in the timeline track of the media video, the timeline track of the media audio, the timeline track of the picture-in-picture, and the timeline track of the text subtitle are selected and displayed accordingly, and respond to subsequent editing operations such as deletion, movement, and copying;

[0133] If the start timestamps of non-text subtitle timelines or sticker timelines are within the target timecode range, the corresponding tracks will be selected and displayed accordingly, and will respond to subsequent editing operations such as deletion, movement, and copying.

[0134] The timeline track of the background music is never selected and displayed accordingly, nor does it respond to subsequent editing operations such as deletion, moving, and copying.

[0135] In some embodiments of the present application, the above-mentioned track linkage logic during editing is the default, and linkage switch controls can also be set for different timeline tracks. The user can determine whether to link the corresponding timeline tracks by triggering the linkage switch control. This is not limited here.

[0136] In some embodiments of the present application, the method provided in the embodiments of the present application may further include the following steps 201 to 202.

[0137] 201 : In response to a second triggering operation on selected text data in the text data to be processed, display a tool menu in an area corresponding to the selected text data.

[0138] Among them, the tool menu includes at least one of the following categories: clipping controls and adding controls. The clipping controls include at least one of the following: a delete control corresponding to the delete operation, a copy control corresponding to the copy operation, a cut control corresponding to the cut control, a modify control corresponding to the modify operation, smart long and short sentence controls, a search control, and a replace control; the adding controls include at least one of the following: adding picture-in-picture controls, adding non-subtitle text controls, adding sticker controls, adding special effects controls, adding sound effects controls, adding filter controls, adding material package controls, and adding background music controls.

[0139] Among them, any of the above adding controls can support users to manually select to add or intelligently match to add.

[0140] 202. In response to a third triggering operation on a target control in the tool menu, perform corresponding processing on at least one of the media video, the text data to be processed, and the media audio.

[0141] In an embodiment of the present application, a tool menu can be displayed through a second trigger operation on the selected text data, and then the corresponding editing processing can be performed on the trigger operation of any control in the tool menu according to actual needs. The interactive operation is simple and convenient, which is conducive to improving the user experience.

[0142] In some embodiments of the present application, the display area where the text data to be processed is located includes a material library control, which can be used to perform text clipping on the global video and rough cuts on the material library content.

[0143] In some embodiments of the present application, after the above step 101, the method provided in the embodiment of the present application may further include the following steps 301 to 305.

[0144] 301. In response to a triggering operation on a material library control, display a plurality of preset video materials.

[0145] 302. In response to a fifth triggering operation on a target video material among a plurality of preset video materials, display a text rough cut floating window.

[0146] Among them, the text rough cut floating window includes the material text data corresponding to the target video material, the target video material includes the target material video and the target material audio, the material time code information of the material text data and the timeline information of the target material video have a fourth mapping relationship, and the material time code information and the timeline information of the target material audio have a fifth mapping relationship.

[0147] 303. In response to the text rough-cutting operation on the selected material text in the material text data, the selected material text is synthesized with the text data to be processed to obtain the text data to be edited.

[0148] 304. Combining the source video clip and the media video into a media video to be edited.

[0149] 305. The material audio clip and the media audio are synthesized into the media audio to be edited, so as to implement editing of the video to be edited based on the text editing operation of the text data to be edited.

[0150] Among them, the material video clip is a video clip corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship; the material audio clip is an audio clip corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship; the video to be edited includes the media video to be edited and the media audio to be edited.

[0151] In an embodiment of the present application, the text clipping panel may also include a material library control. By triggering the material library control, multiple preset video materials can be displayed. The user can select the required target material video from the multiple preset video materials according to needs, and then pull up the text rough-cut floating window by performing a text rough-cut operation on the target material video. The rough-cut floating window displays the material text data corresponding to the target material video. Therefore, the user can perform a text rough-cut operation on the selected material text in the material text data to clip the required clipping material segment in the target material video, obtain the material video segment and the material audio segment, and then synthesize the selected material text with the text data to be processed to obtain the text data to be edited, synthesize the material video segment with the media video into the media video to be edited, and synthesize the material audio segment with the media audio into the media audio to be edited, so that the text editing operation based on the text data to be edited can realize the editing of the video to be edited.

[0152] In an embodiment of the present application, the video to be processed can also be synthesized by selecting at least one material from the material library and then roughly cutting the text to obtain multiple material clips. The specific details can be determined according to actual conditions and are not limited here.

[0153] In some embodiments of this application, you can select any material in the material library, right-click it, and add a new text rough cut entry. Clicking the text rough cut entry will call out the text rough cut floating window and start ASR recognition of the selected material. You can also use the text rough cut floating window to add the material to the timeline.

[0154] The text rough cut floating window includes a floating text preview area, and may also include a floating video preview area, a floating timeline display area, etc. For details, please refer to the relevant description of the text clipping panel, and it can be determined according to actual conditions. There is no limitation here.

[0155] The ASR recognition results (text data) for the selected material will be displayed in the floating text preview area. The specific format of the floating text preview area displaying the material text data can be referred to the specific format of the text preview area displaying the text data to be processed in the text clipping panel above, which will not be repeated here.

[0156] Text rough cut operation: Select the material text, drag the selected material text to perform the rough cut operation; or select the material text, call out the rough cut control by email, and click the rough cut control. The specific operation can be determined according to the actual situation and is not limited here.

[0157] In some embodiments of the present application, local materials are intelligently searched for keywords. When the search results are selected and the text rough cut is entered, the corresponding text is matched by default and selected to be collapsed to quickly complete the rough cut.

[0158] For input text data, support for rough cutting of text through the editing floating window. You can call out the text rough cutting control by triggering the input text data or matching video. Then, by triggering the text rough cutting control, a new modal window is called out. You can use this panel to rough cut the input text data and perform the corresponding editing of the matching video. The specific process can be referred to the editing operation of the processed text data above to achieve the editing of the processed video, and will not be repeated here.

[0159] In an embodiment of the present application, the matching video corresponding to the input text data is clipped through a clipping pop-up window, and the clipped video to be processed and the clipped matching video are synthesized to obtain a target clipped video.

[0160] In some embodiments of the present application, different display objects can be displayed in different areas in the text clipping panel. For example, the text clipping panel can be composed of some or all of the following areas: a video preview area, a text preview area, a text clipping menu area, a global function area, and a timeline track display area. Among them, the video preview area is used to display the video to be processed, the text preview area is used to display the text data to be processed, the text clipping menu area is used to display the text clipping menu, and the global function area is used to display the global function menu.

[0161] In some embodiments of the present application, when the text clipping panel is pulled up, the default layout mode is adopted. Then, it supports determining the partitions that can be displayed in the text clipping panel by selecting a preset layout mode (such as the "Work Mode" and "Panel View Mode" mode menus), and also supports independent selection of whether each partition is displayed (check -> Show; uncheck -> Hide. For example, each panel partition can be displayed / undisplayed through "√ / x" in the view menu.). Moreover, the panel partition can be dragged to float outside the editor. When dragging to restore, the panel partition can only be returned to the corresponding position under the default layout (that is, it cannot be adsorbed at will).

[0162] The text clipping panel also includes Restore, Full Screen, Minimize, and Close options.

[0163] The text preview area includes the text data to be processed, the text time code information corresponding to the text data to be processed, and a timeline cursor. The text data to be processed includes text data corresponding to at least one user's voice, and the text data to be processed includes at least one text paragraph (including a silent segment [···]). The timeline cursor is used to indicate the current playback progress of the video to be processed. The corresponding text word or sentence will be highlighted in the current playback timeline of the video to be processed. The timeline cursor position and highlighted text word or sentence are updated in real time while the video to be processed is playing.

[0164] The text segmentation logic in the embodiment of the present application can be segmented according to the following priority order: the priority of manual line return segmentation of the text is higher than the priority of speaker segmentation, the priority of speaker segmentation is higher than the priority of segmentation according to pause duration, and the priority of segmentation according to pause duration is higher than the priority of segmentation according to the number of words in the paragraph.

[0165] The silent segment logic in the embodiments of the present application may include: Option 1: silent segments greater than or equal to 1s are regarded as silent segments and displayed in the text preview area of ​​the text clipping panel; Option 2: silent segments greater than or equal to 10 times the speaking speed are regarded as silent segments and displayed in the text preview area of ​​the text clipping panel.

[0166] The positioning jump logic in the embodiment of the present application can be that the manual scrolling text area is unresponsive and only jumps when clicking to switch the selection.

[0167] In an embodiment of the present application, the tool menu can be called up by triggering the selected text, and then the text clipping mode can be entered by triggering any clipping control in the tool menu or directly by triggering any clipping control in the text clipping menu to perform text and video editing.

[0168] The text data to be processed is in preview mode (the default state). The cursor is positioned at the playback time point, and the currently read text word or sentence (whether paused or playing) is highlighted. At this point, you can trigger an operation to enter edit mode to modify the content. For example, by triggering a selection control, the currently highlighted sentence is selected and enters edit mode.

[0169] When there is selected text in the text data to be processed, the editing state is entered, and the following operations can be performed: delete operation, insert xxx (such as text template / picture-in-picture, etc.), modify operation, move operation, copy operation, etc.

[0170] Delete operation: When text is selected, the tool menu is called out and the delete control is visible. Select the delete control to delete the selected text and the track content corresponding to the selected text (the specific track to be deleted depends on the track linkage logic). After deletion is completed, the preview state is returned.

[0171] To insert xxx (such as text templates / picture-in-picture, etc.), when you select text, call out the tool menu, and you can see the Add xxx control. Pull up the corresponding panel to display the optional materials of the packaging objects that can be added for the user to choose. You can also intelligently match the materials of the packaging objects to be added based on the selected text. The specific method can be determined according to the actual situation and is not limited here.

[0172] After selecting any clip using the user's own method or using Smart Match, the clip will be overwritten with the start timecode of the selected text as a new track. After adding, the result will not be displayed in any form in the text preview area, but it will be visible in the video preview area (video preview window, video player), so it can only be edited on the screen. After the insertion is completed, the editing state remains.

[0173] Move operation: After selecting the text, long press the selected area, you can drag the selected text paragraph to any position (not within the word), and the corresponding video clip will also move with it. After moving, it will still be in editing state.

[0174] Copy operation: When text is selected, call out the tool menu, and the copy control is visible. Select the copy control, copy the text and track content (the specific track copied depends on the track linkage switch), and paste it into the selected text. After the copy is completed, the text remains in editing mode.

[0175] Modification operation, this operation acts on the text data to be processed and media audio, and will also be linked to subtitles if there are any, but will not act on media videos. When text is selected, call out the tool menu and the modification control will be visible. After selecting, pull up the input box, and the entire text data to be processed will enter the input box, and the currently selected sentence will also be selected by default. The result of this modification operation only affects the text data to be processed and media audio (subtitle text), and does not affect media videos. Therefore, the modification operation response results are as follows: line break - subtitles are divided into two sentences; removing line breaks - subtitles are merged into one sentence; correcting typos, the subtitles are modified accordingly, and sometimes the media audio pronunciation is modified accordingly. Entering new text - the video time code will not be extended. The text modification results only affect the subtitles. Select the completion control on the input box to end the modification operation and return to the preview state, with the cursor at the last input position.

[0176] In the embodiment of the present application, in the text clipping mode, the following global function operations can be performed: search operation, watermark removal operation, and subtitle switch operation.

[0177] To search, enter text clipping mode, select the search control, expand the search box, enter your search keywords, and click Confirm. Matching text will be highlighted. You can select the previous and next controls to quickly jump to search results. Click the close control in the search box to close it.

[0178] To remove water words, enter text editing mode and select the remove water words control to start identifying water words. After the water word recognition is completed, the water words in the text preview area are highlighted, and a new confirmation (remove water words) control is added at the bottom. By triggering the confirmation control, the water word removal function can be realized with one click. The user can confirm whether the water word recognition result is accurate. If it is inaccurate, select a single water word to call out the recovery control (cancel the corresponding highlight mark). If confirmed, select the confirmation control to continue the subsequent operation. Select the confirmation control to delete all highlighted water words and return to text editing mode.

[0179] In some embodiments of the present application, the watermark removal operation will trigger the full text to be watermarked. If the watermark has already been processed, a second click will respond with another process.

[0180] Subtitles are switched on and off, and the subtitle content is linked to the text content. The subtitle switch control (which can include the subtitle on and subtitle off controls) only controls whether the subtitles are visible. Selecting the subtitle control on adds subtitles, and the subtitles will be updated when the text data is updated; selecting the subtitle control off does not add subtitles.

[0181] Example 1: Due to the limited display size of mobile electronic devices and the fact that most mobile electronic devices are for non-professional users, the text clipping panel for mobile electronic devices can display by default the text preview area (corresponding to displaying the text data to be processed), the video preview area (corresponding to displaying the video to be processed), the text clipping menu area (corresponding to displaying the text clipping menu) and the global function area (corresponding to displaying the global function menu) as shown in FIG3 , wherein the area indicated by mark “31” is the video preview area, the area indicated by mark “32” is the text preview area, the area indicated by mark “33” is the global function area, and the area indicated by mark “34” is the text clipping menu area. Among them, the first timeline track, the second timeline track, and the timeline tracks corresponding to other packaging objects in the video to be processed can be controlled to be displayed accordingly in the text clipping panel through corresponding trigger operations according to usage requirements, and are not limited here.

[0182] Example 2: On a non-mobile electronic device, a three-part layout mode of the text preview area, video preview area, and timeline track display area as shown in Figure 4 can be displayed by default, or a four-part layout mode of the text preview area, video preview area, timeline track display area, and parameter panel display area as shown in Figure 5 can be displayed by default. Among them, the area indicated by mark "41" is the video preview area, the area indicated by mark "42" is the text preview area, the area indicated by mark "43" is the timeline track display area, and the area indicated by mark "51" is the parameter panel display area.

[0183] FIG6 is a structural block diagram of a video processing device according to an embodiment of the present disclosure, as shown in FIG6 , including:

[0184] The display module 601 is used to display the text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first timeline information corresponding to the media video; the text time code information has a second mapping relationship with the second timeline information corresponding to the media audio; the determination module 602 is used to respond to a first trigger operation on the target text data in the text data to be processed, and determine the target video segment corresponding to the target text data from the media video based on the first mapping relationship; the determination module 602 is also used to determine the target audio segment corresponding to the target text data from the media audio based on the second mapping relationship; the processing module 603 is used to perform corresponding processing on the target text data, the target video segment and the target audio segment based on the first trigger operation.

[0185] In some embodiments of the present application, the video to be processed also includes at least one packaging object, and there is a mapping relationship between the text time code information and the timeline information corresponding to each packaging object; wherein the object type of each packaging object includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packages, and background music.

[0186] In some embodiments of the present application, the first trigger operation includes any one of the following: a delete operation, a drag operation, a copy operation, and a cut operation; the determination module 602 is also used to determine, in response to the first trigger operation on the target text data in the text data to be processed, a target packaging object corresponding to the target text data from at least one packaging object; the processing module 603 is also used to, based on the first trigger operation, perform corresponding processing on the target packaging object when the object type of the target packaging object is any one of the first types; and prohibit corresponding processing on the target packaging object when the object type of the target packaging object is any one of the second types; wherein the first type includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages; the second type includes: background music.

[0187] In some embodiments of the present application, the first type includes the third type and the fourth type; the processing module 603 is specifically used to perform corresponding processing on the target packaging segment in the target packaging object when the object type of the target packaging object is any one of the third types, and the target packaging segment is the part of the target packaging object corresponding to the target text data; or, when the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is within the time code range corresponding to the target text data, perform corresponding processing on the entire target packaging object; wherein the third type includes at least one of the following: subtitle text, picture-in-picture; the fourth type includes at least one of the following: non-subtitle text, stickers, special effects, filters, and material packages.

[0188] In some embodiments of the present application, the processing module 603 is also used to prohibit corresponding processing of the target packaging object when the object type of the target packaging object is any of the fourth types and the start timestamp of the target packaging object is not within the time code range corresponding to the target text data.

[0189] In some embodiments of the present application, the device also includes: an acquisition module, a generation module and a replacement module, the acquisition module is used to respond to a modification operation on the first text data in the text data to be processed, and obtain the modified second text data corresponding to the modification operation; the determination module 602 is also used to determine the first audio segment corresponding to the first text data from the media audio based on the second mapping relationship; the generation module is used to generate a second audio segment based on the second text data and the timbre information of the first audio segment; the replacement module is used to replace the first audio segment in the media audio with the second audio segment.

[0190] In some embodiments of the present application, the video to be processed also includes subtitle text, and a third mapping relationship exists between the text time code information and the third timeline information corresponding to the subtitle text; the device also includes: a modification module and a replacement module; the determination module 602 is also used to respond to a modification operation on the first text data in the text data to be processed, and based on the third mapping relationship, determine the first subtitle segment corresponding to the first text data from the subtitle text; the modification module is used to modify the first subtitle segment based on the second text data to obtain a second subtitle segment; the replacement module is used to replace the first subtitle segment in the subtitle text with the second subtitle segment.

[0191] In some embodiments of the present application, the first trigger operation also includes a selection operation; the display module 601 is also used to display the first timeline track and the second timeline track before determining the target video segment corresponding to the target text data from the media video based on the first mapping relationship in response to the first trigger operation for the target text data in the text data to be processed, the first timeline track is the timeline track corresponding to the media video, and the second timeline track is the timeline track corresponding to the media audio; the processing module 603 is specifically used to display the target text data according to the first display style; display the part of the first timeline track corresponding to the target video segment according to the second display style; and display the part of the second timeline track corresponding to the target audio segment according to the third display style.

[0192] In some embodiments of the present application, the display module 601 is also used to display a tool menu in the area corresponding to the selected text data in response to a second trigger operation on the selected text data in the text data to be processed, and the tool menu includes at least one of the following categories: clipping controls and adding controls. The clipping controls include at least one of the following: a delete control corresponding to the delete operation, a copy control corresponding to the copy operation, a cut control corresponding to the cut control, a modification control corresponding to the modification operation, an intelligent long and short sentence control, a search control, and a replacement control; the adding controls include at least one of the following: adding a picture-in-picture control, adding a non-subtitle text control, adding a sticker control, adding a special effect control, adding a sound effect control, adding a filter control, adding a material package control, and adding a background music control; the processing module 603 is also used to perform corresponding processing on at least one of the media video, the text data to be processed, and the media audio in response to a third trigger operation on the target control in the tool menu.

[0193] In some embodiments of the present application, the device further includes: a synthesis module; the display module 601 is used to display a plurality of preset video materials in response to a trigger operation on a material library control after displaying the text data to be processed corresponding to the video to be processed; in response to a fifth trigger operation on a target video material among the plurality of preset video materials, display a text rough cut floating window, the text rough cut floating window includes material text data corresponding to the target video material, the target video material includes a target material video and a target material audio, the material time code information of the material text data and the timeline information of the target material video have a fourth mapping relationship, and the material time code information and the timeline information of the target material audio have a fifth mapping relationship; the synthesis module is used to display a text rough cut floating window in response to a fifth trigger operation on a target video material among the plurality of preset video materials. The text rough cutting operation synthesizes the selected material text with the text data to be processed to obtain the text data to be edited, synthesizes the material video clip with the media video into the media video to be edited, and synthesizes the material audio clip with the media audio into the media audio to be edited, so that the text editing operation based on the text data to be edited can realize the editing of the video to be edited; wherein, the material video clip is a video clip corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship; the material audio clip is an audio clip corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship; the video to be edited includes the media video to be edited and the media audio to be edited.

[0194] In the embodiments of the present disclosure, each module can implement the video processing method provided by the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0195] FIG7 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure, which is used to exemplify an electronic device that implements any video processing method in an embodiment of the present disclosure and should not be understood as a specific limitation on the embodiment of the present disclosure.

[0196] As shown in Figure 7, electronic device 700 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 are also stored in RAM 703. Processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0197] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 700 is shown as having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0198] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processor 701, the functions defined in any video processing method provided by the embodiment of the present disclosure can be executed.

[0199] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0200] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0201] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0202] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: displays the text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first timeline information corresponding to the media video; the text time code information has a second mapping relationship with the second timeline information corresponding to the media audio; in response to a first trigger operation for the target text data in the text data to be processed, based on the first mapping relationship, the target video segment corresponding to the target text data is determined from the media video; based on the second mapping relationship, the target audio segment corresponding to the target text data is determined from the media audio; based on the first trigger operation, the target text data, the target video segment and the target audio segment are processed accordingly.

[0203] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the computer, partially on the computer, as a separate software package, partially on the computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0204] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0205] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0206] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0207] In the context of the present disclosure, a computer-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0208] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0209] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0210] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video processing method, comprising: Displaying the text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first time line information corresponding to the media video; and the text time code information has a second mapping relationship with the second time line information corresponding to the media audio; In response to a first trigger operation on target text data in the text data to be processed, determining a target video segment corresponding to the target text data from the media video based on the first mapping relationship; Based on the second mapping relationship, determining a target audio segment corresponding to the target text data from the media audio; Based on the first trigger operation, corresponding processing is performed on the target text data, the target video segment, and the target audio segment.

2. The method according to claim 1, wherein the video to be processed further comprises at least one packaging object, and a mapping relationship exists between the text time code information and the timeline information corresponding to each packaging object; in, The object type of each packaging object includes at least one of the following: Subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packs, background music.

3. The method according to claim 2, wherein the first trigger operation comprises any one of the following: Delete operation, drag operation, copy operation, cut operation; The method further comprises: In response to a first trigger operation on target text data in the text data to be processed, determining a target packaging object corresponding to the target text data from the at least one packaging object; Based on the first trigger operation, when the object type of the target packaging object is any one of the first types, performing corresponding processing on the target packaging object; When the object type of the target packaging object is any one of the second types, prohibiting corresponding processing of the target packaging object; The first type includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages; The second type includes: background music.

4. The method of claim 3, wherein the first type includes a third type and a fourth type; When the object type of the target packaging object is any one of the first types, performing corresponding processing on the target packaging object includes: In the case where the object type of the target packaging object is any one of the third types, a target packaging segment in the target packaging object is processed accordingly, the target packaging segment being a portion of the target packaging object corresponding to the target text data; or, When the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is within the time code range corresponding to the target text data, the entire target packaging object is processed accordingly; The third type includes at least one of the following: subtitle text, picture-in-picture; The fourth type includes at least one of the following: non-subtitle text, stickers, special effects, filters, and material packages.

5. The method according to claim 4, wherein the method further comprises: When the object type of the target packaging object is any one of the fourth types and the start timestamp of the target packaging object is not within the time code range corresponding to the target text data, corresponding processing of the target packaging object is prohibited.

6. The method according to claim 1, wherein the method further comprises: In response to a modification operation on first text data in the text data to be processed, Acquire the modified second text data corresponding to the modification operation; Based on the second mapping relationship, determining a first audio segment corresponding to the first text data from the media audio; generating a second audio segment based on the second text data and the timbre information of the first audio segment; The first audio segment in the media audio is replaced with the second audio segment.

7. The method according to claim 6, wherein the video to be processed further includes subtitle text, and a third mapping relationship exists between the text time code information and the third timeline information corresponding to the subtitle text; the method further includes: In response to a modification operation on first text data in the text data to be processed, determining a first subtitle segment corresponding to the first text data from the subtitle text based on the third mapping relationship; Based on the second text data, modify the first subtitle segment to obtain a second subtitle segment; The first subtitle segment in the subtitle text is replaced by the second subtitle segment.

8. The method according to any one of claims 1 to 7, wherein the first trigger operation further comprises a selection operation; and before the first trigger operation for the target text data in the text data to be processed is determined from the media video based on the first mapping relationship, the method further comprises: Displaying a first timeline track and a second timeline track, wherein the first timeline track is a timeline track corresponding to the media video, and the second timeline track is a timeline track corresponding to the media audio; The performing corresponding processing on the target text data, the target video segment and the target audio segment based on the first trigger operation includes: Displaying the target text data according to a first display style; Displaying the portion of the first timeline track corresponding to the target video segment according to a second display style; According to the third display style, the portion of the second timeline track corresponding to the target audio segment is displayed.

9. The method according to any one of claims 1 to 7, wherein the method further comprises: In response to a second trigger operation on selected text data in the text data to be processed, a tool menu is displayed in an area corresponding to the selected text data, the tool menu including at least one of the following categories: a clipping control and an adding control, the clipping control including at least one of the following: a deletion control corresponding to a deletion operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut control, a modification control corresponding to a modification operation, an intelligent long and short sentence control, a search control, and a replacement control; the adding control including at least one of the following: an adding picture-in-picture control, an adding non-subtitle text control, an adding sticker control, an adding special effect control, an adding sound effect control, an adding filter control, an adding material package control, and an adding background music control; In response to a third trigger operation on a target control in the tool menu, corresponding processing is performed on at least one of the media video, the text data to be processed, and the media audio.

10. The method according to any one of claims 1 to 7, wherein after displaying the text data to be processed corresponding to the video to be processed, the method further comprises: In response to a triggering operation on a material library control, displaying a plurality of preset video materials; In response to a fifth trigger operation on a target video material among the plurality of preset video materials, a text rough cut floating window is displayed, wherein the text rough cut floating window includes material text data corresponding to the target video material, the target video material includes a target material video and a target material audio, a fourth mapping relationship exists between material time code information of the material text data and timeline information of the target material video, and a fifth mapping relationship exists between the material time code information and timeline information of the target material audio; In response to a text rough cut operation on the selected material text in the material text data, The method comprises the following steps: synthesizing the selected material text with the text data to be processed to obtain the text data to be edited, synthesizing the material video clip with the media video into the media video to be edited, and synthesizing the material audio clip with the media audio into the media audio to be edited, so as to realize the editing of the video to be edited based on the text editing operation on the text data to be edited; Among them, the material video clip is a video clip corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship; the material audio clip is an audio clip corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship; the video to be edited includes the media video to be edited and the media audio to be edited.

11. A video processing device, comprising: A display module, used for displaying the text data to be processed corresponding to the video to be processed; wherein the video to be processed includes media video and media audio, and the text time code information corresponding to the text data to be processed has a first mapping relationship with the first time line information corresponding to the media video; and the text time code information has a second mapping relationship with the second time line information corresponding to the media audio; A determination module, configured to determine, in response to a first trigger operation on target text data in the text data to be processed, a target video segment corresponding to the target text data from the media video based on the first mapping relationship; The determination module is further used to determine, from the media audio, a target audio segment corresponding to the target text data based on the second mapping relationship; A processing module is used to perform corresponding processing on the target text data, the target video segment and the target audio segment based on the first trigger operation.

12. An electronic device comprising: A memory and a processor, the memory is used to store computer programs; The processor is used to execute the video processing method according to any one of claims 1 to 10 when calling the computer program.

13. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the video processing method according to any one of claims 1 to 10 is implemented.

Images