Video processing methods, apparatus, electronic devices, storage media, and programs

Abstract

This disclosure relates to a video processing method, apparatus, electronic device, and storage medium. The method includes displaying text data to be processed corresponding to a video to be processed, which includes media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, and in response to a first trigger operation on target text data among the text data to be processed, the method determines a target video segment corresponding to the target text data from the media video based on the first mapping relationship, determines a target audio segment corresponding to the target text data from the media audio based on the second mapping relationship, and performs processing corresponding to the target text data, target video segment, and target audio segment based on the first trigger operation.

Description

Technical Field

[0001] (Cross-reference to Related Applications) This application claims priority based on a Chinese patent application filed with the China National Intellectual Property Administration on August 31, 2023, with an application number of 202311120475.X and an invention title of "Video Processing Method, Apparatus, Electronic Device, and Storage Medium", and all of its content is incorporated herein by reference.

[0002] This disclosure relates to the technical field of video, and particularly to video processing methods, apparatuses, electronic devices, storage media, and program products.

Background Art

[0003] As Internet information becomes increasingly rich, watching videos has become an entertainment activity in people's daily lives. In order to improve the user's viewing experience, editing videos before posting various videos is an important part.

[0004] When creating, the creator can use video editing tools to edit the captured video materials. Currently, most video editing tools mainly achieve video editing by editing video frames, such as dragging, rearranging, switching materials, adding subtitles, etc. However, when editing video frames, certain video editing knowledge is required from the creator, and at the same time, the creator cannot complete the video editing work quickly. Therefore, the editing operation of video frames is relatively complex and time-consuming, affecting the creator's usage experience.

Summary of the Invention

[0005] To solve the above problems, or at least partially solve the above problems, this disclosure provides a video processing method, apparatus, electronic device, storage medium, and program product.

[0006] According to a first aspect of the embodiments of this disclosure, the text data to be processed is displayed in relation to a video to be processed, which includes media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between text timecode information and second timeline information corresponding to the media audio. In response to a first trigger operation on the target text data among the text data to be processed, the target video segment corresponding to the target text data is determined from the media video based on a first mapping relationship, Based on the second mapping relationship, the target audio segment corresponding to the target text data is determined from the media audio, The present invention provides a video processing method that includes performing corresponding processing on target text data, target video segments, and target audio segments based on a first trigger operation.

[0007] In some embodiments of the present invention, the video to be processed further comprises at least one package object, and there is a mapping relationship between text timecode information and timeline information corresponding to each package object, and the object type of each package object comprises at least one of subtitle text, picture-in-picture, non-subtitle text, sticker, special effects, filter, material package, and background music.

[0008] In some embodiments of the present application, the first trigger operation includes one of the following: delete operation, drag operation, copy operation and cut operation. The method involves determining a target package object corresponding to the target text data from at least one package object in response to a first trigger operation on the target text data among the text data to be processed, Based on the first trigger operation, if the object type of the target package object is one of the first types, which includes at least one of subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages, then the corresponding processing is performed on the target package object. This further includes prohibiting the performance of a corresponding operation on a target package object if the object type of the target package object is one of the second types, including background music.

[0009] In some embodiments of the present application, the first type includes a third type which includes at least one of subtitle text and picture-in-picture, and a fourth type which includes at least one of non-subtitle text, stickers, special effects, filters and material packages. If the object type of the target package object is one of the first types, the corresponding processing will be performed on the target package object. If the object type of the target package object is one of the third types, the process includes performing the corresponding operation on the target package segment, which is the part of the target package object that corresponds to the target text data, or, if the object type of the target package object is one of the fourth types and the start timestamp of the target package object is within the timecode range corresponding to the target text data, the process includes performing the corresponding operation on the entire target package object.

[0010] In some embodiments of the present application, the method further includes prohibiting the corresponding processing of the target package object if the object type of the target package object is one of the fourth types and the start timestamp of the target package object is outside the timecode range corresponding to the target text data.

[0011] In some embodiments of the present application, the method further includes: obtaining a modified second text data corresponding to a modification operation in text data to be processed; determining a first audio segment corresponding to the first text data from media audio based on a second mapping relationship; generating a second audio segment based on the second text data and timbre information of the first audio segment; and replacing the first audio segment in media audio with the second audio segment.

[0012] In some embodiments of the present invention, the video to be processed further includes subtitle text, and there is a third mapping relationship between text timecode information and third timeline information corresponding to the subtitle text, and the method further includes, in response to a modification operation on first text data in the text data to be processed, determining a first subtitle segment from the subtitle text corresponding to the first text data based on the third mapping relationship, modifying the first subtitle segment to obtain a second subtitle segment based on second text data, and replacing the first subtitle segment in the subtitle text with the second subtitle segment.

[0013] In some embodiments of the present application, the first trigger operation further includes a selection operation, and in response to the first trigger operation on target text data among the text data to be processed, the method further includes displaying a first timeline track which is a timeline track corresponding to media video and a second timeline track which is a timeline track corresponding to media audio, before determining a target video segment corresponding to the target text data from media video based on a first mapping relationship, and the processing corresponding to the target text data, target video segment and target audio segment based on the first trigger operation includes displaying the target text data according to a first display style, displaying the portion of the first timeline track corresponding to the target video segment according to a second display style, and displaying the portion of the second timeline track corresponding to the target audio segment according to a third display style.

[0014] In some embodiments of the present application, the method involves displaying a tool menu in an area corresponding to selected text data in response to a second trigger operation on selected text data from the text data to be processed, wherein the tool menu includes at least one of edit controls and add controls, the edit controls include at least one of a delete control corresponding to a delete operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut operation, a change control corresponding to a change operation, an intelligent short / long text control, a search control, and a replace control, and the add controls include at least one of a picture-in-picture add control, a non-subtitled text add control, a sticker add control, a special effect add control, a sound effect add control, a filter add control, a material package add control, and a background music add control. The further includes performing a corresponding process on at least one of media video, text data to be processed, and media audio in response to a third trigger operation on the target control in the Tools menu.

[0015] In some embodiments of the present invention, after displaying text data to be processed corresponding to the video to be processed, the method displays a plurality of pre-configured video materials in response to a trigger operation on the material library control. In response to a fifth trigger operation on a target video material from among multiple pre-configured video materials, a text roughcut floating window is displayed, wherein the text roughcut floating window contains material text data corresponding to the target video material, the target video material package includes the target material video and the target material audio, there is a fourth mapping relationship between the material timecode information of the material text data and the timeline information of the target material video, and there is a fifth mapping relationship between the material timecode information and the timeline information of the target material audio. The process involves, in response to a text rough-cut operation on selected material text from the material text data, compositing the selected material text with the text data to be processed to obtain text data to be edited, compositing the material video segment and media video into the media video to be edited, and compositing the material audio segment and media audio into the media audio to be edited, so that editing of the video to be edited is realized based on a text editing operation on the text data to be edited, wherein the material video segment is a video segment corresponding to the selected material text obtained from the target material video based on a fourth mapping relationship, the material audio segment is an audio segment corresponding to the selected material text obtained from the target material audio based on a fifth mapping relationship, and the video to be edited includes the media video to be edited and the media audio to be edited.

[0016] According to a second aspect of the embodiments of the present disclosure, a display module for displaying text data to be processed corresponding to a video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, In response to a first trigger operation on the target text data among the text data to be processed, a method for determining the target video segment corresponding to the target text data from the media video based on a first mapping relationship, and, Based on the second mapping relationship, a decision module for determining the target audio segment corresponding to the target text data from media audio, A processing module for performing corresponding processing on target text data, target video segment, and target audio segment based on a first trigger operation, The present invention provides a video processing device that includes [a specific component / function].

[0017] In some embodiments of the present invention, the video to be processed further comprises at least one package object, wherein there is a mapping relationship between text timecode information and timeline information corresponding to each package object, and the object type of each package object comprises at least one of subtitle text, picture-in-picture, non-subtitle text, sticker, special effects, filters, material package, and background music.

[0018] In some embodiments of the present application, the first trigger operation includes one of the following: delete operation, drag operation, copy operation and cut operation. The decision module is also used to determine the target package object corresponding to the target text data from at least one package object in response to a first trigger operation on the target text data among the text data to be processed. The processing module is also used to perform the corresponding processing on the target package object if the object type of the target package object is one of the first types, which include at least one of subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages, based on a first trigger operation, and to prohibit the corresponding processing on the target package object if the object type of the target package object is one of the second types, which includes background music.

[0019] In some embodiments of the present application, the first type includes a third type which includes at least one of subtitle text and picture-in-picture, and a fourth type which includes at least one of non-subtitle text, stickers, special effects, filters and material packages. Specifically, this processing module is used to perform processing on the target package segment, which is the part of the target package object corresponding to the target text data, when the object type of the target package object is one of the third types, or to perform processing on the entire target package object when the object type of the target package object is one of the fourth types and the start timestamp of the target package object is within the time code range corresponding to the target text data.

[0020] In some embodiments of the present application, when the object type of the target package object is any one of the fourth type and the start timestamp of the target package object is outside the time code range corresponding to the target text data, the processing module is also used to prohibit performing corresponding processing on the target package object.

[0021] In some embodiments of the present application, the apparatus further includes an acquisition module for acquiring the modified second text data corresponding to the modification operation in response to the modification operation on the first text data in the text data to be processed, a determination module for determining the first audio segment corresponding to the first text data from the media audio based on the second mapping relationship, a generation module for generating the second audio segment based on the timbre information of the second text data and the first audio segment, and a replacement module for replacing the first audio segment in the media audio with the second audio segment.

[0022] In some embodiments of the present application, the video to be processed further includes subtitle text, and there is a third mapping relationship between the text time code information and the third timeline information corresponding to the subtitle text. The apparatus further includes a determination module for determining the first subtitle segment corresponding to the first text data from the subtitle text based on the third mapping relationship in response to the modification operation on the first text data in the text data to be processed, a modification module for modifying the first subtitle segment based on the second text data to obtain the second subtitle segment, and a replacement module for replacing the first subtitle segment in the subtitle text with the second subtitle segment.

[0023] In some embodiments of the present application, the first trigger operation further includes a selection operation, and in response to the first trigger operation on the target text data among the text data to be processed, the display module, based on the first mapping relationship, before determining the target video segment corresponding to the target text data from the media video, also uses to display a first timeline track that is a timeline track corresponding to the media video and a second timeline track that is a timeline track corresponding to the media audio. Specifically, the processing module is used to display the target text data according to the first display style, display the part of the first timeline track corresponding to the target video segment according to the second display style, and display the part of the second timeline track corresponding to the target audio segment according to the third display style.

[0024] In some embodiments of the present application, in response to a second trigger operation on the selected text data among the text data to be processed, the display module is to display a tool menu in the area corresponding to the selected text data, the tool menu includes at least one of an editing control and an adding control, the editing control includes at least one of a deletion control corresponding to a deletion operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut operation, a change control corresponding to a change operation, an intelligent long / short text control, a search control, and a replacement control, and the adding control includes at least one of a picture-in-picture adding control, a non-subtitle text adding control, a sticker adding control, a special effect adding control, a sound effect adding control, a filter adding control, a material package adding control, and a background music adding control. The processing module is also used to perform corresponding processing on at least one of media video, text data to be processed, and media audio in response to a third trigger operation on the target control in the tool menu.

[0025] In some embodiments of the present invention, the display module displays text data to be processed corresponding to the video to be processed, and then, in response to a trigger operation on the material library control, displays a plurality of pre-configured video materials. In response to a fifth trigger operation on a target video material from among multiple pre-configured video materials, a text roughcut floating window is displayed, wherein the text roughcut floating window contains material text data corresponding to the target video material, the target video material package includes the target material video and the target material audio, there is a fourth mapping relationship between the material timecode information of the material text data and the timeline information of the target material video, and there is a fifth mapping relationship between the material timecode information and the timeline information of the target material audio. The process involves, in response to a text rough-cut operation on selected material text from the material text data, compositing the selected material text with the text data to be processed to obtain text data to be edited, compositing the material video segment and media video into the media video to be edited, and compositing the material audio segment and media audio into the media audio to be edited, so that editing of the video to be edited is realized based on a text editing operation on the text data to be edited, wherein the material video segment is a video segment corresponding to the selected material text obtained from the target material video based on a fourth mapping relationship, the material audio segment is an audio segment corresponding to the selected material text obtained from the target material audio based on a fifth mapping relationship, and the video to be edited includes the media video to be edited and the media audio to be edited.

[0026] According to a third embodiment of the embodiments of the present disclosure, an electronic device is provided that includes a processor, memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, an electronic device is provided that realizes the video processing method described in the first embodiment.

[0027] According to a fourth embodiment of the embodiments of this disclosure, a computer-readable storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, the video processing method described in the first embodiment is realized.

[0028] According to a fifth embodiment of the embodiments of this disclosure, a computer program product including a computer program is provided, which, when executed on a processor, realizes the video processing method described in the first embodiment.

[0029] According to a sixth embodiment of the embodiments of this disclosure, a chip is provided that includes a processor and a communication interface coupled to the processor, wherein the processor executes program instructions and realizes the video processing method described in the first embodiment. [Brief explanation of the drawing]

[0030] The drawings herein are incorporated into the specification, constitute part of this specification, illustrate embodiments conforming to the disclosure, and are used together with the specification to illustrate the principles of the disclosure.

[0031] To more clearly illustrate the embodiments of this disclosure or the technical means in the prior art, the following is a brief introduction of the drawings that may be used in the description of the embodiments or the prior art. It will be obvious to those skilled in the art that other drawings can be obtained from these drawings without any special ingenuity. [Figure 1] This is flowchart 1 of a video processing method according to an embodiment of the present disclosure. [Figure 2]This is flowchart 2 of a video processing method according to an embodiment of the present disclosure. [Figure 3] This is schematic diagram 1 of the interface of a video processing method according to an embodiment of the present disclosure. [Figure 4] This is schematic diagram 2 of the interface of a video processing method according to an embodiment of the present disclosure. [Figure 5] This is schematic diagram 3 of the interface of a video processing method according to an embodiment of the present disclosure. [Figure 6] This is a structural block diagram of a video processing apparatus according to an embodiment of the present disclosure. [Figure 7] This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0032] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the embodiments of this disclosure will be further described below. The embodiments and features of the embodiments of this disclosure may be combined with each other, provided they do not conflict.

[0033] The following description provides many specific details to help you fully understand this disclosure, but this disclosure can also be implemented in forms other than those described herein. It is clear that the examples in the specification are not all examples, but only a selection of the examples of this disclosure.

[0034] The terms “first,” “second,” etc., in the specification and claims of this disclosure are not intended to indicate a specific order or priority, but rather to distinguish similar subjects. The data used in this manner may be substituted for each other where appropriate, so that the embodiments of this disclosure may be carried out in an order other than that illustrated or described herein. It should be understood that subjects distinguished by “first,” “second,” etc., are usually of the same kind, and the number of subjects is not limited; for example, there may be one first subject or more. Also, “and / or” in the specification and claims indicates at least one connected subject, and the letter “ / ” usually indicates that the preceding and following related subjects are in an “or” relationship.

[0035] The electronic devices in the embodiments of this disclosure may be mobile electronic devices or non-mobile electronic devices. Mobile electronic devices may include mobile phones, tablet computers, laptop computers, palmtop computers, automotive electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). Non-mobile electronic devices may include personal computers (PCs), televisions (TVs), teller machines, and self-service machines, but are not specifically limited in the embodiments of this disclosure.

[0036] The entity that executes the video processing method according to the embodiments of this disclosure may be the electronic device (including mobile electronic devices and non-mobile electronic devices), or a functional module and / or functional entity capable of implementing the video processing method in the electronic device. The details are not limited to the embodiments of this disclosure, as they can be determined according to the actual usage needs.

[0037] The video processing method according to the embodiment of this disclosure will be described in detail below, with reference to the drawings, based on specific embodiments and their application scenes.

[0038] As shown in Figure 1, embodiments of the present disclosure provide a video processing method that may include the following steps 101 to 104.

[0039] 101: Displays the text data to be processed corresponding to the video to be processed.

[0040] The video to be processed includes media video and media audio. There is a first mapping relationship between the text timecode information corresponding to the text data to be processed and the first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and the second timeline information corresponding to the media audio.

[0041] The text timecode information includes a timestamp corresponding to each semantic character in the text data to be processed, and the first timeline information includes a timestamp for each video frame of the media video. The first mapping relationship is the correspondence between the timestamp of each semantic character in the text data to be processed and the timestamp of each video frame in the media video. For example, the timestamp of one semantic character in the text data to be processed may correspond to the timestamp of one video frame (i.e., the timestamp of a semantic character in the text data to be processed corresponds one-to-one with the timestamp of a video frame in the media video), or the timestamp of one semantic character in the text data to be processed may correspond to the timestamps of multiple video frames.

[0042] Semantic characters are characters that have a single meaning. For example, in Chinese, one semantic character represents one Chinese character, and in English, one semantic character represents one word.

[0043] The second timeline information includes the timestamp of each audio frame in the media audio. The second mapping relationship is the correspondence between the timestamp of each semantic character in the text data to be processed and the timestamp of each audio frame in the media audio. For example, the timestamp of one semantic character in the text data to be processed may correspond to the timestamp of one audio frame (i.e., the timestamp of a semantic character in the text data to be processed corresponds one-to-one with the timestamp of an audio frame in the media audio), or the timestamp of one semantic character in the text data to be processed may correspond to the timestamps of multiple audio frames.

[0044] To make it easier to understand, based on the first mapping relationship, we may determine at least one video frame corresponding to any semantic character in the text data to be processed from the media video, or we may determine multiple video frames corresponding to any multiple semantic characters in the text data to be processed from the media video.

[0045] To make it easier to understand, based on the second mapping relationship, we may determine at least one audio frame corresponding to any semantic character in the text data to be processed from the media audio, or we may determine multiple audio frames corresponding to any multiple semantic characters in the text data to be processed from the media audio.

[0046] In the embodiments of this application, the video to be processed includes, but is not limited to, recorded video, video obtained based on a script, etc. The text data to be processed may be obtained by speech recognition of media audio or the video to be processed, or it may be a script. If the text data to be processed is a script, a first mapping relationship between the text data to be processed and the video timestamp of the media video can be obtained by matching the text data to be processed with the media video, and the speech recognition method includes, but is not limited to, automatic speech recognition (ASR) technology.

[0047] In some embodiments of the present invention, if the text data to be processed is a script, a second mapping relationship between the text data to be processed and the audio timestamp of the media audio may be obtained by matching the text data to be processed with the media audio, or a second mapping relationship may be obtained based on the mapping relationship between the video timestamp of the media video and the audio timestamp of the media audio, and the first mapping relationship. The details are not limited here as they can be determined according to the actual situation.

[0048] In some embodiments of the present invention, in response to a trigger operation on a text editing control in an interface that displays a video to be processed, a text editing panel can be invoked, the media video and media audio of the video to be processed can be extracted, the media audio can be speech-recognized to obtain text data to be processed, the text data to be processed can be displayed in the text editing panel, and a first mapping relationship and a second mapping relationship can be obtained.

[0049] In some embodiments of the present invention, the text editing panel may display text data to be processed, as well as at least one of the following: a video to be processed, a first timeline track corresponding to media video, a second timeline track corresponding to media audio, a text editing menu, and a global function menu.

[0050] The text editing menu may include at least one of the following controls: delete control for deletion operations, copy control for copy operations, cut control for cut operations, modify control for modification operations, intelligent short / long sentence control (intelligent short / long sentence replacement control), replace control, picture-in-picture add control, non-subtitle text add control, sticker add control, special effect add control, sound effect add control, filter add control, material package add control, and background music add control.

[0051] The global functions menu may include at least one of the following: a search control, a meaningless content removal control, and a subtitle show / hide control.

[0052] The meaningless part removal control is used to instruct the meaningless part removal process, which involves removing meaningless words, particles, silent segments, and other invalid words or segments from the text data to be processed, thereby ensuring the semantic consistency of the processed text data.

[0053] In some embodiments of the present invention, the text editing panel further includes an exit control that closes (exits) the text editing panel in response to a trigger operation on the exit control.

[0054] In some embodiments of the present invention, if ASR recognition of media audio is still running when the text editing panel is closed, the recognition is not interrupted and continues in the background.

[0055] In this embodiment, the editing process for the video to be processed can be achieved by displaying the text data to be processed on the interface and editing the text data to be processed.

[0056] 102: In response to a first trigger operation on the target text data among the text data to be processed, the target video segment corresponding to the target text data is determined from the media video based on the first mapping relationship.

[0057] The first trigger operation may be a click, slide, drag, or other operation. The specifics are not limited here and can be determined based on the actual situation.

[0058] To make it easier to understand, the timestamp of the target video segment corresponds to the timestamp of the target text data.

[0059] 103: Based on the second mapping relationship, determine the target audio segment corresponding to the target text data from the media audio.

[0060] To make it easier to understand, the timestamp of the target audio segment corresponds to the timestamp of the target text data.

[0061] 104: Based on the first trigger operation, the corresponding processing is performed on the target text data, target video segment, and target audio segment.

[0062] To make it easier to understand, based on the first trigger operation, the target text data, target video segment, and target audio segment are processed in accordance with the first trigger operation.

[0063] In the embodiments of this application, corresponding processing of media video and media audio is realized by performing corresponding processing on text data to be processed by a first trigger operation based on a first mapping relationship and a second mapping relationship, and further editing processing of the video to be processed is realized. Compared to editing operations on video frames, editing operations on text data are relatively simple and require less expertise from the creator, thus improving the editing efficiency of the video to be processed and enhancing the user experience for the creator.

[0064] In some embodiments of the present invention, the video to be processed further comprises at least one package object, wherein there is a mapping relationship between text timecode information and timeline information corresponding to each package object, and the object type of each package object comprises at least one of subtitle text, picture-in-picture, non-subtitle text, sticker, special effects, filters, material package, and background music.

[0065] A package object may correspond to complete text data to be processed (e.g., the package object is background music, subtitle text, picture-in-picture, etc.) or to only a portion of the text data to be processed (e.g., the package object is stickers, special effects, filters, etc.). The specifics are not limited here and can be determined on a case-by-case basis.

[0066] Taking package object 1 and package object 2, which are contained in at least one package object, as an example, there is mapping relationship 1 between the text timecode information and the timeline information corresponding to package object 1, and there is mapping relationship 2 between the text timecode information and the timeline information corresponding to package object 2.

[0067] Subtitle text is the text corresponding to the video content or media audio content to be processed. Non-subtitle text is text other than subtitle text.

[0068] Picture-in-picture is a method of displaying video content in which one video is played in full screen while another video is played in a smaller area of ​​the screen. It is widely used in televisions, video recorders, monitors, presentation devices, and other applications.

[0069] Stickers can include text, photos, images, etc.

[0070] Special effects can include visual special effects, sound special effects (abbreviated as sound effects), and others.

[0071] Filters are primarily used to achieve various special effects on images; for details, please refer to conventional related technologies.

[0072] Material packages are like small, combined templates that include elements such as screens, subtitles, and special effects. They can be added with a single click, and the elements can be modified as needed afterward.

[0073] Background music, also known as accompaniment music or soundtrack, typically refers to music used to enhance the atmosphere in television dramas, movies, anime, video games, websites, etc. By inserting it into dialogue, it enhances emotional expression and gives viewers a sense of presence as if they were actually there. In the embodiments of this application, background music is audio content distinct from media audio or sound effects.

[0074] In the embodiments of this invention, the video to be processed includes at least one package object, and there is a mapping relationship between the timeline information of each package object and the text timecode information of the text data to be processed. Therefore, in the process of editing some of the text data in the text data to be processed, it may be necessary to perform corresponding processing on package objects that match some of the text data. In the embodiments of this invention, the media video, media music, and at least one package object are extracted from the video to be processed, and the editing of the video to be processed is realized by performing corresponding processing on at least one of the media video, media music, and at least one package object in response to editing operations on the text data to be processed. Compared to editing based on the text data to be processed, editing the entire video to be processed makes the editing of the video to be processed more detailed and rational, the user operation is simple and easy to understand, and the user experience can be improved.

[0075] In some embodiments of the present application, the first trigger operation includes one of the following: delete, drag, copy, and cut operations, and the first trigger operation may further include, but is not limited to, other operations. In embodiments of the present application, the first trigger operation can support a variety of editing operations, making user operation simple and intuitive, and improving the user experience.

[0076] The invention relating to the embodiments of this disclosure has the following advantages over the prior art: Displaying text data to be processed corresponding to a video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio; determining a target video segment corresponding to the target text data from the media video based on the first mapping relationship in response to a first trigger operation on target text data among the text data to be processed; determining a target audio segment corresponding to the target text data from the media audio based on the second mapping relationship; and performing processing corresponding to the target text data, target video segment and target audio segment based on the first trigger operation. In the embodiments of this application, processing corresponding to the media video and media audio is achieved by performing processing corresponding to the text data to be processed by the first trigger operation based on the first and second mapping relationships, and further, editing processing of the video to be processed is achieved. Compared to editing video frames, editing text data is relatively simple and requires less expertise from the creator, thus improving the efficiency of video editing and enhancing the user experience.

[0077] In some embodiments of the present application, as shown in Figures 1 and 2, after step 101, the video processing method according to the embodiment of the present application may further include the following steps 105 to 107.

[0078] 105: In response to a first trigger operation on the target text data among the text data to be processed, determine the target package object corresponding to the target text data from at least one package object.

[0079] To make it clear, the target package object is at least one package object whose timestamp overlaps with the timestamp of the target text data.

[0080] 106: Based on the first trigger operation, if the object type of the target package object is one of the first types, perform the corresponding action on the target package object.

[0081] Type 1 includes at least one of the following: subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages.

[0082] 107: If the object type of the target package object is one of the second types, the corresponding processing on the target package object is prohibited.

[0083] The second type includes background music.

[0084] To make it easier to understand, the embodiments of this application distinguish between package object types: some package objects respond to editing of target text data, while others do not respond to editing operations on the target text data. This allows for a decision on whether or not to edit a package object in conjunction with it, depending on its type.

[0085] To make it clear, if the package object type is background music, editing it in conjunction with the target text data may result in inconsistent or repetitive background music, potentially failing to enhance emotional expression and create a sense of presence for the listener.

[0086] In some embodiments of the present application, the first type includes the third type and the fourth type, and step 106 above can be specifically implemented by step 106a or step 106b below.

[0087] 106a: If the object type of the target package object is one of the third types, the corresponding processing is performed on the target package segment of the target package object, which is the part of the target package object that corresponds to the target text data.

[0088] The third type includes at least one of subtitle text and picture-in-picture.

[0089] In the embodiments of this invention, the subtitle text and the picture-in-picture type (third type) package object typically correspond one-to-one with the media video, and can therefore be edited in conjunction with the target text data.

[0090] 106b: If the object type of the target package object is one of the fourth types, and the start timestamp of the target package object is within the timecode range corresponding to the target text data, the corresponding processing is performed on the entire target package object.

[0091] The timecode is divided into a time range between the timecode (timestamp) corresponding to the first semantic character in the target text data and the timecode corresponding to the last semantic character.

[0092] To make it easier to understand, the fact that the start timestamp of the target package object is within the timecode range corresponding to the target text data means that the start timestamp of the target package object is the same as a timestamp within that timecode range.

[0093] The fourth type includes at least one of the following: non-subtitled text, stickers, special effects, filters, and material packages.

[0094] In the embodiments of this application, package objects of non-subtitled text, stickers, special effects, filters, and material package types (fourth type) typically correspond to only a portion of the media video. If editing is performed directly in conjunction with the target text data without making a decision on whether to edit in conjunction with the target text data, the package objects may lack consistency, or one package object may become multiple package objects, potentially eliminating the original effect of the package objects and increasing the likelihood of negative effects. Therefore, in the embodiments of this application, if the object type of the target package object is one of the fourth types and the start timestamp of the target package object is within the timecode range corresponding to the target text data, the corresponding processing is performed on the entire target package object. That is, if it is decided to edit in conjunction with the target text data, the corresponding processing is performed on the entire package object. This ensures that when it is decided to edit in conjunction with the target text data, the corresponding processing is performed on the entire package object, thus preventing an increase in negative effects.

[0095] In some embodiments of the present application, after step 105, the video processing method according to the embodiment of the present application may further include the following step 108.

[0096] 108: If the object type of the target package object is one of the fourth types, and the start timestamp of the target package object is outside the timecode range corresponding to the target text data, the corresponding processing of the target package object is prohibited.

[0097] To make it easier to understand, if the object type of the target package object is one of the fourth types, and the start timestamp of the target package object is outside the timecode range corresponding to the target text data, then the corresponding processing of the target package object is prohibited. In other words, it is decided not to respond to editing in conjunction with the target text data (not to edit in conjunction with the target text data). This ensures that the package object has its intended effect.

[0098] In the embodiments of this invention, the modification operation affects the text data and media audio to be processed, and if subtitle text is present, it is modified accordingly, but it does not affect the media video. In other words, normally, the modification operation edits the media audio in conjunction with the text data to be processed, and if the video to be processed includes subtitle text, the subtitle text is edited in conjunction with the text data to be processed, but the media video is not edited in conjunction with the text data to be processed. As a result, when the text data to be processed is changed, the media audio and subtitle text are changed accordingly, but there is no need to change the media video, making the logic simpler, more practical, and providing a better user experience.

[0099] In some embodiments of the present application, after step 101, the video processing method according to the embodiment of the present application may further include steps 109 to 112 below.

[0100] 109: In response to a modification operation on the first text data in the text data to be processed, obtain the modified second text data corresponding to the modification operation.

[0101] 110: Based on the second mapping relationship, determine the first audio segment corresponding to the first text data from the media audio.

[0102] 111: A second audio segment is generated based on the second text data and the timbre information of the first audio segment.

[0103] In the embodiments of the present invention, by generating a second audio segment based on the second text data and the timbre information of the first audio segment, it is possible to ensure that the content of the second audio segment corresponds to the second text data, while also ensuring that the timbre information of the second audio segment is the same as that of the first audio segment. This ensures that not only is the media audio changed in accordance with changes in the text data to be processed, but the timbre information at the point where the media audio is changed is not changed, thereby improving the user experience.

[0104] 112: Replace the first audio segment with the second audio segment in media audio.

[0105] To make it easier to understand, the synchronization between the media audio and the first text data is achieved by replacing the first audio segment in the media audio with the second audio segment.

[0106] In some embodiments of the present invention, the video to be processed further includes subtitle text, and there is a third mapping relationship between the text timecode information and third timeline information corresponding to the subtitle text. After step 109 above, the video processing method according to the embodiments of the present invention may further include the following steps 113 to 115.

[0107] 113: In response to a modification operation on the first text data in the text data to be processed, the first subtitle segment corresponding to the first text data is determined from the subtitle text based on the third mapping relationship.

[0108] 114: Based on the second text data, modify the first subtitle segment to obtain the second subtitle segment.

[0109] To make it easier to understand, the second subtitle segment corresponds to the second text data, and the second audio segment also corresponds to the second text data, so the second subtitle segment also corresponds to the second audio segment. This ensures that when viewers watch, the subtitles will always correspond to the audio.

[0110] 115: Replace the first subtitle segment with the second subtitle segment in the subtitle text.

[0111] To facilitate understanding, the synchronization between the subtitle text and the first text data is achieved by replacing the first subtitle segment with the second subtitle segment in the subtitle text.

[0112] In some embodiments of the present application, if the media video timeline track and media audio timeline track are not displayed, the selection operation is performed only on text data, and the selected text data is displayed.

[0113] In some embodiments of the present invention, when media video timeline tracks and media audio timeline tracks are displayed, the selection operation selects and displays the selected text data, the corresponding track within the media video timeline tracks for the selected text data, and the corresponding track within the media audio timeline tracks for the selected text data. This allows the user to have a better visual experience and understand how the corresponding editing process is implemented.

[0114] In some embodiments of the present application, the first trigger operation further includes a selection operation, and prior to step 102, the video processing method according to an embodiment of the present application may further include the following step 116. Step 104 can be specifically implemented by the following steps 104a to 104c.

[0115] 116: Display the first timeline track and the second timeline track.

[0116] The first timeline track is for media video, and the second timeline track is for media audio.

[0117] 104a: Display the target text data according to the first display style.

[0118] 104b: Display the portion of the first timeline track corresponding to the target video segment, according to the second display style.

[0119] 104c: In accordance with the third display style, the portion of the second timeline track corresponding to the target audio segment is displayed.

[0120] The first, second, and third display styles may be partially the same, partially different, entirely the same, or entirely different, but are not limited to these.

[0121] In some embodiments of the present invention, when not only are media video timeline tracks and media audio timeline tracks displayed, but the timeline tracks for each package object are also displayed, the decision is made whether to select and display the timeline tracks for the package objects depending on the type of package object and whether the start timestamp of the package object is within the timecode range corresponding to the selected text data.

[0122] The editing track linkage logic in the embodiment of this application (specifically, the logic that corresponds to whether the above media video, media audio, and package object are edited in conjunction with the target text data) is: First, determine the start and end timecodes of the selected text, and then determine the target timecode range for the selected text. The system selects and displays tracks corresponding to the target timecode range in media video timeline tracks, media audio timeline tracks, picture-in-picture timeline tracks, and text subtitle timeline tracks, and responds to subsequent editing operations such as deletion, movement, and copying. When the start timestamp of a timeline track with non-text subtitles, a timeline track that supports stickers, etc., falls within the target timecode range, the entire corresponding track will be selected and displayed, and will respond to subsequent editing operations such as deletion, movement, and copy. This includes not always selecting and displaying the background music timeline track, and not responding to subsequent editing operations such as deletion, move, or copy.

[0123] In some embodiments of this application, the editing-time track linkage logic described above is the default, but it is also possible to set linkage on / off controls for different timeline tracks. The user can determine whether or not to link to the corresponding timeline track by triggering the linkage on / off control, and is not limited thereto.

[0124] In some embodiments of the present application, the method relating to the embodiments of the present application may further include steps 201 to 202 below.

[0125] 201: In response to a second trigger operation on selected text data from the text data to be processed, a tool menu is displayed in the area corresponding to the selected text data.

[0126] The Tools menu includes at least one of the Edit controls and the Add controls. The Edit controls include at least one of the following: a delete control for deletion operations, a copy control for copy operations, a cut control for cut operations, a change control for modification operations, an intelligent short / long text control, a search control, and a replace control. The Add controls include at least one of the following: a picture-in-picture add control, a non-subtitled text add control, a sticker add control, a special effects add control, a sound effect add control, a filter add control, a material package add control, and a background music add control.

[0127] All of the above additional controls can support users in either manually selecting and adding them or intelligently matching and adding them.

[0128] 202: In response to a third trigger operation on the target control in the Tools menu, the corresponding processing is performed on at least one of the media video, text data to be processed, and media audio.

[0129] In the embodiments of this invention, a second trigger operation on selected text data can display a tool menu. Then, as needed, a trigger operation on any control in the tool menu can execute the corresponding editing process, making interactive operation simple and convenient, and leading to an improved user experience.

[0130] In some embodiments of the present invention, the display area containing text data to be processed includes a material library control, which may perform text editing on the global video or rough cuts on the material library content.

[0131] In some embodiments of the present application, after step 101, the method according to the embodiment of the present application may further include the following steps 301 to 305.

[0132] 301: In response to a trigger operation on the material library control, display multiple pre-configured video materials.

[0133] 302: In response to a fifth trigger operation on the target video material from among multiple pre-configured video materials, a text rough cut floating window is displayed.

[0134] The Text Roughcut floating window contains material text data corresponding to the target video material, the target video material package includes the target video and target audio, there is a fourth mapping relationship between the material timecode information of the material text data and the timeline information of the target video, and a fifth mapping relationship between the material timecode information and the timeline information of the target audio.

[0135] 303: In response to a text rough cut operation on the selected source text from the source text data, the selected source text and the text data to be processed are combined to obtain the text data to be edited.

[0136] 304: Combine the source video segment and media video into the media video to be edited.

[0137] 305: Based on text editing operations on the text data to be edited, the source audio segment and media audio are combined into the media audio to be edited so that the video to be edited is edited.

[0138] The source video segment is a video segment corresponding to the selected source text obtained from the target source video based on the fourth mapping relationship, and the source audio segment is an audio segment corresponding to the selected source text obtained from the target source audio based on the fifth mapping relationship, and the video to be edited includes the media video to be edited and the media audio to be edited.

[0139] In embodiments of the present invention, the text editing panel may further include a material library control. A trigger operation on the material library control can display multiple pre-configured video materials. The user can select the required target material video from the multiple pre-configured video materials as needed and perform a text rough cut operation on the target material video to pull up the text rough cut floating window. The rough cut floating window displays material text data corresponding to the target material video. The user can then perform a text rough cut operation on the selected material text from the material text data to edit the material segment required by the user in the target material video to obtain a material video segment and a material audio segment. The selected material text and the text data to be processed are combined to obtain text data to be edited. Based on the text editing operation on the text data to be edited, the material video segment and media video are combined into the media video to be edited, and the material audio segment and media audio are combined into the media audio to be edited, so that editing of the video to be edited is realized.

[0140] In the embodiments of this invention, the video to be processed may be obtained by selecting at least one material from a material library and compositing multiple material editing segments obtained by text rough cuts. The details are not limited here and can be determined according to the actual situation.

[0141] In some embodiments of this application, any material in the material library can be selected, and a new text roughcut entry can be added by right-clicking the menu. Clicking the text roughcut entry invokes the text roughcut floating window, initiates ASR recognition of the selected material, and supports roughcutting the material in the text roughcut floating window and adding it to the timeline.

[0142] The Text Rough Cut floating window includes a floating window text preview area and may further include a floating window video preview area, a floating window timeline display area, etc. Further details can be found in the relevant explanations of the text editing panel and may be determined on a case-by-case basis, so they are not limited to those described here.

[0143] The ASR recognition results (material text data) of the selected material are displayed in the floating window text preview area. The specific form of the floating window text preview area that displays the material text data can be found by referring to the specific form of the text preview area in the text editing panel mentioned above, which displays the text data to be processed, so details are omitted here.

[0144] In text rough-cutting operations, you can either select the source text and drag the selected text to perform the rough-cutting operation, or select the source text, call up the rough-cutting control via email, and click the rough-cutting control. The specifics are not limited here and can be determined based on the actual situation.

[0145] In some embodiments of this invention, keywords are intelligently searched for in local materials, search results are selected, and text rough cutting is performed. The corresponding text is matched and selected by default, allowing the rough cutting to be completed quickly.

[0146] The system supports rough cutting of input text data via an editing floating window. Trigger operations on the input text data or matching video invoke the text rough cutting control, and trigger operations on the text rough cutting control invoke a new modal window. This panel supports rough cutting of the input text data and corresponding editing of the matching video. The specific process is described in detail in the process of achieving video editing based on editing operations on the text data to be processed, so details are omitted here.

[0147] In the embodiment of this invention, editing of a matching video corresponding to input text data is performed via an editing floating window, and the edited video to be processed and the edited matching video are combined to obtain the target edited video.

[0148] In some embodiments of the present invention, different display objects can be displayed in different areas of the text editing panel. For example, the text editing panel may consist of a video preview area for displaying the video to be processed, a text preview area for displaying the text data to be processed, a text editing menu area for displaying the text editing menu, a global functions area for displaying the global functions menu, and part or all of the timeline track display area.

[0149] In some embodiments of this application, when the text editing panel is pulled up, the default layout mode is used. Subsequently, a pre-configured layout mode (e.g., a mode menu such as "Work Mode" or "Panel Display Mode") can be selected to determine which partitions can be displayed in the text editing panel, although the display / hide of each partition can also be selected individually (checked). - >Display, uncheck ->Hide. For example, you can toggle the visibility of each panel partition using "√ / x" in the View menu. You can also drag panel partitions to make them float outside the editor. When dragging and restoring, panel partitions can only be returned to their corresponding positions in the default layout (i.e., free floating placement is not possible).

[0150] The text editing panel also includes options for "Restore," "Full Screen," "Minimize," and "Close."

[0151] The text preview area includes the text data to be processed, the text timecode information corresponding to the text data to be processed, and a timeline cursor. The text data to be processed includes text data corresponding to at least one user's voice, and the text data to be processed includes at least one text paragraph (including silent segments […]). The timeline cursor indicates the current playback progress corresponding to the video to be processed, corresponds to the current playback timeline of the video to be processed, and the corresponding text word or sentence is highlighted. The position of the timeline cursor and the highlighted text word or sentence are updated in real time as the video to be processed is played.

[0152] The text splitting logic in the embodiment of the present invention can split text according to the following priority order: Splitting by manual line breaks has a higher priority than splitting by the presenter, splitting by the presenter has a higher priority than splitting by pauses, and splitting by pauses has a higher priority than splitting by the number of words in a paragraph.

[0153] The silent segment logic in the embodiment of the present invention may include: Option 1: Treating silent segments of 1 second or longer as silent segments and displaying them in the text preview area of ​​the text editing panel; Option 2: Treating silent segments of 10 times or more the speech speed as silent segments and displaying them in the text preview area of ​​the text editing panel.

[0154] The positioning jump logic in the embodiment of the present invention may not respond when the text area is manually scrolled, but may only jump when a toggle selection is clicked.

[0155] In the embodiments of this invention, a trigger operation on selected text can bring up a tools menu, trigger any editing control in the tools menu, or directly trigger any editing control in the text editing menu to enter text editing mode and edit text and video.

[0156] When the text data to be processed is in preview mode (the default state), the cursor is at the playback point, and the currently read text word or sentence (regardless of whether it is paused or playing) is highlighted. At this time, you can enter edit mode and modify the content using a trigger operation. For example, a trigger operation on a selection control will select the currently highlighted sentence and enter edit mode.

[0157] If the text data to be processed contains selected text, it enters edit mode, allowing you to perform operations such as delete, insert (text template / picture-in-picture, etc.), modify, move, and copy.

[0158] During the delete operation, if text is selected, calling up the Tools menu will display the delete control. Selecting the delete control will delete the selected text and the track content corresponding to the selected text (the specific track to be deleted is determined by the track linkage logic), and the deletion will be completed, returning to the preview state.

[0159] In xxx (text template / picture-in-picture, etc.) insertion operations, if text is selected, calling up the Tools menu will display the xxx Add Controls, which may pull up the corresponding panel to show optional materials for package objects that can be added for the user to select themselves, or it may intelligently match the materials of the package objects that need to be added according to the selected text, but the details are determined on a case-by-case basis and are not limited to those details here.

[0160] After the user selects their desired material, or after smart material matching, the material overwrites the start timecode of the selected text on the new track. After addition, the result is not displayed in any form in the text preview area, but it is displayed in the video preview area (video preview window, video player), so it can only be edited on screen, and the insertion is complete and the editing state remains.

[0161] In the move operation, after selecting text, long-pressing the selection area allows you to drag the selected text paragraph to any position (but not within words), and the corresponding video segment will move with it. The move is then complete, and the editing state remains active.

[0162] During the copy operation, if text is selected, calling up the Tools menu will display the copy control. Selecting the copy control copies the text and track content (the specific track copied depends on whether track linking is enabled or disabled), and after pasting it into the selected text, the copy is complete and the editing state is maintained.

[0163] The modification operation affects the text data and media audio to be processed, and if subtitle text exists, it will be affected accordingly, but not the media video. When text is selected, calling up the Tools menu displays the modification controls. After selection, the input box is pulled up, the entire text data to be processed is entered into the input box, and the currently selected sentence is also selected by default. The results of this modification operation do not affect the media video, but only the text data and media audio (subtitle text) to be processed. Therefore, the response of the modification operation includes splitting the subtitle into two sentences by adding line breaks, deleting line breaks to make the subtitle into a single sentence, correcting typos and changing the subtitle accordingly, and in some cases changing the pronunciation of the media audio as well. Entering new text does not extend the video timecode, and the text modification results only affect the subtitle. Selecting the End control in the input box ends the modification operation, returns to the preview state, and the cursor is at the last input position.

[0164] In the embodiment of this invention, global functions such as search operations, meaningless portion removal operations, and subtitle display / hide operations can be performed in text editing mode.

[0165] The search operation involves entering text editing mode, selecting the search control, expanding the search box, entering search keywords, clicking "Confirm," highlighting matching text, and selecting the previous and next controls to quickly jump to search results. To close the search box, click the close control within the search box.

[0166] In the meaningless section removal operation, after entering text editing mode, the meaningless section removal control is selected to begin recognition of meaningless sections. Once recognition is complete, the meaningless sections are highlighted in the text preview area, and a (meaningless section removal) confirmation control is added at the bottom. Triggering the confirmation control enables one-click removal of meaningless sections. The user can verify whether the meaningless section recognition result is correct. If incorrect, they can select the individual meaningless section recall recovery control (which removes the corresponding highlight). To confirm, the user can select the confirmation control and continue the operation. Selecting the confirmation control deletes all highlighted meaningless sections and returns to text editing mode.

[0167] In some embodiments of the present application, a meaningless part removal operation triggers a full text removal. If a meaningless part removal process has already been performed, a second click responds as a reprocess.

[0168] When displaying or hiding subtitles, the subtitle content is linked to the text content, and the subtitle display / hide controls (which can include controls to show and hide subtitles) only control whether the subtitles are displayed or hidden. If you select the control to show subtitles, subtitles are added, and when the text data is updated, the subtitles are automatically updated in conjunction. If you select the control to hide subtitles, no subtitles are added.

[0169] In Example 1, because the display size of mobile devices is limited and most mobile devices are designed for non-expert users, the text editing panel on mobile devices can, by default, display the text preview area (displays the text data to be processed), the video preview area (displays the video to be processed), the text editing menu area (displays the text editing menu), and the global functions area (displays the global functions menu) as shown in Figure 3. The area indicated by mark "31" is the video preview area, the area indicated by mark "32" is the text preview area, the area indicated by mark "33" is the global functions area, and the area indicated by mark "34" is the text editing menu area. The first timeline track, the second timeline track, and timeline tracks corresponding to other package objects in the video to be processed can be controlled to be displayed in the text editing panel by corresponding trigger operations, as needed, and are not limited to those mentioned here.

[0170] In Example 2, non-mobile electronic devices can display three layout modes by default: the text preview area, the video preview area, and the timeline track display area, as shown in Figure 4. Alternatively, they can display four layout modes by default: the text preview area, the video preview area, the timeline track display area, and the parameter adjustment panel display area, as shown in Figure 5. The area indicated by mark "41" is the video preview area, the area indicated by mark "42" is the text preview area, the area indicated by mark "43" is the timeline track display area, and the area indicated by mark "51" is the parameter adjustment panel display area.

[0171] Figure 6 is a structural block diagram of a video processing apparatus according to an embodiment of the present disclosure. As shown in Figure 6, the video processing apparatus is A display module 601 for displaying text data to be processed corresponding to a video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, In response to a first trigger operation on the target text data among the text data to be processed, a method for determining the target video segment corresponding to the target text data from the media video based on a first mapping relationship, and, Based on the second mapping relationship, a determination module 602 is used to determine the target audio segment corresponding to the target text data from the media audio, The system includes a processing module 603 for performing corresponding processing on target text data, target video segments, and target audio segments based on a first trigger operation.

[0172] In some embodiments of the present invention, the video to be processed further comprises at least one package object, wherein there is a mapping relationship between text timecode information and timeline information corresponding to each package object, and the object type of each package object comprises at least one of subtitle text, picture-in-picture, non-subtitle text, sticker, special effects, filters, material package, and background music.

[0173] In some embodiments of the present application, the first trigger operation includes one of the following: delete operation, drag operation, copy operation and cut operation. The decision module 602 is also used to determine the target package object corresponding to the target text data from at least one package object in response to a first trigger operation on the target text data among the text data to be processed. The processing module 603 is also used to perform a corresponding process on a target package object if the object type of the target package object is one of a first type, which includes at least one of subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages, and to prohibit the performance of a corresponding process on the target package object if the object type of the target package object is one of a second type, which includes background music.

[0174] In some embodiments of the present application, the first type includes a third type which includes at least one of subtitle text and picture-in-picture, and a fourth type which includes at least one of non-subtitle text, stickers, special effects, filters and material packages. Specifically, the processing module 603 is used to perform processing on the target package segment, which is the part of the target package object that corresponds to the target text data, when the object type of the target package object is one of the third types, or to perform processing on the entire target package object when the object type of the target package object is one of the fourth types and the start timestamp of the target package object is within the time code range corresponding to the target text data.

[0175] In some embodiments of the present invention, the processing module 603 is also used to prohibit the processing of a target package object if the object type of the target package object is one of the fourth types and the start timestamp of the target package object is outside the timecode range corresponding to the target text data.

[0176] In some embodiments of the present invention, the apparatus further includes: an acquisition module for acquiring modified second text data corresponding to a modification operation in text data to be processed; a determination module 602 for determining a first audio segment corresponding to the first text data from media audio based on a second mapping relationship; a generation module for generating a second audio segment based on the second text data and timbre information of the first audio segment; and a replacement module for replacing the first audio segment in media audio with the second audio segment.

[0177] In some embodiments of the present invention, the video to be processed further includes subtitle text, and there is a third mapping relationship between the text timecode information and third timeline information corresponding to the subtitle text. The device includes a modification module for obtaining a second subtitle segment by modifying the first subtitle segment based on second text data, Furthermore, a determination module 602 for determining a first subtitle segment corresponding to the first text data from the subtitle text based on a third mapping relationship in response to a modification operation on the first text data in the text data to be processed, It further includes a replacement module for replacing the first subtitle segment with the second subtitle segment in the subtitle text.

[0178] In some embodiments of the present application, the first trigger operation further includes a selection operation. The display module 601 is also used to display a first timeline track, which is a timeline track corresponding to media video, and a second timeline track, which is a timeline track corresponding to media audio, before determining the target video segment corresponding to the target text data from the media video based on a first mapping relationship, in response to a first trigger operation on the target text data among the text data to be processed. Specifically, the processing module 603 is used to display the target text data according to a first display style, to display the portion of the first timeline track corresponding to the target video segment according to a second display style, and to display the portion of the second timeline track corresponding to the target audio segment according to a third display style.

[0179] In some embodiments of the present application, the display module 601 displays a tool menu in an area corresponding to selected text data in response to a second trigger operation on selected text data from the text data to be processed, wherein the tool menu includes at least one of an edit control and an add control, the edit control includes at least one of a delete control corresponding to a delete operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut operation, a change control corresponding to a change operation, an intelligent short / long text control, a search control, and a replace control, and the add control also includes at least one of a picture-in-picture add control, a non-subtitled text add control, a sticker add control, a special effect add control, a sound effect add control, a filter add control, a material package add control, and a background music add control. The processing module 603 is also used to perform a corresponding process on at least one of media video, text data to be processed, and media audio in response to a third trigger operation on the target control in the tool menu.

[0180] In some embodiments of the present application, the apparatus further includes a synthetic module. The display module 601 displays text data to be processed corresponding to the video to be processed, and then, in response to a trigger operation on the material library control, displays multiple pre-configured video materials, and in response to a fifth trigger operation on the target video material among the multiple pre-configured video materials, it displays a text rough cut floating window, the text rough cut floating window contains material text data corresponding to the target video material, the target video material package includes the target material video and the target material audio, there is a fourth mapping relationship between the material timecode information of the material text data and the timeline information of the target material video, and there is a fifth mapping relationship between the material timecode information and the timeline information of the target material audio. The compositing module, in response to a text rough-cut operation on selected material text from the material text data, combines the selected material text with the text data to be processed to obtain text data to be edited, and based on a text editing operation on the text data to be edited, it combines the material video segment and media video into the media video to be edited, and combines the material audio segment and media audio into the media audio to be edited, so that editing of the video to be edited is realized. The material video segment is a video segment corresponding to the selected material text obtained from the target material video based on a fourth mapping relationship, and the material audio segment is an audio segment corresponding to the selected material text obtained from the target material audio based on a fifth mapping relationship. The video to be edited is used to include the media video to be edited and the media audio to be edited.

[0181] In the embodiments of this disclosure, each module can implement the video processing method according to the above-described embodiment and achieve similar effects. Details are omitted here to avoid redundancy.

[0182] Figure 7 is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure, and is intended to exemplify an electronic device that implements any of the video processing methods in the embodiments of the present disclosure, and should not be understood as limiting the present disclosure to the embodiments.

[0183] As shown in Figure 7, the electronic device 700 may include a processor (e.g., a central processor, graphics processor, etc.) 701 which can perform various appropriate operations and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage device 708 into random access memory (RAM) 703. RAM 703 further stores various programs and data necessary for the operation of the electronic device 700. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0184] Generally, input devices 706, including, for example, touchscreens, touchboards, keyboards, mice, cameras, microphones, accelerometers, and gyroscopes; output devices 707, including, for example, liquid crystal displays (LCDs), speakers, and vibrators; storage devices 708, including, for example, magnetic tape and hard disks; and communication devices 709 may be connected to the I / O interface 705. The communication devices 709 can enable the electronic device 700 to communicate wirelessly or via wired means to exchange data with other devices. Figure 7 shows an electronic device 700 with various devices, but it should be understood that it is not necessary to implement or have all of the illustrated devices. Alternatively, more or fewer devices may be implemented or have been included.

[0185] In particular, according to embodiments of the present disclosure, the process described with reference to the flowchart above may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product which includes a computer program carried on a non-temporary computer-readable medium which includes program code for performing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it can perform functions limited to any of the video processing methods according to embodiments of the present disclosure.

[0186] The computer-readable medium described herein may be a computer-readable signal medium, a computer-readable storage medium, or a combination of both. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the above. More specific examples of computer-readable storage mediums may include, but are not limited to, electrical connections having one or more wires, portable computer magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact magnetic disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any appropriate combination of the above. In this disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device. In this disclosure, the computer-readable signal medium may include data signals propagating in the baseband or as part of a carrier, on which computer-readable program code is carried. Such propagating data signals may include, but are not limited to, electromagnetic signals, optical signals, or a suitable combination of the above, and may be in various forms. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, and this computer-readable signal medium may transmit, propagate, or transmit programs used by or in combination with instruction execution systems, apparatus, or devices. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, wires, cables, RF (radio frequency), or a suitable combination of the above.

[0187] In some embodiments, clients and servers may communicate using any currently known or future-developed network protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with digital data communications (e.g., communication networks) in any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), extranets (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), and any currently known or future-developed networks.

[0188] The computer-readable medium described above may be included in the electronic device described above, or it may exist separately without being incorporated into the electronic device.

[0189] The computer-readable medium described above carries one or more programs, and when the one or more programs are executed by this electronic device, the electronic device displays text data to be processed corresponding to the video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, and in response to a first trigger operation on target text data among the text data to be processed, the electronic device determines a target video segment corresponding to the target text data from the media video based on the first mapping relationship, determines a target audio segment corresponding to the target text data from the media audio based on the second mapping relationship, and performs processing corresponding to the target text data, target video segment and target audio segment based on the first trigger operation.

[0190] In embodiments of the Disclosure, computer program code for performing the operations of the Disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, Java®, Smalltalk, and the C++ object-oriented programming language, and further including, for example, the general procedural programming language "C" or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, run as a single standalone software package, run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or server. If a remote computer is involved, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, connected via the Internet through an Internet service provider).

[0191] The flowcharts and block diagrams in the drawings illustrate the implementable system architectures, functions, and operations of systems, methods, and computer program products relating to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for performing a defined logical function. It should be noted that in some alternative implementations, the functions marked in a block may occur in a different order than those marked in the drawings. For example, two consecutively represented blocks may actually be executed essentially in parallel, or in reverse order depending on their function. It should be noted that each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented in a dedicated system based on hardware that performs the defined function or operation, or in a combination of dedicated hardware and computer instructions.

[0192] The units described in the embodiments of this disclosure may be implemented in software or in hardware. The names of the units are not necessarily limited to the units themselves.

[0193] The functions described herein may be performed at least partially by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used non-limitingly include field-programmable gate arrays (FPGAs), dedicated integrated circuits (ASICs), dedicated standard products (ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0194] In the context of this disclosure, a computer-readable medium may be a tangible medium and may contain or store programs used by or in combination with an instruction execution system, device, or apparatus. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatus, or any appropriate combination of the above. More specific examples of computer-readable storage media include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any appropriate combination of the above.

[0195] The above description is merely a description of preferred embodiments and the technical principles used in the present disclosure. As those skilled in the art should understand, the scope of the disclosure is not limited to technical solutions formed by specific combinations of the above technical features, but should also include other technical solutions formed by arbitrarily combining the above technical features or their equivalents, without departing from the concept of the disclosure, such as technical solutions formed by substituting the above features with similar functional technical features disclosed in the present disclosure.

[0196] While each operation is illustrated using specific procedures, this should not be interpreted as requiring these operations to be performed in the specific illustrated procedures or order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of this disclosure. Some features described in the context of individual embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented individually or in any suitable subcombination in multiple embodiments.

[0197] While the present invention has been described using language specific to its structural features and / or methodological logic, it should be understood that the invention limited to the appended claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and behaviors described above are merely illustrative forms of realizing the claims.

Claims

1. Displaying text data to be processed corresponding to a video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, In response to a first trigger operation on the target text data among the text data to be processed, the target video segment corresponding to the target text data is determined from the media video based on the first mapping relationship, Based on the second mapping relationship described above, the target audio segment corresponding to the target text data is determined from the media audio, Based on the first trigger operation, the corresponding processing is performed on the target text data, the target video segment, and the target audio segment. A video processing method that includes [this].

2. The video to be processed further includes at least one package object, and there is a mapping relationship between the text timecode information and the timeline information corresponding to each package object. The object type of each of the aforementioned package objects is: Includes at least one of subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, material packages, and background music. The method according to claim 1.

3. The first trigger operation includes one of the following: delete operation, drag operation, copy operation, and cut operation. The aforementioned method, In response to a first trigger operation on the target text data among the text data to be processed, the target package object corresponding to the target text data is determined from the at least one package object, Based on the first trigger operation, if the object type of the target package object is one of the first types, which includes at least one of subtitle text, picture-in-picture, non-subtitle text, stickers, special effects, filters, and material packages, then the corresponding processing is performed on the target package object. If the object type of the target package object is one of the second types, including background music, then the corresponding processing of the target package object is prohibited, further comprising: The method according to claim 2.

4. The first type includes a third type which includes at least one of subtitle text and picture-in-picture, and a fourth type which includes at least one of non-subtitle text, stickers, special effects, filters and material packages. If the object type of the target package object is one of the first types, then the corresponding processing is performed on the target package object. If the object type of the target package object is one of the third types, then the corresponding processing is performed on the target package segment of the target package object, which is the part of the target package object that corresponds to the target text data. Or, If the object type of the target package object is one of the fourth types, and the start timestamp of the target package object is within the time code range corresponding to the target text data, the process includes performing the corresponding operation on the entire target package object. The method according to claim 3.

5. The aforementioned method, The further includes prohibiting the performance of a corresponding process on the target package object if the object type of the target package object is one of the four types and the start timestamp of the target package object is outside the timecode range corresponding to the target text data. The method according to claim 4.

6. The aforementioned method, In response to a modification operation on the first text data in the text data to be processed, the modified second text data corresponding to the modification operation is obtained. Based on the second mapping relationship, a first audio segment corresponding to the first text data is determined from the media audio, A second audio segment is generated based on the second text data and the timbre information of the first audio segment. Replacing the first audio segment in the media audio with the second audio segment, The method according to claim 1, further comprising:

7. The video to be processed further includes subtitle text, and there is a third mapping relationship between the text timecode information and the third timeline information corresponding to the subtitle text. The aforementioned method, In response to a modification operation on the first text data in the text data to be processed, a first subtitle segment corresponding to the first text data is determined from the subtitle text based on the third mapping relationship, Based on the second text data, the first subtitle segment is modified to obtain the second subtitle segment, Replacing the first subtitle segment in the subtitle text with the second subtitle segment, The method according to claim 6, further comprising:

8. The first trigger operation further includes a selection operation, and in response to the first trigger operation on the target text data among the text data to be processed, the method determines, based on the first mapping relationship, the target video segment corresponding to the target text data from the media video, The method further includes displaying a first timeline track which is a timeline track corresponding to the media video, and a second timeline track which is a timeline track corresponding to the media audio. Performing corresponding processing on the target text data, target video segment, and target audio segment based on the first trigger operation is: Display the target text data according to the first display style, In accordance with the second display style, the portion of the first timeline track corresponding to the target video segment is displayed, In accordance with the third display style, the portion of the second timeline track corresponding to the target audio segment is displayed, The method according to any one of claims 1 to 7, including the method described in any one of claims 1 to 7.

9. The aforementioned method, In response to a second trigger operation on selected text data from the text data to be processed, a tool menu is displayed in the area corresponding to the selected text data, wherein the tool menu includes at least one of editing controls and add controls, the editing controls include at least one of a delete control corresponding to a delete operation, a copy control corresponding to a copy operation, a cut control corresponding to a cut operation, a change control corresponding to a change operation, an intelligent short / long sentence control, a search control, and a replace control, and the add controls include at least one of a picture-in-picture add control, a non-subtitled text add control, a sticker add control, a special effect add control, a sound effect add control, a filter add control, a material package add control, and a background music add control. In response to a third trigger operation on the target control in the tool menu, the corresponding processing is performed on at least one of the media video, the text data to be processed, and the media audio. The method according to any one of claims 1 to 7, further comprising:

10. After displaying the text data to be processed corresponding to the video to be processed, the method proceeds as follows: In response to a trigger operation on the material library control, multiple pre-configured video materials will be displayed, In response to a fifth trigger operation on a target video material from among the plurality of pre-configured video materials, a text roughcut floating window is displayed, wherein the text roughcut floating window contains material text data corresponding to the target video material, the target video material package includes a target material video and a target material audio, there is a fourth mapping relationship between the material timecode information of the material text data and the timeline information of the target material video, and there is a fifth mapping relationship between the material timecode information and the timeline information of the target material audio. In response to a text rough-cut operation on selected material text from the material text data, the selected material text and the text data to be processed are combined to obtain text data to be edited; based on a text editing operation on the text data to be edited, the material video segment and the media video are combined into the media video to be edited, and the material audio segment and the media audio are combined into the media audio to be edited, wherein the material video segment is a video segment corresponding to the selected material text obtained from the target material video based on the fourth mapping relationship, the material audio segment is an audio segment corresponding to the selected material text obtained from the target material audio based on the fifth mapping relationship, and the video to be edited includes the media video to be edited and the media audio to be edited. The method according to any one of claims 1 to 7, further comprising:

11. A display module for displaying text data to be processed corresponding to a video to be processed, including media video and media audio, wherein there is a first mapping relationship between text timecode information corresponding to the text data to be processed and first timeline information corresponding to the media video, and a second mapping relationship between the text timecode information and second timeline information corresponding to the media audio, In response to a first trigger operation on the target text data among the text data to be processed, a method for determining the target video segment corresponding to the target text data from the media video based on the first mapping relationship, and A determination module for determining the target audio segment corresponding to the target text data from the media audio based on the second mapping relationship described above, A processing module for performing corresponding processing on the target text data, the target video segment, and the target audio segment based on the first trigger operation, A video processing device, including a video processing device.

12. An electronic device comprising: memory for storing a computer program; and a processor for executing the video processing method described in any one of claims 1 to 10 when the computer program is called.

13. A computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, enables the video processing method according to any one of claims 1 to 10.

Images