Subtitle processing method and device

Abstract

The present disclosure relates to a subtitle processing method and apparatus, the method including the steps of: performing speech recognition on audio corresponding to a multimedia material during editing of the multimedia material clip to obtain subtitle text corresponding to the audio and time stamp information of audio clips corresponding to each text element in the subtitle text; determining a material clip that matches the text element among the multimedia material clips based on the time stamp information of the audio clip corresponding to each text element; and synthesizing each text element with a material clip within a time corresponding to each of the text elements to obtain a target multimedia material having an animation effect in which the subtitle text pops out one character at a time. The embodiment of the present disclosure can realize a subtitle animation effect in which a corresponding text subtitle appears when a certain word is spoken, and further realizes automatic dynamic subtitle generation by inputting an instruction by the user, which is easy for the user to operate and favorable for improving the user experience.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to Chinese Patent Application No. 202211117721.1 filed on September 14, 2022, the contents of which are hereby incorporated by reference in their entirety into this application.

[0002] SUMMARY OF THE INVENTION An embodiment of the present disclosure relates to a subtitle processing method and apparatus. [Background technology]

[0003] Subtitles in videos serve as an aid to understanding the content of the video, and are therefore often added during video editing.

[0004] Currently, the typical method is to manually input subtitle text or to use a subtitle recognition tool to recognize the corresponding audio to obtain the subtitle text, then repeatedly listen to the audio to adjust and divide the subtitle text to obtain a large number of text clips, and then combine these text clips with the video to add subtitles to the video. For batch text scenes such as subtitles, if a user wants to achieve the intended subtitle effect, they must repeatedly adjust the subtitle text division results, combine them, and preview the subtitle effect, which makes subtitle editing inefficient. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the above technical problems, the present disclosure provides a subtitle processing method and device. [Means for solving the problem]

[0006] In a first aspect, embodiments of the present disclosure include: During editing of the multimedia material, performing speech recognition on audio corresponding to the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips respectively corresponding to each text element included in the subtitle text; a step of matching each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each of the text elements, and determining a material clip in the multimedia material that matches each of the text elements, wherein the time on an editing timeline of the material clip that matches the text element and the time on the editing timeline of the audio clip that corresponds to the text element are the same; and synthesizing each of the text elements with a material clip within a corresponding time range to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character.

[0007] In a second aspect, embodiments of the present disclosure include: a speech recognition module for performing speech recognition on audio corresponding to the multimedia material during editing of the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips respectively corresponding to each text element included in the subtitle text; a matching module that matches each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each of the text elements, and determines a material clip in the multimedia material that matches each of the text elements, wherein the time on an editing timeline of the material clip that matches the text element matches the time on the editing timeline of the audio clip that corresponds to the text element; and a subtitle synthesis module for synthesizing each of the text elements with a material clip within a matching time range to obtain a target multimedia material having an animation effect in which the subtitle text pops out character by character.

[0008] In a third aspect, embodiments of the present disclosure include: Provided is an electronic device including a memory and a processor, wherein the memory is configured to store computer program instructions, and the processor is configured to execute the computer program instructions to cause the electronic device to implement the subtitle processing method described in the first aspect.

[0009] In a fourth aspect, embodiments of the present disclosure include: There is provided a readable storage medium containing computer program instructions, the computer program instructions being executed by at least one processor of an electronic device to cause the electronic device to implement the subtitle processing method according to the first aspect.

[0010] In a fifth aspect, embodiments of the present disclosure include: An electronic device provides a computer program product that executes the computer program product to cause the electronic device to realize the subtitle processing method described in the first aspect. [Brief explanation of the drawings]

[0011] The drawings herein, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. In order to more clearly describe the technical solutions of the embodiments of the present disclosure, the drawings that need to be used for the embodiments are briefly described below, and it is obvious that those skilled in the art can obtain other drawings from these drawings without any creative efforts. [Figure 1] FIG. 1 is a flowchart of a subtitle processing method according to one embodiment of the present disclosure. [Figure 2]FIG. 2 is a flowchart of a subtitle processing method according to another embodiment of the present disclosure. [Figure 3] FIG. 3 is a flowchart of a subtitle processing method according to another embodiment of the present disclosure. [Figure 4A] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4B] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4C] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4D] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4E] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4F] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4G] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4H] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 4I] 4A-4I are schematic diagrams of a human-computer interaction interface according to the present disclosure. [Figure 5] FIG. 5 is a structural schematic diagram of a subtitle processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] In order to make the above-mentioned objects, features, and advantages of the present disclosure more clearly understood, the aspects of the present disclosure will be further described below. Note that the embodiments and features in the embodiments of the present disclosure may be combined with each other without contradiction.

[0013] Although numerous specific details are set forth in the following description to facilitate a thorough understanding of the present disclosure, the present disclosure may be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only some embodiments of the present disclosure, but not all embodiments.

[0014] Subtitles help users understand video content, and different subtitle effects can express more multidimensional content. For example, subtitle effects that display corresponding text when a word is spoken in a video's audio are often used to convey narration in a dramatic context or to express the speaker's confidence and excitement in a talking-type context. To achieve these specific subtitle effects, users often manually enter the subtitles, dividing the subtitle text into individual characters and then repeatedly listening to the audio to adjust them. Furthermore, users can enter complete sentences and use keyframe masks to create the effect of text appearing one by one. Subtitle editing is not only inefficient, but also extremely inconvenient to use on mobile devices.

[0015] Based on the above, an embodiment of the present disclosure provides a subtitle processing method and apparatus, which includes the steps of: performing speech recognition on audio corresponding to the multimedia material during editing of the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips corresponding to each text element in the subtitle text; determining material clips from the multimedia material clips that match the text elements based on the timestamp information of the audio clips corresponding to each text element; and combining each text element with the material clips within its corresponding time to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character. In the present disclosure, the start time of the time range of a video frame image matching a text element is the same as the start time of the audio clip corresponding to the text element, thereby achieving a subtitle animation effect in which the corresponding text subtitle appears when a word is spoken. Furthermore, dynamic automatic generation of subtitles can be achieved through user input, which simplifies user operation and improves user experience. Furthermore, the method of the present disclosure can be applied to various types of devices and has a wide range of applications.

[0016] The methods according to the present disclosure may be performed by electronic devices, which may be, but are not limited to, tablets, mobile phones (e.g., flip phones, large screen phones, etc.), wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and the present disclosure does not impose any limitations on the specific type of electronic device.

[0017] 1 is a flow diagram of a subtitle processing method according to an embodiment of the present disclosure. The method is described using an electronic device as an example, where an editing application is installed on the electronic device, and a user can edit multimedia material through the application. As shown in FIG. 1, the method of this embodiment includes steps S101 to S103.

[0018] S101: During editing of multimedia material, speech recognition is performed on audio corresponding to the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips respectively corresponding to each text element included in the subtitle text.

[0019] The multimedia material may be video material recorded by a user in real time, video material previously edited, or video material stored in an electronic device, and the present disclosure is not limited thereto, and may also be audio material, image material, etc., and the present disclosure is not limited to the type of multimedia material. Furthermore, the present disclosure is not limited to the number of multimedia materials, and when there are multiple multimedia materials, the multiple multimedia materials may be arranged according to the order of introduction and considered as one whole.

[0020] Editing of multimedia material can be understood as pre-recording or introducing audio material or audio material, or adding background music to multimedia material (e.g., video material or image material), but is not limited to these editing methods.

[0021] The subtitle text is obtained by recognizing text against audio corresponding to the multimedia material currently being edited, where the audio corresponding to the multimedia material may be the original audio contained in the multimedia material or background music added to the multimedia material by a user, and the background music may be audio within the application, such as a complete song, a segment of a song, or a cut audio clip, etc., and this disclosure is not limited thereto. If the multimedia material is audio material, the multimedia material itself may be speech-recognized.

[0022] In some embodiments, an application can send audio via an electronic device to a middle office service, which invokes a subtitle recognition tool to perform text recognition on the audio, and obtains timestamp information for the corresponding subtitle text and audio clips corresponding to each text element in the subtitle text, where the timestamp information may include a start time and an end time of the audio clip.

[0023] For example, the total audio duration of a multimedia material clip is 7 seconds, and the subtitle text obtained by speech recognition of the audio is:

number

[0024] [Table 1]

[0025] In the above example, if the audio is in Chinese, the corresponding text element is a word. If the audio is in another language, the text element will be the corresponding word, for example, if the audio is in English, the text element will be an English word.

[0026] In some examples, an application may perform speech recognition in response to a command entered by a user. This disclosure does not limit the implementation of the command that triggers speech recognition. In some examples, the command for speech recognition may include, but is not limited to, a type of action such as a click, double-click, long press, swipe, etc. For example, if a corresponding area / control for adding recognition subtitles to multimedia material is located on one page of the application, the command for speech recognition may be an action received on that area / control.

[0027] S102: Based on the timestamp information of the audio clips corresponding to each text element, matching is performed with each material unit in the multimedia material, and a material clip that matches each text element in the multimedia material is determined, and the time on the editing timeline of the material clip that matches the text element matches the time on the editing timeline of the audio clip corresponding to the text element.

[0028] If the multimedia material is an image or video material, the material clips that match the text elements of the multimedia material can be considered as image clips or video clips, and the material clips include image frames or video frames combined with the text elements. If the multimedia material is an audio material, the material clips that match the text of the multimedia material can be considered as audio clips, and the material clips include one or more phonetic pronunciation units combined with the text elements.

[0029] The subtitle processing method of the present disclosure aims to achieve a subtitle effect in which a corresponding text subtitle appears when a certain word is spoken, and therefore when a matching material clip is determined based on the timestamp information of the audio clip corresponding to each text element, the time on the editing timeline of the audio clip corresponding to the text element matches the time on the editing timeline of the material clip.

[0030] Here, matching on the editing timeline means that the start time on the editing timeline of the material clip that matches the text element matches the start time on the editing timeline of the audio clip that corresponds to the text element.

[0031] On the other hand, the time at which a text element in a subtitle disappears is flexible and may disappear at the end of the corresponding audio clip, or when the sentence (or text clip of a specified length) to which it belongs reaches the end position, or after a preset time has elapsed after the end of the corresponding audio clip, but the present disclosure is not limited thereto.

[0032] Therefore, in the editing timeline, the end time of the time corresponding to the material clip that matches a text element may be the same as the end time of the audio clip that corresponds to that text element, thereby creating an animation effect in which the subtitle text pops out character by character, and when the corresponding audio clip reaches its end time, the previously appearing text element disappears.

[0033] The end time of the time range to which the video frame image matching a text element belongs may be later than the end time of the audio clip corresponding to this text element, so that the text elements are displayed one by one, and the previously displayed text element is held for a predetermined time after the corresponding audio clip ends and then disappears.

[0034] Here, the speed at which the text elements switch depends on the speaking rate of the target in the audio.

[0035] S103: Each text element is composited with a material clip within a corresponding time range to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character.

[0036] When combining the text element with a matching material clip, a first subtitle animation style preset for the text element can be used, and the subtitles automatically added to the multimedia material will automatically retain the subtitle effect corresponding to the first subtitle animation style when generated, thereby satisfying the user's requirements for subtitle effects and reducing the user's subsequent operations. The first subtitle animation style may include one or more of the start style, end style, and loop style of the text element.

[0037] Steps S102 and S103 may be performed automatically by calling a dynamic subtitle resource package (also called a subtitle animation resource package), in which the subtitle text and timestamp information of each text element contained in the subtitle text are transmitted to the dynamic subtitle resource package, which then collectively applies a preset subtitle animation style to each text element and superimposes the text elements having the preset subtitle animation style on the matching material clips, thereby adding subtitles with subtitle text using an animation effect in which each character pops out using a first subtitle animation style to the multimedia material.

[0038] The method of this embodiment can realize a subtitle animation effect in which a corresponding text subtitle appears when a certain word is spoken. Furthermore, dynamic subtitles can be automatically generated through user input, which simplifies user operation and is beneficial for improving the user experience. The method of this embodiment can also be applied to various types of devices and has a wide range of applications. In batch text scenes, subtitles with specified effects can be quickly added to multimedia material clips, even for mobile devices with small screens.

[0039] After adding subtitles to a multimedia material clip using the method of the embodiment shown in Figure 1, a user may re-edit the content of the subtitle text, including but not limited to deleting text elements, inserting new text elements, and replacing text elements. Figure 2 is a flowchart of a subtitle processing method according to another embodiment of the present disclosure. As shown in Figure 2, the method of this embodiment further includes steps S104 to S108 in addition to the steps of the embodiment shown in Figure 1.

[0040] S104: In response to the text deletion command, the corresponding text element is deleted from the subtitle text to obtain updated subtitle text.

[0041] Here, by deleting the text elements in the subtitle text and retaining the remaining text elements and the timestamp information of the remaining text elements, the updated subtitle text and the timestamp information of each text element in the updated subtitle text can be obtained.

[0042]

number

[0043]

number

[0044] If text elements at other positions are to be deleted, they can be processed in a similar manner.

[0045] S105: In response to the text insertion command, a new text element is inserted into the subtitle text to obtain an updated subtitle text.

[0046] The text insertion performed in this step does not remove existing text elements in the subtitle text, but inserts a new text element. In some embodiments, different processing may be configured depending on the insertion location of the new text element. In some embodiments, if the new text element is inserted in the middle of the subtitle text or at the end of the text, the new text element is merged with the previous adjacent text element and shares the timestamp of the audio clip corresponding to the previous adjacent text element. If the new text element is inserted at the beginning of the subtitle text, the new text element is merged with the first text element in the subtitle text and shares the timestamp of the audio clip corresponding to the first text element.

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052] S106: In response to a text replacement command, one or more text elements in the subtitle text are replaced with replacement text to obtain updated subtitle text.

[0053] During replacement, the timestamp information corresponding to the replacement text is equivalent to the timestamp information of the audio clip corresponding to the text element to be replaced. In one replacement, the replacement text may include one or more text elements, and the replacement text may be understood as a whole, and the number of text elements to be replaced may be one or more text elements in consecutive positions.

[0054]

number

[0055]

number

[0056]

number

[0057] To edit the subtitle text, select one or more of the above editing methods as needed.

[0058] S107: Based on the timestamp information of the audio clips corresponding to the text elements in the updated subtitle text, material clips that match the text elements are determined from the multimedia material.

[0059] S108: Composite each of the text elements with a corresponding in-time material clip so as to re-subtitle the multimedia material.

[0060] Steps S107 to S108 are the same as steps S102 and S103 in the embodiment shown in FIG. 1, respectively, and the detailed description of the embodiment shown in FIG. 1 can be referred to.

[0061] When this is automatically achieved by invoking a dynamic subtitle resource package, the updated subtitle text and the timestamp information of each text element contained in the updated subtitle text are retransmitted to the dynamic subtitle resource package, and the dynamic subtitle resource package collectively applies a preset subtitle animation style to each text element contained in the updated subtitle text, and re-adds the subtitles to the multimedia material by superimposing the text elements having the preset subtitle animation style on the matching material clips.

[0062] The method of this embodiment can meet the user's need to adjust the subtitle content when adding subtitles to multimedia materials, and can automatically generate subtitles with specified subtitle effects for the updated subtitle text, which is convenient for users and favorable for improving the user experience.

[0063] After adding subtitles to multimedia material using the method of the embodiment shown in Figure 1, the user can also adjust the subtitle animation style used by the current subtitles to achieve the subtitle effect that meets the user's expectations. Figure 3 is a flowchart of a subtitle processing method according to another embodiment of the present disclosure. As shown in Figure 3, the method of this embodiment includes steps S301 to S304.

[0064] S301: During editing of multimedia material, speech recognition is performed on audio corresponding to the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips corresponding to each text element included in the subtitle text.

[0065] S302: Based on the timestamp information of the audio clips corresponding to each text element, matching is performed with each material unit in the multimedia material, and a material clip that matches each text element in the multimedia material is determined, and the time on the editing timeline of the material clip that matches the text element matches the time on the editing timeline of the audio clip corresponding to the text element.

[0066] S303: A specified first subtitle animation style is applied to each text element at once, and the text elements having the first subtitle animation style are composited with a material clip within a time period that matches the first subtitle animation style to obtain a target multimedia material having subtitle text using an animation effect in which each character pops out using the first subtitle animation style.

[0067] In this embodiment, steps S301 to S302 are respectively the same as steps S101 to S103 in the embodiment shown in Fig. 1, and the detailed description of the embodiment shown in Fig. 1 may be referred to, and will not be described in detail here. Note that the first subtitle animation style can be understood as the default subtitle animation style of the application.

[0068] S304: In response to a subtitle animation style switching command, the second subtitle animation style is applied to each text element in a batch, and the text elements having the second subtitle animation style are composited with a material clip within a time period that matches the second subtitle animation style to obtain a target multimedia material having subtitle text using an animation effect in which each character pops out using the second subtitle animation style.

[0069] Here, the application may provide a subtitle animation style editing page to the user via the electronic device, and this page may display areas or controls corresponding to one or more subtitle animation styles that the user can select, and the user may input a subtitle animation style switching command by operating the area or control corresponding to the subtitle animation style.

[0070] When this is automatically achieved by invoking a dynamic subtitle resource package, the updated subtitle text and the timestamp information of each text element contained in the updated subtitle text are retransmitted to the dynamic subtitle resource package, and the dynamic subtitle resource package collectively reapplies the second subtitle animation style specified by the user to each text element contained in the updated subtitle text, and re-adds the subtitles to the multimedia material by superimposing the text elements having the second subtitle animation style on the material clips that match them.

[0071] The method of this embodiment can meet the user's need for adjusting the subtitle effect later, support the batch editing of subtitle animation styles, and achieve high efficiency in subtitle processing.

[0072] Based on the above description, the subtitle processing method according to the present disclosure will be described with reference to drawings and application scenarios, taking an electronic device as an example of an embodiment of the present disclosure. For ease of explanation, FIGS. 4A to 4K show an example in which the electronic device is a mobile phone and a video editing application (abbreviated as application 1) is installed on the mobile phone. Furthermore, the multimedia material that the user introduces into application 1 is video material.

[0073] As shown in FIGS. 4A-4I, FIGS. 4A-4I are schematic diagrams of a human-computer interaction interface according to an embodiment of the present disclosure.

[0074] The application 1 can exemplarily display a user interface 11 on the mobile phone for displaying a multimedia material editing page (hereinafter referred to as editing page) as shown in FIG. 4A, and the application 1 can display the editing page for performing a set of functions such as preview playback of clip effects of the multimedia material, adding background music to the multimedia material, and adding filters, stickers, and text to the multimedia material.

[0075] 4A, the user interface 11 includes an area a1 that is a preview area for clip effects of the multimedia material, and an area a2 that can display the multimedia material and other clip materials added during editing along a time axis. The user interface 11 further includes an area a3 that can provide the user with multiple types of clip function entries. For example, the area a3 includes a control 101 for accessing a text function collection page of the application 1, which includes multiple controls corresponding to different text functions.

[0076] For example, when the application 1 accepts a user's operation such as clicking on control 101 in the user interface 11 shown in FIG. 4A, the application 1 can exemplarily display on the mobile phone a user interface 12 shown in FIG. 4B, which shows a text function collection page provided by the application 1, in which entries of various different text functions are provided to the user, and the user can access the corresponding text function operation page via the entry to add text content to the multimedia material.

[0077] The user interface 12 includes an area a4 containing entries corresponding to a new text creation function, a text template function, a subtitle recognition function, a lyric recognition function, a sticker function, and a doodle pen function, respectively. Here, the control 102 shown on the user interface 12 is an entry corresponding to the subtitle recognition function.

[0078] When the application 1 accepts a user's operation such as clicking control 102 in the user interface 12 shown in FIG. 4B, the application 1 can exemplarily display a user interface 13 shown in FIG. 4C on the mobile phone, which shows a subtitle recognition panel provided by the application 1, and in the subtitle recognition panel, the user can be provided with a recognition type option, a language type selection entry, a switch to mark invalid segments, a switch for dynamic subtitles, and a switch to simultaneously empty existing subtitles.

[0079] Here, dynamic subtitles refers to the function of adding subtitles to multimedia material with an animation effect in which the subtitle text pops out character by character. Specifically, when the dynamic subtitles switch is in the off state, the subtitle effect presented by the added subtitles is that sentence segments appear as single subtitles, and when the dynamic subtitles switch is in the on state, the added subtitles provide the effect of the subtitle text popping out character by character, i.e., the text elements in the subtitle text are displayed one by one, and the text elements appear at the start of the corresponding audio clip.

[0080] In some embodiments, the user's selection can be remembered, and the dynamic subtitle switch state when the user last finished subtitle recognition can be displayed when the subtitle recognition panel is opened, to better suit the user's usage habits. When the application 1 updates the dynamic subtitle function for the first time, the dynamic subtitle switch may be in the off state, as shown in the user interface 13.

[0081] When the user performs an operation such as clicking the dynamic subtitle switch button on the user interface 13 shown in FIG. 4C, the application 1 displays a user interface 14 in which the dynamic subtitle switch is in the ON state, as shown in FIG. 4D.

[0082] The user interface 14 also includes a control 103 for instructing the start of voice recognition and adding subtitles with an animation effect in which each character pops out. In response to a user performing an operation such as clicking the control 103 on the user interface 14, the application 1 exemplarily displays a user interface 15 on the mobile phone as shown in FIG. 4E , in which the subtitle recognition panel is turned off and hint content such as animation and hint text is displayed in the region a4 to indicate that the user is currently creating a dynamic subtitle animation. To reduce the occlusion of the hint animation and hint text on the preview screen displayed in the region a1, the region a4 may be located above the region a1, but it should be understood that the region a4 may be located at another position, and the present disclosure is not limited thereto.

[0083] Here, as mentioned above, when the user operates the dynamic subtitle switch and control 103, the application 1 performs speech recognition on the audio corresponding to the multimedia material and automatically adds dynamic subtitles with an animation effect in which each character pops out.

[0084] Once the creation of the dynamic subtitle animation is completed, the application 1 can exemplarily display the user interface 16 shown in FIG. 4F on the mobile phone, and the user interface 16 can display hint content such as a message in area a4 saying, "Recognition successful. Subtitles have been automatically generated."

[0085] Then, the user can click the preview play button to preview the subtitle effect in area a1, and if it meets the user's expectations, the edited multimedia material can be exported as a target video for distribution or storage.

[0086] 4A to 4F, the present disclosure provides a dynamic subtitle switch to the user prior to the subtitle recognition process, making it easier for the user to use. In addition, by storing the switch state of the dynamic subtitle switch when the user last exits the subtitle recognition panel, the user does not need to perform any additional operations when reusing the dynamic subtitle switch, which has the effect of reusing the switch and eliminates the need for the user to perform any additional operations.

[0087] To better meet the needs of users, the application 1 provides users with the ability to add dynamic subtitles or modify existing subtitle animation styles in subsequent steps.

[0088] 4F, area a2 displays recognizers corresponding to the multimedia material and the subtitle text along a time axis, and subtitle re-editing can be triggered by operating (e.g., clicking) the recognizer of the subtitle text displayed in area a2. Application 1 can display user interface 17 shown in FIG. 4G on the mobile phone by accepting a user's click operation on a text clip included in any of the subtitles in area a2 of user interface 16.

[0089] In user interface 17, region a1 displays text box 104 corresponding to subtitle text. The text box 104 contains text content, which may be one or more sentences (i.e., a text clip), corresponding to the current preview position. Region a1 may also display controls for manipulating the text box, such as rotating, copying, and allowing the user to enlarge or reduce the size of the text box via a two-finger trigger. The size of the text elements in the text box also changes as the size of the text box changes. User interface 17 also includes region a5, which displays a subtitle editing function set page that provides entries for various editing functions for editing currently added subtitles. For example, the page may include entries corresponding to functions such as batch editing of subtitles, subtitle splitting, subtitle copy, subtitle editing, subtitle deletion, cursive writing, and subtitle animation style. Here, user interface 17 includes control 105 for accessing a subtitle animation panel to add subtitle effects (including dynamic subtitle effects) for the current subtitle or to change the subtitle animation style used in the current subtitle.

[0090] When the application 1 accepts that the user has performed an operation such as clicking on the control 105 in the user interface 17, it displays the user interface 18 including the area a6 shown in FIG. 4H.

[0091] Here, the user displays a subtitle animation panel in area a6, which includes tags 106 for setting animation styles, font tags, style tags, cursive tags, text template tags, etc. In some embodiments, as shown in FIG. 4H, upon entering the subtitle animation style panel, the user may be navigated to tab 106 by default, and related content for tab 106 may be displayed. In another embodiment, the user may be able to navigate to other tags, and application 1 may display related content for tag 106 when the user clicks on tag 106.

[0092] As shown in FIG. 4H, area a6 further includes a dynamic subtitle switch 107, and by operating this dynamic subtitle switch 107, a subtitle effect in which text elements are displayed one by one can be added to the current subtitle.

[0093] In some embodiments, if the user added dynamic subtitles in the previous stage, this may be displayed as an on state here, and if the user did not use dynamic subtitles in the previous stage, this may be displayed as an off state here, and the user may switch the switch state of dynamic subtitle switch 107 displayed in user interface 18 to the on state. Here, in the embodiment shown in Figure 4H, dynamic subtitle switch 107 is in the off state.

[0094] Furthermore, area a6 includes tag 108 for setting a subtitle start style, tag 109 for setting a subtitle end style, tag 110 for setting a subtitle loop style, tag 111 for setting a video subtitle animation style, and area a7 for displaying the content of the corresponding tag depending on the tag currently located. In some cases, when dynamic subtitle switch 107 is in the off state, any tag-related content may be displayed by default; for example, user interface 17 shown in FIG. 4H may display the related content corresponding to tag 108 by default.

[0095] Application 1 accepts a user operation (e.g., a click) on dynamic subtitle switch 107 of user interface 18. When dynamic subtitle switch 107 is switched from an off state to an on state, user interface 19 as shown in FIG. 4I can be displayed on the mobile phone. As shown in FIG. 4I, in user interface 19, dynamic subtitle switch 107 is on, tab 111 is selected, and area a7 displays one or more user-selectable dynamic subtitle animation styles related to dynamic subtitles. Display identifiers corresponding to the multiple dynamic subtitle animation styles are arranged in order from left to right, and the user can check them back and forth by sliding the screen left and right. Here, the default dynamic subtitle animation style of application 1 is displayed in the initial position from left to right, allowing the user to clearly understand the dynamic subtitle animation style used by application 1 by default.

[0096] The area a7 may further include a disable button 112, which may be located at the leftmost position of the area a7 or, of course, may be located at another position, but the present disclosure is not limited thereto. When the user clicks the disable button 112 to turn off the corresponding dynamic subtitle effect, the dynamic subtitle switch 107 is switched to the off state.

[0097] If the user clicks on the second dynamic subtitle animation style from the left in area a7, this will constitute an instruction to switch subtitle animation styles to application 1. In response to the instruction to switch subtitle animation styles, application 1 will apply the second dynamic subtitle style to each text element included in the subtitle text. The user can switch between dynamic subtitle animation styles multiple times until they achieve the subtitle effect they expect.

[0098] In addition to the user interface 18 shown in Fig. 4H and the user interface 19 shown in Fig. 4I, the region a5 includes a region a8 displaying a text editing box that allows the user to delete text elements in the subtitle text, insert new text, or replace the original text elements, and the user's operations on the text editing box correspond to inputting delete, insert, and replace instructions to the application 1. When editing the text content in the text editing box of the region a8, the edited text content is displayed in synchronization with the text box 104 shown in the region a1, which is advantageous for the user to preview the edited subtitle content and the display effect of the subtitle content in the video frame image of the multimedia material clip.

[0099] In the embodiment shown in Figures 4F to 4I above, by setting the dynamic subtitle switch and the dynamic subtitle animation style tag in the subtitle animation style panel at a later stage, the user's needs for adding dynamic subtitles and adjusting the subtitle dynamic style used for dynamic subtitles at a later stage can be met.

[0100] It should be understood that the schematic diagrams of the interactive interfaces shown in Figures 4A to 4I above do not limit the subtitle processing method according to the present disclosure, and some controls, panels, tag styles, trigger methods, etc. can be flexibly adjusted as needed.

[0101] 5 is a structural schematic diagram of a subtitle processing device according to an embodiment of the present disclosure. As shown in FIG. 5, the device 500 according to this embodiment includes: a speech recognition module 501 for performing speech recognition on audio corresponding to the multimedia material during editing of the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips corresponding to each text element included in the subtitle text; a matching module 502 that matches each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each of the text elements, and determines a material clip in the multimedia material that matches each of the text elements, wherein the time on an editing timeline of the material clip that matches the text element and the time on the editing timeline of the audio clip that corresponds to the text element are the same; and a subtitle synthesis module 503 for synthesizing each of the text elements with a material clip within a corresponding time range to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character.

[0102] In some embodiments, the start time of the time to which the material clip matching the text element belongs in the editing timeline coincides with the start time of the audio clip corresponding to the text element, and the end time of the time to which the material clip matching the text element belongs in the editing timeline coincides with the end time of the audio clip corresponding to the text element, or the end time of the time to which the material clip matching the text element belongs is later than the end time of the audio clip corresponding to the text element.

[0103] In some embodiments, the subtitle synthesis module 503 specifically applies a specified first subtitle animation style to each of the text elements collectively, and synthesizes the text elements having the first subtitle animation style with a matching in-time material clip to obtain the target multimedia material with subtitle text using the first subtitle animation style 1 with a character-by-character pop-up animation effect.

[0104] Optionally, the device 500 further includes a subtitle text update module 504 .

[0105] In some embodiments, the subtitle text update module 504 responds to a text deletion command by deleting the text element from the subtitle text to obtain updated subtitle text.

[0106] In such a case, the matching module 502 also determines material clips of the multimedia material that match each of the text elements based on the timestamp information of the audio clips that correspond to each of the text elements in the updated subtitle text.

[0107] The subtitle synthesis module 503 also synthesizes each of the text elements contained in the updated subtitle text with a corresponding in-time material clip so as to re-add an animation effect in which the subtitle text pops out character by character to the multimedia material.

[0108] In some embodiments, the subtitle text update module 504 also inserts new text elements into the subtitle text in response to a text insertion command to obtain updated subtitle text.

[0109] In such a case, the matching module 502 also matches each material unit in the multimedia material based on the timestamp information of the audio clip corresponding to each text element in the updated subtitle text, determines a material clip in the multimedia material that matches each text element, and the new text element and its adjacent text elements are combined to share the timestamp information of the audio clip corresponding to the adjacent text element.

[0110] The subtitle synthesis module 503 also combines each of the text elements contained in the updated subtitle text with a corresponding material clip within a time range, so as to re-add an animation effect in which the subtitle text pops out character by character to the multimedia material.

[0111] In some embodiments, if the insertion position of the new text element is at the beginning of the subtitle text, the new text element and a first text element in the subtitle text will share the timestamp of the audio clip corresponding to the first text element, and if the insertion position of the new text element is in the middle or at the end of the subtitle text, the new text element and a previous adjacent text element will share the timestamp of the audio clip corresponding to the previous text element.

[0112] In some embodiments, the subtitle text update module 504 is also responsive to a text replacement command to replace one or more text elements in the subtitle text with replacement text to obtain updated subtitle text.

[0113] In such a case, the matching module 502 also determines material clips of the multimedia material that match each of the text elements based on the timestamp information of the audio clips corresponding to each of the text elements in the updated subtitle text, and the replacement text corresponds to the timestamp information of the audio clips corresponding to the text elements to be replaced.

[0114] The subtitle synthesis module 503 also synthesizes each of the text elements contained in the updated subtitle text with a corresponding in-time material clip so as to re-add an animation effect in which the subtitle text pops out character by character to the multimedia material.

[0115] In some embodiments, the subtitle synthesis module 503 also, in response to the subtitle animation style switch command, applies a second subtitle animation style to each of the text elements collectively, and composites the text elements having the second subtitle animation style with material clips within a matching time range to obtain target multimedia material with subtitle text using a character-by-character pop-up animation effect using the second subtitle animation style.

[0116] In some embodiments, the audio corresponding to the multimedia material is the original audio included in the multimedia material or background music added to the multimedia material.

[0117] The subtitle processing device according to this embodiment may implement the technical solutions of any of the method embodiments described above, and their implementation principles and technical effects are similar, so reference may be made to the detailed descriptions of the method embodiments described above, and for the sake of brevity, they will not be described in detail here.

[0118] Illustratively, the present disclosure provides an electronic device that includes one or more processors, a memory, and one or more computer programs, the one or more computer programs being stored in the memory, and that, when the one or more processors execute the one or more computer programs, causes the electronic device to realize the subtitle processing method of the above-described embodiment.

[0119] Illustratively, the present disclosure provides a chip system that is applied to an electronic device including a memory and a sensor, and includes a processor, and the processor executes the subtitle processing method of the above embodiment.

[0120] Illustratively, the present disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes an electronic device to implement the subtitle processing method of the above embodiment. Illustratively, the present disclosure provides a computer program product that, when implemented on a computer, causes the computer to perform the subtitle processing method of the above embodiment.

[0121] It should be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another and do not necessarily require or imply that such an actual relationship or order exists between those entities or operations. Furthermore, the terms "comprise," "contain," or any other variation thereof are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that includes a set of elements includes not only those elements but also other elements not expressly listed or that are inherent in such process, method, article, or apparatus. Absent further limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of further identical elements in the process, method, article, or apparatus that includes that element.

[0122] The foregoing are merely specific embodiments of the present disclosure that will enable those skilled in the art to understand or practice the present disclosure. Various modifications to these examples will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other examples without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the examples described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A subtitle processing method, comprising: during editing of the multimedia material, performing speech recognition on audio corresponding to the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips respectively corresponding to each text element included in the subtitle text, the text element being the smallest semantic unit of a corresponding language; a step of matching each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each of the text elements, and determining a material clip in the multimedia material that matches each of the text elements, wherein the time on an editing timeline of the material clip that matches the text element and the time on the editing timeline of the audio clip that corresponds to the text element are the same; and combining each of the text elements with a corresponding in-time material clip to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character; The method comprises: inserting a new text element into the subtitle text in response to a text insertion command to obtain an updated subtitle text; a step of matching each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each text element in the updated subtitle text, and determining a material clip in the multimedia material that matches each of the text elements, wherein the new text element and its adjacent text elements are combined to share timestamp information of the audio clip corresponding to the adjacent text element; and combining each of the text elements included in the updated subtitle text with a material clip within a corresponding time range so as to re-add an animation effect of the subtitle text popping out character by character to the multimedia material; If the insertion position of the new text element is at the beginning of the subtitle text, the new text element and a first text element in the subtitle text are combined to share a timestamp of an audio clip corresponding to the first text element; a timestamp of an audio clip corresponding to the previous text element is shared with the new text element and the previous text element adjacent to the new text element when the insertion position of the new text element is in the middle or at the end of the subtitle text;

2. 2. The method of claim 1, wherein, in the editing timeline, the start time of a time to which a material clip matching the text element belongs coincides with the start time of an audio clip corresponding to the text element, and, in the editing timeline, the end time of a time to which a material clip matching the text element belongs coincides with the end time of the audio clip corresponding to the text element, or the end time of a time to which a material clip matching the text element belongs is later than the end time of the audio clip corresponding to the text element.

3. The step of synthesizing each of the text elements with a corresponding in-time material clip to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character, 3. The method of claim 1, further comprising applying a specified first subtitle animation style to each of the text elements collectively, and compositing the text elements having the first subtitle animation style with a matching in-time material clip to obtain the target multimedia material having subtitle text with a character-by-character pop-up animation effect using the first subtitle animation style.

4. in response to a text deletion command, deleting the text element from the subtitle text to obtain an updated subtitle text; determining material clips from the multimedia material that match each of the text elements based on timestamp information of audio clips that correspond to each of the text elements in the updated subtitle text; 3. The method of claim 1, further comprising: combining each of the text elements included in the updated subtitle text with a corresponding in-time material clip so as to re-add an animation effect of the subtitle text popping out character by character to the multimedia material.

5. in response to a text replacement command, replacing one or more text elements in the subtitle text with replacement text to obtain updated subtitle text; determining a material clip from the multimedia material that matches each of the text elements based on timestamp information of an audio clip corresponding to each of the text elements in the updated subtitle text, wherein the replacement text corresponds to timestamp information of an audio clip corresponding to the text element to be replaced; 3. The method of claim 1, further comprising: combining each of the text elements included in the updated subtitle text with a corresponding in-time material clip so as to re-add an animation effect of the subtitle text popping out character by character to the multimedia material.

6. 4. The method of claim 3, further comprising the step of: in response to a subtitle animation style switch command, applying a second subtitle animation style to each of the text elements collectively; and compositing the text elements having the second subtitle animation style with material clips within a matching time range to obtain a target multimedia material having subtitle text with a character-by-character pop-up animation effect using the second subtitle animation style.

7. 3. The method of claim 1, wherein the audio corresponding to the multimedia material is original audio included in the multimedia material or background music added to the multimedia material.

8. A subtitle processing device, a speech recognition module for performing speech recognition on audio corresponding to the multimedia material during editing of the multimedia material to obtain subtitle text corresponding to the audio and timestamp information of audio clips respectively corresponding to each text element included in the subtitle text, the text element being the smallest semantic unit of a corresponding language; a matching module that matches each material unit in the multimedia material based on timestamp information of an audio clip corresponding to each of the text elements, and determines a material clip in the multimedia material that matches each of the text elements, wherein the time on an editing timeline of the material clip that matches the text element matches the time on the editing timeline of the audio clip that corresponds to the text element; a subtitle synthesis module for synthesizing each of the text elements with a material clip within a corresponding time range to obtain a target multimedia material with an animation effect in which the subtitle text pops out character by character; a subtitle text update module that, in response to a text insertion command, inserts a new text element into the subtitle text to obtain an updated subtitle text; The matching module is further used to match each material unit in the multimedia material based on timestamp information of audio clips corresponding to each text element in the updated subtitle text, and to determine material clips matching each of the text elements in the multimedia material, and the new text element and its adjacent text elements are combined to share timestamp information of audio clips corresponding to the adjacent text elements; the subtitle synthesis module is further adapted to combine each of the text elements included in the updated subtitle text with a corresponding material clip within a corresponding time range, so as to re-add an animation effect of the subtitle text popping out character by character to the multimedia material; If the insertion position of the new text element is at the beginning of the subtitle text, the new text element and a first text element in the subtitle text are combined to share a timestamp of an audio clip corresponding to the first text element; A subtitle processing device that, when the insertion position of the new text element is in the middle or at the end of the subtitle text, shares the timestamp of the audio clip corresponding to the previous text element with the new text element and the previous text element adjacent to it.

9. An electronic device, a memory and a processor, the memory configured to store computer program instructions; The processor executes the computer program instructions to cause the electronic device to implement the subtitle processing method according to claim 1 or 2.

10. A readable storage medium, containing computer program instructions, A readable storage medium, in which at least one processor of an electronic device executes the computer program instructions to cause the electronic device to implement the subtitle processing method according to claim 1 or 2.

11. 1. A computer program product comprising: A computer program product that causes an electronic device to execute the computer program product and realize the subtitle processing method according to claim 1 or 2.

Images