Multi-line dynamic subtitle adding method and device, medium and program product

Abstract

The invention relates to a multi-line dynamic subtitle adding method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an initial video, and carrying out audio-video separation on the initial video to obtain an initial audio and a separated video; performing subtitle recognition on the initial audio to obtain a subtitle recognition result; acquiring subtitle template data; subtitle effect setting is carried out according to the subtitle template data and the subtitle recognition result, and a multi-row subtitle effect sequence is obtained; and carrying out sequence fusion on the multi-row subtitle effect sequence and the separated video to obtain a target video. By the adoption of the method, a user only needs to input one video, the multi-line dynamic subtitle effect that multiple lines of subtitles exist at the same time, are not mutually overlapped in space and have various effects can be automatically achieved, and subtitle adding convenience is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a multi-action subtitle adding method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of video subtitle technology, multi-line video subtitle technology appears to realize the simultaneous appearance of multi-action subtitles on the screen. However, the subtitles have appearance and stay time in the video, and the subtitles occupy the position of the screen. If the video wants to add multi-action subtitles, it is difficult to grasp the time relationship and position relationship between the subtitles. In the prior art, the key frame method is usually used to make animation to realize the effect of multiple subtitles appearing in time overlap but not in space overlap. SUMMARY

[0003] Therefore, the present application provides a multi-action subtitle adding method and device, computer equipment, computer readable storage medium and computer program product, which can automatically add multi-action subtitles to the input video, realize the simultaneous existence of multi-line subtitles, and achieve various multi-action subtitle effects without spatial overlap.

[0004] In one aspect, the present application provides a multi-action subtitle adding method, comprising:

[0005] obtaining an initial video and performing audio and video separation on the initial video to obtain an initial audio and a separated video;

[0006] performing subtitle recognition on the initial audio to obtain a subtitle recognition result;

[0007] obtaining subtitle template data;

[0008] setting a subtitle effect according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence;

[0009] performing sequence fusion on the multi-line subtitle effect sequence and the separated video to obtain a target video.

[0010] In one embodiment, the subtitle recognition on the initial audio to obtain a subtitle recognition result comprises:

[0011] performing speech recognition on the initial audio to obtain a speech recognition result;

[0012] performing subtitle analysis on the speech recognition result to obtain subtitle data;

[0013] performing word analysis on the subtitle data to obtain vocabulary data; and the subtitle recognition result comprises the subtitle data and the vocabulary data.

[0014] In one of the embodiments, the subtitle template data includes:

[0015] The subtitle frame data is acquired, and the subtitle frame size and the subtitle position are determined according to the subtitle frame data;

[0016] The subtitle line data is acquired, and the subtitle line spacing, the subtitle line and column position, and the subtitle style data are determined according to the subtitle line data; the subtitle template data includes the subtitle frame size, the subtitle position, the subtitle line spacing, the subtitle line and column position, and the subtitle style data.

[0017] In one of the embodiments, the subtitle recognition result includes the subtitle data and the vocabulary data; the subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain the multi-line subtitle effect sequence includes:

[0018] The start time position calculation is performed according to the subtitle template data and the subtitle data to obtain the appearance time position data of the subtitle data on the time axis;

[0019] The real-time position effect calculation is performed on the subtitle unit of the subtitle data according to the subtitle template data, the subtitle data, and the vocabulary data to obtain the position effect data of the subtitle unit on the time axis following the time change;

[0020] The disappearance effect calculation is performed on the subtitle unit according to the subtitle template data and the subtitle data to obtain the disappearance time effect data of the subtitle unit on the time axis;

[0021] The subtitle effect setting is performed according to the appearance time position data, the position effect data, and the disappearance time effect data to obtain the multi-line subtitle effect sequence.

[0022] In one of the embodiments, the real-time position effect calculation on the subtitle unit of the subtitle data according to the subtitle template data, the subtitle data, and the vocabulary data to obtain the position effect data of the subtitle unit on the time axis following the time change includes:

[0023] The real-time position calculation is performed on the subtitle unit according to the subtitle template data and the subtitle time data of the subtitle data to obtain the real-time subtitle position data;

[0024] The real-time position calculation is performed on the vocabulary of the subtitle unit according to the vocabulary time data of the subtitle template data and the vocabulary data to obtain the real-time vocabulary position data;

[0025] determine real-time animation effect and font style data of the subtitle unit according to the subtitle template data; the position effect data comprises: the real-time subtitle position data, the font style data, the real-time word position data and the real-time animation effect.

[0026] In one of the embodiments, the method further comprises:

[0027] determine the disappearing time data of the subtitle unit according to the subtitle data;

[0028] determine the disappearing animation effect according to the disappearing time data and the subtitle template data; the disappearing time effect data comprises: the disappearing time data and the disappearing animation effect.

[0029] In one of the embodiments, the method further comprises:

[0030] obtain a user editing instruction;

[0031] edit the subtitle template data according to the user editing instruction to obtain edited subtitle template data;

[0032] update the subtitle effect according to the edited subtitle template data and the subtitle recognition result to obtain a plurality of updated subtitle effect sequences;

[0033] sequence fuse the plurality of updated subtitle effect sequences with the separated video to obtain an updated video.

[0034] In one aspect, the application further provides a multi-action dynamic subtitle adding device, comprising:

[0035] a first obtaining module, configured to obtain an initial video and separate audio and video of the initial video to obtain an initial audio and a separated video;

[0036] a subtitle recognition module, configured to perform subtitle recognition on the initial audio to obtain a subtitle recognition result;

[0037] a second obtaining module, configured to obtain subtitle template data;

[0038] a subtitle effect setting module, configured to set subtitle effect according to the subtitle template data and the subtitle recognition result to obtain a plurality of subtitle effect sequences;

[0039] a sequence fusion module, configured to sequence fuse the plurality of subtitle effect sequences with the separated video to obtain a target video.

[0040] In an aspect, the present application also provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0041] obtaining an initial video, and performing audio-video separation on the initial video to obtain an initial audio and a separated video;

[0042] performing subtitle recognition on the initial audio to obtain a subtitle recognition result;

[0043] obtaining subtitle template data;

[0044] performing subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence;

[0045] performing sequence fusion of the multi-line subtitle effect sequence and the separated video to obtain a target video.

[0046] In an aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0047] obtaining an initial video, and performing audio-video separation on the initial video to obtain an initial audio and a separated video;

[0048] performing subtitle recognition on the initial audio to obtain a subtitle recognition result;

[0049] obtaining subtitle template data;

[0050] performing subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence;

[0051] performing sequence fusion of the multi-line subtitle effect sequence and the separated video to obtain a target video.

[0052] In an aspect, the present application also provides a computer program product comprising a computer program, which obtains an initial video, and performs audio-video separation on the initial video to obtain an initial audio and a separated video;

[0053] performing subtitle recognition on the initial audio to obtain a subtitle recognition result;

[0054] obtaining subtitle template data;

[0055] performing subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence;

[0056] performing sequence fusion of the multi-line subtitle effect sequence and the separated video to obtain a target video.

[0057] The multi-action dynamic subtitle adding method, device, computer device, computer readable storage medium and computer program product described above, by obtaining an initial video, and performing audio and video separation on the initial video to obtain an initial audio and a separated video, performing subtitle recognition on the initial audio to obtain a subtitle recognition result, obtaining subtitle template data, performing subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence, and performing sequence fusion of the multi-line subtitle effect sequence and the separated video to obtain a target video. Therefore, by implementing an automatic processing mode of multi-action dynamic subtitles, multi-line subtitle effects are described in a subtitle template data, and a user only needs to select a subtitle template to generate multi-line real-time changing subtitle effect display on a screen. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0059] Figure 1 An application environment diagram of the multi-action dynamic subtitle adding method in an embodiment;

[0060] Figure 2 A flowchart of the multi-action dynamic subtitle adding method in an embodiment;

[0061] Figure 3 A flowchart of the multi-action dynamic subtitle adding method in another embodiment;

[0062] Figure 4 A structural block diagram of the multi-action dynamic subtitle adding device in an embodiment;

[0063] Figure 5 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions and beneficial effects of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0065] The multi-action dynamic subtitle adding method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains the initial video, separates the audio and video of the initial video to obtain the initial audio and the separated video, identifies the subtitles of the initial audio to obtain the subtitle recognition result, obtains the subtitle template data, sets the subtitle effect according to the subtitle template data and the subtitle recognition result to obtain the multi-line subtitle effect sequence, and sequence fusion The multi-line subtitle effect sequence and the separated video obtain the target video.

[0066] Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0067] In an exemplary embodiment, as Figure 2 shown, a multi-action dynamic subtitle adding method is provided, which is applied to Figure 1 the server 104 in the above-mentioned application environment for example, including the following steps 202 to 210. Among them:

[0068] Step 202, obtaining an initial video, and separating the audio and video of the initial video to obtain an initial audio and a separated video.

[0069] Among them, the initial video can be a video of different video formats, not limited to this.

[0070] In some embodiments, the initial video can be obtained by clicking the video file, or can be obtained from an online material database, not limited to this.

[0071] In some embodiments, the initial audio can be obtained by splitting the audio track of the initial video, and the separated video can be obtained by splitting the video track of the initial video, so as to realize the audio and video separation of the initial video, not limited to this.

[0072] Step 204, identifying the subtitles of the initial audio to obtain a subtitle recognition result.

[0073] The subtitle recognition processing is a process of speech recognition and analysis processing on the text information in the audio.

[0074] In some embodiments, the subtitle recognition on the initial audio obtains a subtitle recognition result, including: performing speech recognition on the initial audio to obtain a speech recognition result; performing subtitle analysis on the speech recognition result to obtain subtitle data; performing word analysis on the subtitle data to obtain vocabulary data; and the subtitle recognition result including the subtitle data and the vocabulary data.

[0075] The subtitle data includes the start time and the end time of each line of subtitles, and the vocabulary data includes the start time and the end time of each word of each line of subtitles.

[0076] In some embodiments, the initial audio is subjected to speech recognition by a pre-trained speech recognition model to obtain a speech recognition result containing all the text content, or the initial audio is subjected to speech recognition by other means.

[0077] In some embodiments, the speech recognition result is subjected to subtitle analysis by a pre-trained subtitle analysis model to obtain subtitle data, or the speech recognition result is subjected to subtitle analysis by a preset sentence time interval, common punctuation keywords, and maximum word limit to accurately punctuate the text content in the speech recognition result and divide each line of subtitle content and the start time and the end time of each line of subtitles.

[0078] In some embodiments, the subtitle data is subjected to word analysis by a pre-trained word analysis model to obtain vocabulary data, or the subtitle data is subjected to word analysis by a preset common vocabulary library to accurately identify each word in the subtitle data and divide the words contained in the subtitle data and the start time and the end time of each word of each line of subtitles.

[0079] Step 206: Obtain subtitle template data.

[0080] The subtitle template data is template data for animation effects on continuous lines of subtitles.

[0081] In some embodiments, the subtitle template data is obtained by: obtaining subtitle box data and determining the size and position of the subtitle box according to the subtitle box data; obtaining subtitle line data and determining the line spacing, line and column positions, and style data of the subtitles according to the subtitle line data; and the subtitle template data including the size, position, line spacing, line and column positions, and style data of the subtitles.

[0082] The subtitle box data is data for determining the size and position of the subtitle box.

[0083] In some embodiments, the subtitle template can be directly obtained by selecting the subtitle template selected by the user in the subtitle template type selection interface, and the subtitle frame data of the subtitle template can be obtained. The subtitle frame data can also be obtained by directly inputting the subtitle frame data in the custom template by the user. The subtitle frame data of the subtitle frame edited by the user can be obtained. If the user does not select the subtitle frame data, the default subtitle frame data can be obtained, and the like.

[0084] In some embodiments, the subtitle position can be determined according to the subtitle frame center coordinate data of the subtitle frame data, and the subtitle frame size can be determined according to the subtitle frame coordinate data and the boundary data of the subtitle frame data, and the like.

[0085] The subtitle frame size refers to the size of the entire subtitle frame, and the subtitle position refers to the position of the subtitle on the screen.

[0086] In some embodiments, the subtitle line data can be directly obtained by selecting the subtitle template selected by the user in the subtitle template type selection interface, and the subtitle line data of the subtitle template can be obtained. The subtitle line data can also be obtained by directly inputting the subtitle line data in the custom template by the user. The subtitle line data of the subtitle line edited by the user can be obtained. If the user does not select the subtitle line data, the default subtitle line data can be obtained, and the like.

[0087] In some embodiments, the distance between the subtitle lines can be determined according to the line spacing data of the subtitle line data, the subtitle line spacing and the subtitle line column position can be obtained, the font, the font size and the animation effect of each subtitle line can be determined according to the font data and the animation data of the subtitle line data, and the subtitle style data can be obtained. The font and the animation effect of the subtitle line can be different for each subtitle.

[0088] Step 208, subtitle effect setting according to the subtitle template data and the subtitle recognition result, to obtain a multi-line subtitle effect sequence.

[0089] In some embodiments, the subtitle recognition result includes: subtitle data and vocabulary data; the subtitle effect setting according to the subtitle template data and the subtitle recognition result, to obtain a multi-line subtitle effect sequence, includes: calculating the starting time position according to the subtitle template data and the subtitle data, to obtain the appearance time position data of the subtitle data on the time axis; calculating the real-time position effect of the subtitle unit of the subtitle data according to the subtitle template data, the subtitle data and the vocabulary data, to obtain the position effect data of the subtitle unit on the time axis following the change of time; calculating the disappearance effect of the subtitle unit according to the subtitle template data and the subtitle data, to obtain the disappearance time effect data of the subtitle unit on the time axis; and setting the subtitle effect according to the appearance time position data, the position effect data and the disappearance time effect data, to obtain the multi-line subtitle effect sequence.

[0090] In some embodiments, the occurrence position and related display effect of the subtitle data on the time axis are quickly determined according to the subtitle box size, subtitle position, subtitle line spacing, subtitle row and column position and subtitle style data of the subtitle template data, and the occurrence time of the subtitle data on the time axis is determined according to the subtitle data, so as to quickly locate the occurrence time position data of the subtitle data on the time axis.

[0091] In some embodiments, real-time position effect calculation is performed on the subtitle unit of the subtitle data according to the subtitle template data, the subtitle data and the vocabulary data, to obtain position effect data of the subtitle unit on the time axis following the change of time, including: performing real-time position calculation on the subtitle unit according to the subtitle time data of the subtitle template data and the subtitle data, to obtain real-time subtitle position data; performing real-time position calculation on the vocabulary of the subtitle unit according to the vocabulary time data of the subtitle template data and the vocabulary data, to obtain real-time vocabulary position data; determining real-time animation effect and font style data of the subtitle unit according to the subtitle template data; the position effect data includes: the real-time subtitle position data, the font style data, the real-time vocabulary position data and the real-time animation effect.

[0092] In some embodiments, the number of subtitle units that can be accommodated in the subtitle box is determined in real time according to the subtitle box size, subtitle line spacing, subtitle row and column position and font size data in the subtitle style data of the subtitle template data, and the corresponding number of subtitle units at the current time is selected from all the subtitle units according to the subtitle time data, and the real-time subtitle position of the selected subtitle unit is determined.

[0093] In some embodiments, the switching mode of the subtitle is determined according to the subtitle style data of the subtitle template data, if the subtitle is in a scrolling mode in the subtitle style data, the vocabulary of the subtitle unit will be scrolled in the subtitle box one after another, at this time, the time when the vocabulary of the subtitle unit appears in the subtitle box and the real-time vocabulary position calculation are needed to be confirmed in real time according to the vocabulary time data of the vocabulary data, and finally the time when the vocabulary leaves the subtitle box, so as to determine the real-time vocabulary position data, if the subtitle is in a switching mode in the subtitle style data, the real-time position calculation is directly performed on the vocabulary of the subtitle unit according to the vocabulary time data of the subtitle template data and the vocabulary data, to obtain the real-time vocabulary position data.

[0094] In some embodiments, the font, font size and animation effect of each row of subtitle units are determined according to the subtitle style data of the subtitle template data, and the position where the first subtitle unit of the subtitle row and column appears, and the new display position, font and animation effect of the originally displayed subtitle row and column when the subsequent subtitle row and column appear, so as to determine the real-time animation effect and font style data of the subtitle unit.

[0095] In some embodiments, the disappearance effect of the subtitle unit is calculated based on the subtitle template data and the subtitle data to obtain the disappearance time effect data of the subtitle unit on the timeline, including: determining the disappearance time data of the subtitle unit based on the subtitle data; determining the disappearance animation effect based on the disappearance time data and the subtitle template data; the disappearance time effect data includes: disappearance time data and disappearance animation effect.

[0096] In some embodiments, the position of the subtitle lines is determined based on the subtitle frame size, subtitle line spacing, subtitle line and column position, and font size data in the subtitle style data of the subtitle template data. When it is determined that the subtitle lines that have appeared have been partially or completely squeezed out of the subtitle frame, the disappearance time data of the subtitle unit is calculated, and the real-time disappearance animation effect is determined based on the disappearance time data and the subtitle style data of the subtitle template data.

[0097] Step 210: Merge the multi-line subtitle effect sequence with the separated video to obtain the target video.

[0098] In some embodiments, the real-time dynamically changing multi-line subtitle effect sequence is fused with the separated video on the same timeline to obtain the target video with real-time dynamically changing subtitles, presenting the effect specified by the template.

[0099] The above method for adding multi-line dynamic subtitles involves: acquiring an initial video and performing audio-video separation to obtain initial audio and separated video; performing subtitle recognition on the initial audio to obtain subtitle recognition results; acquiring subtitle template data; setting subtitle effects based on the subtitle template data and subtitle recognition results to obtain a multi-line subtitle effect sequence; and merging the multi-line subtitle effect sequence with the separated video to obtain the target video. Therefore, by implementing an automated processing method for multi-line dynamic subtitles, describing multi-line subtitle effects within a single subtitle template, users only need to select a subtitle template to generate multi-line, real-time changing subtitle effects on the screen.

[0100] In one exemplary embodiment, such as Figure 3 As shown, the template editing steps include steps 302 to 308. Wherein:

[0101] Step 302: Obtain user editing instructions.

[0102] The user editing instructions can be click-based interaction instructions or data input instructions, and are not limited to these.

[0103] In some embodiments, the user editing instruction can be obtained by clicking the corresponding editing interaction button or directly editing the template effect, and obtaining the template data corresponding to the editing interaction button or the template update data as the user editing instruction. Alternatively, the user editing instruction can be obtained by directly inputting the editing data by the user, and is not limited thereto.

[0104] At step 304, the subtitle template data is edited according to the user editing instruction to obtain edited subtitle template data.

[0105] In some embodiments, the user editing instruction can be used to edit the size and position of the subtitle box in the subtitle template data, and the font and animation effect of each subtitle line in the subtitle template data can be edited differently. The font and animation effect can be modified by the user editing instruction to obtain the edited subtitle template data.

[0106] At step 306, the subtitle effect is updated according to the edited subtitle template data and the subtitle recognition result to obtain a multi-line updated subtitle effect sequence.

[0107] In some embodiments, the subtitle effect is updated according to the edited subtitle template data and the subtitle recognition result, i.e., the subtitle effect is reconfigured, to obtain an updated multi-line updated subtitle effect sequence.

[0108] At step 308, the multi-line updated subtitle effect sequence is sequentially fused with the separated video to obtain an updated video.

[0109] In some embodiments, the multi-line updated subtitle effect sequence that changes dynamically in real time is sequentially fused with the separated video on the same time axis to obtain an updated video with real-time dynamic changes in the subtitle, which presents the updated subtitle presentation effect of the template.

[0110] It should be understood that although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiments of the present application also provide a multi-action dynamic subtitle adding device for implementing the multi-action dynamic subtitle adding method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-action dynamic subtitle adding device embodiments provided below can refer to the limitations of the multi-action dynamic subtitle adding method described above, which will not be repeated here.

[0112] In an exemplary embodiment, as shown in Figure 4 A multi-action dynamic subtitle adding device is provided, comprising: a first acquisition module 401, a subtitle recognition module 402, a second acquisition module 403, a subtitle effect setting module 404, and a sequence fusion module 405, wherein:

[0113] The first acquisition module 401 is configured to acquire an initial video and perform audio and video separation on the initial video to obtain an initial audio and a separated video.

[0114] The subtitle recognition module 402 is configured to perform subtitle recognition on the initial audio to obtain a subtitle recognition result.

[0115] The second acquisition module 403 is configured to acquire subtitle template data.

[0116] The subtitle effect setting module 404 is configured to perform subtitle effect setting according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence.

[0117] The sequence fusion module 405 is configured to perform sequence fusion on the multi-line subtitle effect sequence and the separated video to obtain a target video.

[0118] In some embodiments, the subtitle recognition module 402 is further configured to perform speech recognition on the initial audio to obtain a speech recognition result, perform subtitle analysis on the speech recognition result to obtain subtitle data, and perform word analysis on the subtitle data to obtain vocabulary data, wherein the subtitle recognition result comprises the subtitle data and the vocabulary data.

[0119] In some embodiments, the second acquisition module 403 is further configured to acquire subtitle box data and determine a subtitle box size and a subtitle position according to the subtitle box data, acquire subtitle line data and determine a subtitle line spacing, a subtitle line and column position, and subtitle style data according to the subtitle line data, and the subtitle template data comprises the subtitle box size, the subtitle position, the subtitle line spacing, the subtitle line and column position, and the subtitle style data.

[0120] In some embodiments, the subtitle recognition result includes: the subtitle data and the vocabulary data; the subtitle effect setting module 404 is further configured to: perform start time position calculation on the subtitle data according to the subtitle template data and the subtitle data, to obtain appearance time position data of the subtitle data on a time axis; perform real-time position effect calculation on a subtitle unit of the subtitle data according to the subtitle template data, the subtitle data and the vocabulary data, to obtain position effect data of the subtitle unit on the time axis following a change in time; perform disappearance effect calculation on the subtitle unit according to the subtitle template data and the subtitle data, to obtain disappearance time effect data of the subtitle unit on the time axis; and perform subtitle effect setting according to the appearance time position data, the position effect data and the disappearance time effect data, to obtain a plurality of lines of subtitle effect sequences.

[0121] In some embodiments, the subtitle effect setting module 404 is further configured to: perform real-time position calculation on a subtitle unit according to the subtitle template data and subtitle time data of the subtitle data, to obtain real-time subtitle position data; perform real-time position calculation on a vocabulary of the subtitle unit according to the subtitle template data and vocabulary time data of the vocabulary data, to obtain real-time vocabulary position data; determine real-time animation effect and font style data of the subtitle unit according to the subtitle template data; and the position effect data includes: the real-time subtitle position data, the font style data, the real-time vocabulary position data and the real-time animation effect.

[0122] In some embodiments, the subtitle effect setting module 404 is further configured to determine disappearance time data of the subtitle unit according to the subtitle data.

[0123] determine disappearance animation effect according to the disappearance time data and the subtitle template data; and the disappearance time effect data includes: the disappearance time data and the disappearance animation effect.

[0124] In some embodiments, the apparatus further includes: a user editing module configured to: obtain a user editing instruction; edit the subtitle template data according to the user editing instruction, to obtain edited subtitle template data; perform subtitle effect updating according to the edited subtitle template data and the subtitle recognition result, to obtain a plurality of lines of updated subtitle effect sequences; and perform sequence fusion of the plurality of lines of updated subtitle effect sequences and the separated video, to obtain an updated video.

[0125] In the multi-action dynamic subtitle adding device, an initial video is obtained, and audio and video of the initial video are separated to obtain an initial audio and a separated video; subtitle recognition is performed on the initial audio to obtain a subtitle recognition result; subtitle template data is obtained; subtitle effects are set according to the subtitle template data and the subtitle recognition result to obtain a multi-line subtitle effect sequence; and the multi-line subtitle effect sequence is fused with the separated video to obtain a target video. Therefore, by implementing an automatic processing mode of a multi-action dynamic subtitle, multi-line subtitle effects are described in a subtitle template data, and a user only needs to select a subtitle template to generate multi-line real-time changing subtitle effect display on a screen.

[0126] The modules in the multi-action dynamic subtitle adding device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in the computer device in a software form, so as to be called and executed by the processor to perform operations corresponding to the modules.

[0127] In an exemplary embodiment, a computer device, which can be a terminal, has an internal structure as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be implemented through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement a multi-action dynamic subtitle adding method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device. The input device can also be an external keyboard, touchpad, or mouse, etc.

[0128] Those skilled in the art can understand that Figure 5The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the multi-action dynamic subtitle adding method when executing the computer program.

[0130] In one embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps of the multi-action dynamic subtitle adding method when executed by a processor.

[0131] In one embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps of the multi-action dynamic subtitle adding method when executed by a processor.

[0132] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0134] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0135] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for adding multi-line dynamic subtitles, characterized in that, The method includes: Obtain the initial video and perform audio-video separation on the initial video to obtain the initial audio and the separated video; The initial audio is subjected to subtitle recognition to obtain the subtitle recognition result; Get subtitle template data; Based on the subtitle template data and the subtitle recognition results, subtitle effects are set to obtain a multi-line subtitle effect sequence; The multi-line subtitle effect sequence is fused with the separated video to obtain the target video.

2. The method according to claim 1, characterized in that, The step of performing subtitle recognition on the initial audio to obtain subtitle recognition results includes: The initial audio is subjected to speech recognition to obtain the speech recognition result; The speech recognition results are analyzed to obtain subtitle data; The subtitle data is analyzed to obtain vocabulary data; the subtitle recognition result includes the subtitle data and the vocabulary data.

3. The method according to claim 1, characterized in that, The acquisition of subtitle template data includes: Obtain subtitle frame data, and determine the subtitle frame size and subtitle position based on the subtitle frame data; Obtain subtitle line data, and determine subtitle line spacing, subtitle row and column position, and subtitle style data based on the subtitle line data; the subtitle template data includes: the subtitle frame size, the subtitle position, the subtitle line spacing, the subtitle row and column position, and the subtitle style data.

4. The method according to claim 1, characterized in that, The subtitle recognition result includes: subtitle data and vocabulary data; the subtitle effect setting based on the subtitle template data and the subtitle recognition result, to obtain a multi-line subtitle effect sequence, includes: The starting time position is calculated based on the subtitle template data and the subtitle data to obtain the appearance time position data of the subtitle data on the timeline; Based on the subtitle template data, the subtitle data, and the vocabulary data, the real-time position effect calculation is performed on the subtitle unit of the subtitle data to obtain the position effect data of the subtitle unit on the time axis that changes with time. Based on the subtitle template data and the subtitle data, the disappearance effect of the subtitle unit is calculated to obtain the disappearance time effect data of the subtitle unit on the time axis; The subtitle effects are set based on the appearance time and location data, location effect data, and disappearance time effect data to obtain the multi-line subtitle effect sequence.

5. The method according to claim 4, characterized in that, The step of calculating the real-time position effect of the subtitle units in the subtitle data based on the subtitle template data, the subtitle data, and the vocabulary data to obtain the position effect data of the subtitle units changing over time on the time axis includes: Based on the subtitle template data and the subtitle time data of the subtitle data, the real-time position of the subtitle unit is calculated to obtain the real-time subtitle position data; Based on the subtitle template data and the vocabulary time data of the vocabulary data, the real-time position of the vocabulary in the subtitle unit is calculated to obtain real-time vocabulary position data; The real-time animation effect and font style data of the subtitle unit are determined based on the subtitle template data; the position effect data includes: the real-time subtitle position data, the font style data, the real-time word position data, and the real-time animation effect.

6. The method according to claim 4, characterized in that, The step of calculating the disappearance effect of the subtitle unit based on the subtitle template data and the subtitle data to obtain the disappearance time effect data of the subtitle unit on the time axis includes: The disappearance time data of the subtitle unit is determined based on the subtitle data; The disappearance animation effect is determined based on the disappearance time data and the subtitle template data; the disappearance time effect data includes: the disappearance time data and the disappearance animation effect.

7. The method according to claim 1, characterized in that, The method further includes: Obtain user editing instructions; Edit the subtitle template data according to the user's editing instructions to obtain edited subtitle template data; The subtitle effect is updated based on the edited subtitle template data and the subtitle recognition result to obtain a multi-line updated subtitle effect sequence; The multi-line updated subtitle effect sequence is merged with the separated video to obtain the updated video.

8. A multi-line dynamic subtitle adding device, characterized in that, The device includes: The first acquisition module is used to acquire an initial video and perform audio-video separation on the initial video to obtain initial audio and separated video; The subtitle recognition module is used to perform subtitle recognition on the initial audio and obtain the subtitle recognition result; The second acquisition module is used to acquire subtitle template data; The subtitle effect setting module is used to set subtitle effects based on the subtitle template data and the subtitle recognition results to obtain a multi-line subtitle effect sequence; The sequence fusion module is used to merge the multi-line subtitle effect sequence with the separated video to obtain the target video.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images