A subtitle generation method and device, electronic equipment and storage medium

Abstract

This disclosure provides a subtitle generation method, apparatus, electronic device, and storage medium. The method includes: acquiring information and tags corresponding to the subtitle to be generated; determining a decoding graph based on the information and tags; performing temporal classification on the information corresponding to the subtitle to be generated to obtain a corresponding directed cyclic graph; combining the decoding graph with the directed cyclic graph to obtain a target decoding graph; and generating the subtitle to be generated based on the target decoding graph.

Description

Technical Field

[0001] This disclosure relates to the field of image communication technology, and in particular to a method, apparatus, electronic device and storage medium for generating subtitles. Background Technology

[0002] With the development of the short video industry, the demand for short video creation is constantly emerging. Short video content often features a variety of complex elements, including background sound effects, original film and television soundtracks, and foreign languages.

[0003] Therefore, solving the alignment problem of text and audio / video content in complex scenarios and generating corresponding subtitles has always been a goal. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for generating subtitles, thereby at least solving the above-mentioned technical problems existing in the prior art.

[0005] According to a first aspect of this disclosure, a subtitle generation method is provided, the method comprising: acquiring information and tags corresponding to a subtitle to be generated; determining a decoding graph based on the information and tags; performing temporal classification on the information corresponding to the subtitle to be generated to obtain a corresponding directed cyclic graph; combining the decoding graph with the directed cyclic graph to obtain a target decoding graph; and generating the subtitle to be generated based on the target decoding graph.

[0006] In one possible implementation, obtaining the information and tags corresponding to the subtitle to be generated includes: obtaining original information related to the subtitle to be generated, the original information being used to generate the subtitle; performing normalization processing on the format of the original information to obtain the information corresponding to the subtitle to be generated; wherein, the information corresponding to the subtitle to be generated includes: text information and audio information; the tags are Hypertext Markup Language tags.

[0007] In one possible implementation, determining the decoded map based on the information and the tag includes: constructing an initial decoded map based on the information; adjusting the structure of the initial decoded map based on the tag, and determining the adjusted decoded map as the decoded map.

[0008] In one embodiment, adjusting the structure of the initial decoding graph based on the marker labels includes: adding the marker labels between two adjacent words in the initial decoding graph and setting corresponding penalty weights for the marker labels; and / or constructing a filling subgraph of the initial decoding graph; adding the marker labels at the beginning and end of each word in the text path of the initial decoding graph and setting corresponding penalty weights for the marker labels; and connecting the added marker labels to the filling subgraph.

[0009] In one possible implementation, the step of performing temporal classification on the information corresponding to the subtitle to be generated to obtain the corresponding directed cyclic graph includes: establishing an alignment model, using the alignment model to perform temporal classification on the information corresponding to the subtitle to be generated; and determining the directed cyclic graph corresponding to the alignment model.

[0010] In one possible implementation, generating the subtitle to be generated based on the target decoding map includes: determining text alignment information based on the target decoding map, wherein the text alignment information includes: decoded text information and corresponding timestamp information; and generating the subtitle to be generated based on the text alignment information.

[0011] In one embodiment, before generating the subtitle to be generated, the method further includes: performing secondary alignment of the alignment information and the original information in the manner of editing distance to obtain the start and end time information of each sentence in the original information, and using the start and end time information to improve the alignment accuracy of the subtitle to be generated.

[0012] According to a second aspect of this disclosure, a subtitle generation apparatus is provided, the apparatus comprising: a determining module, configured to acquire information and tags corresponding to a subtitle to be generated; determine a decoding graph based on the information and tags; perform temporal classification on the information corresponding to the subtitle to be generated to obtain a corresponding directed cyclic graph; a compositing module, configured to composite the decoding graph with the directed cyclic graph to obtain a target decoding graph; and a generating module, configured to generate the subtitle to be generated based on the target decoding graph.

[0013] In one possible implementation, the determining module is specifically used to acquire original information related to the subtitle to be generated, the original information being used to generate the subtitle; and to perform normalization processing on the format of the original information to obtain information corresponding to the subtitle to be generated; wherein, the information corresponding to the subtitle to be generated includes: text information and audio information; and the tag is a Hypertext Markup Language tag.

[0014] In one embodiment, the determining module includes: a construction submodule for constructing an initial decoding map based on the information; and an adjustment submodule for adjusting the structure of the initial decoding map based on the tag, and determining the adjusted decoding map as the decoding map.

[0015] In one possible implementation, the adjustment submodule is specifically used to add the marker label between two adjacent words in the initial decoding graph and set a corresponding penalty weight for the marker label; and / or, construct a filling subgraph of the initial decoding graph; add the marker label at the beginning and end of each word in the text path of the initial decoding graph and set a corresponding penalty weight for the marker label; and connect the added marker label with the filling subgraph.

[0016] In one possible implementation, the determining module is specifically used to establish an alignment model, use the alignment model to perform temporal classification on the information corresponding to the subtitle to be generated, and determine the directed cyclic graph corresponding to the alignment model.

[0017] In one possible implementation, the generation module is specifically used to determine text alignment information based on the target decoded image, the text alignment information including: decoded text information and corresponding timestamp information; and to generate the subtitle to be generated based on the text alignment information.

[0018] In one possible implementation, the generation module is specifically used to perform secondary alignment of the alignment information and the original information in the manner of editing distance, to obtain the start and end time information of each sentence in the original information, and to use the start and end time information to improve the alignment accuracy of the subtitle to be generated.

[0019] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0023] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0024] The present invention discloses a subtitle generation method, apparatus, electronic device, and storage medium that adjusts the structure of the target decoding graph by marking tags and a directed cyclic graph related to the information corresponding to the subtitle to be generated, making the target decoding graph smaller than a normal decoding graph, thereby reducing the overall computing resource requirements; at the same time, it can handle audio and video alignment problems in complex scenarios and generate subtitles to be generated.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0026] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0027] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0028] Figure 1 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 1 ;

[0029] Figure 2 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 2 ;

[0030] Figure 3 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 3 ;

[0031] Figure 4 This illustration shows a schematic diagram of the construction of a decoded map after adjusting the initial decoded map in a subtitle generation method according to an embodiment of the present disclosure;

[0032] Figure 5 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 4 ;

[0033] Figure 6 This illustration shows a schematic diagram of a model in a subtitle generation method according to an embodiment of the present disclosure;

[0034] Figure 7 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 5 ;

[0035] Figure 8 This diagram illustrates the overall processing flow of a subtitle generation method according to an embodiment of the present disclosure.

[0036] Figure 9 A schematic diagram of the composition structure of a subtitle generation device according to an embodiment of the present disclosure is shown;

[0037] Figure 10 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0038] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0039] Subtitle generation can generally be divided into two scenarios: generating subtitles solely from audio and video, and generating subtitles from both text and audio / video. Currently, both scenarios primarily rely on speech recognition, using timestamp information generated during the speech recognition process to generate subtitles. In the case of generating subtitles from both text and audio / video, a forced alignment scheme is also used to achieve subtitle generation, that is, synchronizing the text with the corresponding audio / video frame numbers to generate subtitles.

[0040] However, forced alignment schemes require a perfect match between the text and audio / video content. If this match is not perfect, it severely impacts the quality of the subtitles, and may even prevent their generation. With the development of the short video industry, the original audio / video information used to generate subtitles is becoming increasingly complex and diverse. When the original audio / video information contains background sound effects, original film and television soundtracks, and foreign languages, the given text cannot cover all the original audio / video information, making it difficult to generate subtitles using the alignment schemes employed in related technologies.

[0041] Figure 1 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 1 .

[0042] refer to Figure 1 The present disclosure discloses a subtitle generation method, which includes at least the following steps:

[0043] Step S101: Obtain the information and tags corresponding to the subtitles to be generated.

[0044] In some embodiments, the information corresponding to the subtitle to be generated may include: text information and audio information.

[0045] Step S102: Determine the decoded map based on the information and the tag.

[0046] Step S103: Perform temporal classification on the information corresponding to the subtitles to be generated to obtain the corresponding directed cyclic graph.

[0047] Step S104: Combine the decoded graph with the directed cyclic graph to obtain the target decoded graph.

[0048] In some embodiments, the target decoding map is a decoding map used for alignment (decoding).

[0049] Step S105: Generate subtitles to be generated based on the target decoding image.

[0050] In this embodiment of the disclosure, by marking tags and a directed cyclic graph related to the information corresponding to the subtitles to be generated, the structure of the target decoding graph is adjusted so that the target decoding graph is smaller than that of a normal decoding graph, thereby reducing the overall computing resource requirements; at the same time, it can handle the audio and video alignment problem in complex scenarios and generate the subtitles to be generated.

[0051] Figure 2 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 2 .

[0052] refer to Figure 2 In a processing flow of a subtitle generation method according to an embodiment of this disclosure, a specific implementation process for obtaining information and tagging labels corresponding to the subtitle to be generated may include at least the following steps:

[0053] Step S201: Obtain the original information related to the subtitle to be generated. The original information is used to generate the subtitle.

[0054] In some embodiments, the original information can be any information related to the subtitles to be generated. It can be audio / video information, such as a short video; or it can be audio / video information and corresponding text information, such as original film / television soundtracks and their corresponding text content.

[0055] Step S202: Normalize the format of the original information to obtain the information corresponding to the subtitle to be generated.

[0056] The information corresponding to the subtitles to be generated includes: text information and audio information; the tag tags are Hypertext Markup Language tag tags.

[0057] In some embodiments, the markup tags of Hyper Text Markup Language (HTML), namely HTML tags, are the most basic units of Hyper Text Markup Language.

[0058] In some embodiments, the audio format in the original information is uniformly converted to a 16kHz sampling rate waveform file (WaveForm, Wav) format to obtain the audio information corresponding to the subtitles to be generated. The audio format in the original information is diverse; Wav is one of the most common sound file formats, capable of recording various mono or stereo sound information while ensuring no sound distortion.

[0059] In some embodiments, the text content in the original information is first segmented using punctuation marks, and then the punctuation marks are removed from the segmented text content to obtain the text information corresponding to the subtitle to be generated.

[0060] In this embodiment of the disclosure, the original information related to the subtitle to be generated is preprocessed to obtain the information corresponding to the subtitle to be generated; at the same time, HTML tags are obtained in order to obtain the target decoding image, thereby generating the subtitle to be generated.

[0061] Figure 3 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 3 .

[0062] refer to Figure 3 In a subtitle generation method according to an embodiment of this disclosure, a specific implementation process for determining the decoding graph based on information and tag labels may include at least the following steps:

[0063] Step S301: Construct an initial decoding graph based on the information.

[0064] The information referred to here is the information corresponding to the subtitles to be generated in the aforementioned embodiments.

[0065] In some embodiments, a decoding graph of a weighted finite-state transducer (WFST), i.e., an initial decoding graph, is constructed based on the text information corresponding to the subtitles to be generated. The decoding graph of the weighted finite-state transducer is also referred to as the WFST decoding graph. In the field of automatic speech recognition (ASR), the weighted finite-state transducer is often referred to as a decoder.

[0066] Step S302: Adjust the structure of the initial decoding graph based on the tag, and determine the adjusted decoding graph as the decoding graph.

[0067] In some embodiments, the tag is an HTML tag.

[0068] In some embodiments, a specific implementation process for adjusting the structure of the initial decoded graph based on the tag may include at least the following steps:

[0069] Step 3021: Add a marker label between two adjacent words in the initial decoding graph and set the corresponding penalty weight for the marker label.

[0070] In some embodiments, the penalty weight is typically a hyperparameter that can be set according to actual needs. For example, it can be set based on the recognition performance on the test set.

[0071] In some embodiments, when there are multiple characters in the text information corresponding to the subtitle to be generated, deletion tags can be added between two adjacent characters in the initial decoding diagram, and a penalty weight can be set for the deletion tags so that the deletion tags can be selected first during decoding and the erroneous deletion operations can be reduced.

[0072] As an example, let the initial decoding notation be G. When there are multiple characters in the text information corresponding to the subtitle to be generated, such as the original audio being "The weather is nice today," while the text information corresponding to the subtitle to be generated is "The weather in Shanghai is nice today," then additional characters can be added between two adjacent characters in G. <del>Tags (Add one) <del>(Edge), and set a corresponding penalty weight. When there are multiple characters, you can prioritize selecting <del>The tag is deleted, and for <del>Setting penalty weights for tags can prevent excessive deletion errors. Specifically, <del>The tag indicates text content that has been removed from the document.

[0073] Step 3022, and / or, construct a filled subgraph of the initial decoding graph; add marker labels to the beginning and end of each word in the text path of the initial decoding graph, and set corresponding penalty weights for the marker labels; connect the added marker labels to the filled subgraph.

[0074] In some embodiments, when there are missing contents in the text information corresponding to the subtitle to be generated, or when there are no missing contents in the text but there are replacement errors caused by factors such as inconsistent pronunciation, a filler subgraph can be introduced to solve the problem. For example, the audio in the original information is "The weather is nice today", while the text information corresponding to the subtitle to be generated is "The weather is nice".

[0075] In some embodiments, the filled subgraph is mainly used to supplement missing text or replace incorrect text, and includes HTML tags (jump edges) for all words in the dictionary.

[0076] In some embodiments, add at the beginning and end of each word of the text path in the initial decoding graph. <ins>Tags, and for added <ins>A corresponding penalty weight is assigned to each label and connected to the filled subgraph to address issues of missing and incorrect text content. <ins>Tags indicate content added to a document.

[0077] Figure 4 The diagram shows the modified decoded graph after adjustments to the initial decoded graph. Here, "Magic Sound Workshop" represents the text content. <ins>and <del>This is an HTML tag used to display the decoding results.

[0078] In this embodiment of the disclosure, by constructing an initial decoding graph and adjusting its structure, not only is the inconsistency between text and audio resolved, but the adjusted decoding graph is also made smaller, reducing the overall computational resource requirements.

[0079] Figure 5 This illustration shows a processing flow diagram of a subtitle generation method according to an embodiment of the present disclosure. Figure 4 .

[0080] refer to Figure 5 In one embodiment of the subtitle generation method disclosed herein, a specific implementation process for performing temporal classification on the information corresponding to the subtitle to be generated to obtain the corresponding directed cyclic graph may include at least the following steps:

[0081] Step S401: Establish an alignment model and use the alignment model to perform temporal classification of the information corresponding to the subtitles to be generated.

[0082] In some embodiments, the alignment model can employ a CTC (Connectionist Temporal Classification) model that uses convolutional enhancement for speech recognition. The CTC model is typically used to solve problems such as speech recognition, text recognition, and temporal classification based on neural networks.

[0083] Step S402: Determine the directed cyclic graph corresponding to the alignment model.

[0084] In some embodiments, the WFST corresponding to the CTC model is obtained. Here, WFST is a directed cyclic graph that satisfies the semi-cyclic theory.

[0085] Figure 6 The WFST corresponding to the CTC model is shown. Among them, <blank>The label is used to cut continuous characters; Eps is epsilon in WFST, representing empty; vK represents each character (modeling unit) in the dictionary, such as: "魔", "音"; Since the CTC Chinese model uses single characters as modeling units, during the process of determining the decoding graph, it needs to be segmented according to the symbol table.

[0086] In some embodiments, the adjusted decoding graph is operated in accordance with the WFST corresponding to the CTC model to obtain a target decoding graph for alignment (decoding) use.

[0087] In the embodiments of the present disclosure, a target decoding graph is determined based on the adjusted initial decoding graph, so that the target decoding graph is used to generate the subtitle to be generated, thereby solving the alignment problem of speech and Chinese text in complex scenarios, and having a strong suppression ability for other languages and background sounds, etc.

[0088] Figure 7 Shows the processing flow schematic of a subtitle generation method according to an embodiment of the present disclosure Figure 5 .

[0089] Reference Figure 7 , in a processing flow of a subtitle generation method according to an embodiment of the present disclosure, a specific implementation process of generating the subtitle to be generated based on the target decoding graph may at least include the following steps:

[0090] Step S501, determining text alignment information based on the target decoding graph, where the text alignment information includes: decoded text information and corresponding timestamp information.

[0091] In some embodiments, the timestamp information corresponding to the text information may include: character string and time information of each character.

[0092] Step S502, generating the subtitle to be generated based on the text alignment information.

[0093] In some embodiments, before generating the subtitle to be generated, the text alignment information and the text information in the original information may also be secondarily aligned in the manner of edit distance to obtain the start and end time information of each sentence in the original information, and the format is converted into a subtitle file format such as SRT (SubRip Text, small segment of text), and the start and end time information is used to improve the alignment accuracy of the subtitle to be generated. Among them, SRT is a subtitle file format.

[0094] In some embodiments, the text alignment information is the recognition result obtained through the foregoing embodiments; the original information may be any information related to the subtitle to be generated, and the text information in the original information may be obtained through user input.

[0095] In this embodiment of the disclosure, the subtitles to be generated are generated by using the target decoding map and then undergo secondary alignment, thereby improving the quality of the generated subtitles.

[0096] Figure 8 A schematic diagram of the overall processing flow of a subtitle generation method according to an embodiment of the present disclosure is shown.

[0097] like Figure 8 As shown, the audio and corresponding text input to the subtitle timing system are preprocessed, with the text potentially in sentence-by-sentence format. The subtitle timing system constructs an initial decoding map based on the preprocessed text and adjusts its structure to ultimately determine the target decoding map. The system then decodes the target decoding map to obtain basic text alignment information and performs post-processing to further improve alignment accuracy. After processing, the output is a subtitle file, such as an SRT file, where the start and end times of each sentence are accurate to the millisecond level. An example is shown below: 1

[0099] 00:00:00,000-->00:00:02,880

[0100] Looking at Country A's steel history to understand Country A's manufacturing industry 2

[0102] 00:00:02,880-->00:00:04,160

[0103] Rise and Capital 3

[0105] 00:00:04,160-->00:00:05,600

[0106] Decline due to greed 4

[0108] 00:00:05,600-->00:00:08,240

[0109] Country A rose to prominence during the Second Industrial Revolution.

[0110] The specific implementation process in this embodiment is the same as that in the foregoing embodiments, and will not be repeated here.

[0111] Figure 9 A schematic diagram of the composition of a subtitle generation device and an electronic device according to an embodiment is shown.

[0112] refer to Figure 9 An embodiment of a subtitle generation device 90 includes: a determining module 901, used to acquire information and tags corresponding to the subtitle to be generated; determine a decoding graph based on the information and tags; perform temporal classification on the information corresponding to the subtitle to be generated to obtain a corresponding directed cyclic graph; a compositing module 902, used to composite the decoding graph and the directed cyclic graph to obtain a target decoding graph; and a generating module 903, used to generate the subtitle to be generated based on the target decoding graph.

[0113] In some embodiments, the determining module 901 is specifically used to obtain the original information related to the subtitle to be generated, the original information being used to generate the subtitle; to perform normalization processing on the format of the original information to obtain the information corresponding to the subtitle to be generated; wherein, the information corresponding to the subtitle to be generated includes: text information and audio information; the tag is a tag of Hypertext Markup Language.

[0114] In some embodiments, the determining module 901 includes: a construction submodule 9011, used to construct an initial decoding map based on information; and an adjustment submodule 9012, used to adjust the structure of the initial decoding map based on the tag, and determine the adjusted decoding map as the decoding map.

[0115] In some embodiments, the adjustment submodule 9012 is specifically used to add marker labels between two adjacent words in the initial decoding graph and set corresponding penalty weights for the marker labels; and / or, construct a filling subgraph of the initial decoding graph; add marker labels at the beginning and end of each word in the text path of the initial decoding graph and set corresponding penalty weights for the marker labels; and connect the added marker labels with the filling subgraph.

[0116] In some embodiments, the determining module 901 is specifically used to establish an alignment model, use the alignment model to perform temporal classification of the information corresponding to the subtitle to be generated, and determine the directed cyclic graph corresponding to the alignment model.

[0117] In some embodiments, the generation module 903 is specifically used to determine text alignment information based on the target decoded image, the text alignment information including: decoded text information and corresponding timestamp information; and to generate subtitles to be generated based on the text alignment information.

[0118] In some embodiments, the generation module 903 is specifically used to perform secondary alignment of the alignment information and the original information in the manner of editing distance, to obtain the start and end time information of each sentence in the original information, and to use the start and end time information to improve the alignment accuracy of the subtitle to be generated.

[0119] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0120] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable electronic devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0121] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0122] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other electronic devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as a subtitle generation method. For example, in some embodiments, a subtitle generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of a subtitle generation method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a subtitle generation method by any other suitable means (e.g., by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: program code implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device; and program code for implementing the methods of this disclosure that can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing apparatus such that, when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution apparatus, electronic device, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices, electronic devices, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the devices, electronic equipment, and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The devices, electronic devices, and technologies described herein can be implemented in computing devices or electronic devices that include backend components (e.g., as a data server), or middleware components (e.g., an application server), or frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the devices, electronic devices, and technologies described herein), or any combination of such backend, middleware, or frontend components. Components of the devices or electronic devices can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0128] Computer devices and electronic devices can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers for distributed devices or electronic devices, or servers incorporating blockchain technology.

[0129] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0131] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.< / blank> < / del> < / ins> < / ins> < / ins> < / ins> < / del> < / del> < / del> < / del> < / del>

Claims

1. A subtitle generation method characterized by, The method includes: Obtain the information and tags corresponding to the subtitle to be generated; wherein, the information corresponding to the subtitle to be generated includes: text information and audio information; the tags are Hypertext Markup Language tags; An initial decoding graph is constructed based on the text information; The structure of the initial decoding graph is adjusted based on the aforementioned tags, and the adjusted decoding graph is determined as the decoding graph. The information corresponding to the subtitles to be generated is classified in time sequence to obtain the corresponding directed cyclic graph; The decoded graph is combined with the directed cyclic graph to obtain the target decoded graph; Based on the target decoding map, the subtitle to be generated is generated; The step of adjusting the structure of the initial decoding graph based on the marker labels includes: adding the marker labels between two adjacent words in the initial decoding graph and setting corresponding penalty weights for the marker labels; and / or constructing a filling subgraph of the initial decoding graph; adding the marker labels at the beginning and end of each word in the text path of the initial decoding graph and setting corresponding penalty weights for the marker labels; and connecting the added marker labels with the filling subgraph.

2. The method of claim 1, wherein, The process of obtaining the information and tags corresponding to the subtitles to be generated includes: Obtain the original information related to the subtitle to be generated, the original information being used to generate the subtitle; The format of the original information is normalized to obtain the information corresponding to the subtitle to be generated.

3. The method of claim 1, wherein, The step of performing time-series classification on the information corresponding to the subtitles to be generated to obtain the corresponding directed cyclic graph includes: An alignment model is established, and the alignment model is used to perform temporal classification on the information corresponding to the subtitles to be generated; Determine the directed cyclic graph corresponding to the alignment model.

4. The method of claim 1, wherein, The step of generating the subtitles to be generated based on the target decoding map includes: The text alignment information is determined based on the target decoding map, and the text alignment information includes: decoded text information and corresponding timestamp information; The subtitle to be generated is generated based on the text alignment information.

5. The method of claim 4, wherein, Before generating the subtitle to be generated, the method further includes: The alignment information and the original information are re-aligned using the edit distance method to obtain the start and end time information of each sentence in the original information. The start and end time information is then used to improve the alignment accuracy of the subtitle to be generated.

6. A subtitle generation apparatus characterized by comprising: The device includes: A determination module is used to acquire information and tags corresponding to the subtitle to be generated; determine a decoding graph based on the information and tags; perform temporal classification on the information corresponding to the subtitle to be generated to obtain the corresponding directed cyclic graph; wherein, the information corresponding to the subtitle to be generated includes: text information and audio information; the tags are Hypertext Markup Language tags; A composite module is used to combine the decoded graph with the directed cyclic graph to obtain a target decoded graph; A generation module is used to generate the subtitles to be generated based on the target decoded image; The determining module includes: a construction submodule, used to construct an initial decoding map based on the text information; and an adjustment submodule, used to adjust the structure of the initial decoding map based on the tag, and determine the adjusted decoding map as the decoding map. The adjustment submodule is specifically used to add the marker label between two adjacent words in the initial decoding graph and set a corresponding penalty weight for the marker label; and / or, construct a filling subgraph of the initial decoding graph; add the marker label at the beginning and end of each word in the text path of the initial decoding graph and set a corresponding penalty weight for the marker label; and connect the added marker label with the filling subgraph.

7. An electronic device, comprising: include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the subtitle generation method according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the subtitle generation method according to any one of claims 1-5.

Images