Subtitle generation methods, devices, equipment and readable storage media

Abstract

This application discloses a subtitle generation method, apparatus, device, and readable storage medium. The method combines the audio and visual content of a target video to generate basic subtitle text corresponding to the target video. Based on this, the application can utilize video sound effects and visuals to avoid missing key details of the target video. To improve the fit between the final generated subtitle text and the target video, the application can generate a knowledge graph containing a summary of the target video's content. The basic subtitle text is then adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video. This allows for targeted adjustments to the basic subtitle text based on the main content and entity information of the target video during the optimization process, ensuring a progressive narrative in the subtitles. Therefore, this application can comprehensively integrate video audio, video visuals, and video summaries to improve the accuracy of the final optimized subtitles and enhance the viewing experience for video viewers.

Description

Technical Field

[0001] This application relates to the field of subtitle technology, and more specifically, to a subtitle generation method, apparatus, device, and readable storage medium. Background Technology

[0002] In modern society, watching videos has become a common form of entertainment. During viewing, viewers can further understand the video content through subtitles. However, existing subtitle generation technologies primarily rely on general speech-to-text models to convert video audio into subtitles. This lack of consideration for contextual factors such as characters and plot within the video results in subtitles that fail to accurately match the specific situation created by the video, often leading to inaccurate emotional delivery, reduced accuracy, and negatively impacting the viewer's experience. Summary of the Invention

[0003] In view of this, this application provides a subtitle generation method, apparatus, device, and readable storage medium to address the shortcomings of low accuracy in subtitle text generated in the prior art.

[0004] To achieve the above objectives, the following solution is proposed:

[0005] A method for generating subtitles, comprising:

[0006] By combining the audio and visual content corresponding to the target video, basic subtitle text corresponding to the target video is generated;

[0007] Generate a knowledge graph containing a summary of the target video content;

[0008] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0009] Optionally, the step of combining the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video includes:

[0010] Using speech recognition technology, the audio content of the target video is converted into first subtitle text in the target language;

[0011] Optical character recognition technology is used to identify and extract the visual content of the target video, and generate second subtitle text in the target language;

[0012] The first subtitle text and the second subtitle text are mutually calibrated to obtain the basic subtitle text expressed in the target language.

[0013] Optionally, generating a knowledge graph containing a summary of the target video content includes:

[0014] Obtain the associated videos of the target video, and obtain the video summary of the associated videos and the content summary of the target video;

[0015] By combining the video summary and the content outline, a global knowledge graph representing the target language is constructed.

[0016] Based on the content summary and referring to the global knowledge graph, a knowledge graph that applies the target language representation and corresponds to the target video is constructed.

[0017] Optionally, obtaining the associated videos of the target video includes:

[0018] Other videos that have a logical relationship with the target video are considered as associated videos of the target video.

[0019] Optionally, obtaining a summary of the target video's content includes:

[0020] Extract a set of key information from the audio and visual content of the target video, and organize the set of key information to form a content outline of the corresponding target video;

[0021] The key information set includes any combination of the following: character identifiers, character relationship information, personality trait information, character emotional information, character interaction scenarios, and event development information.

[0022] Optionally, the step of constructing a knowledge graph based on the content outline and referring to the global knowledge graph, applying the target language representation and corresponding to the target video, includes:

[0023] Extract all entities and entity relationships from the aforementioned content outline to construct a preliminary graph;

[0024] Based on the content summary and the video summary, the target language terminology database corresponding to the target video is determined;

[0025] By combining the target language terminology database, the content summary, and the global knowledge graph, the preliminary knowledge graph is optimized to form the knowledge graph of the target video.

[0026] Optionally, adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video includes:

[0027] Obtain a trained multimodal model;

[0028] Using the multimodal model and referring to the knowledge graph, the basic subtitle text is optimized and adjusted to obtain the optimized subtitles output by the multimodal model.

[0029] Optionally, obtaining the trained multimodal model includes:

[0030] Obtain an initial multimodal model and multiple training samples. Each training sample contains a training knowledge graph, basic training subtitles, and optimized training subtitles for the corresponding training video.

[0031] The initial multimodal model is trained sequentially using each training sample to obtain the predicted text corresponding to each training sample output by the initial multimodal model;

[0032] Calculate the similarity between the predicted text and the trained optimized captions corresponding to the same training sample;

[0033] Based on each similarity, the parameters of the initial multimodal model are updated until the latest initial multimodal model meets the preset conditions. The final initial multimodal model is then used as the trained multimodal model.

[0034] A subtitle generation device, comprising:

[0035] The module is used to combine the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video;

[0036] The generation module is used to generate a knowledge graph containing a summary of the target video content;

[0037] The adjustment module is used to adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0038] A subtitle generation device, including a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to execute the program and implement each step of the above-described subtitle generation method.

[0041] A readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the various steps of the above-described subtitle generation method.

[0042] As can be seen from the above technical solutions, the subtitle generation method provided in this application can combine the audio and visual content corresponding to the target video to generate basic subtitle text corresponding to the target video. Based on this, this application can utilize audio and visuals to generate basic subtitle text. Thus, by referring to the sound effects of the target video and its matching visual images, the method can identify key information such as the emotions and actions of the sound effect generation object, avoid missing key details of the target video, and improve the accuracy of the basic subtitle text. In order to improve the fit between the final generated subtitle text and the target video, this application can generate a knowledge graph containing a summary of the target video content. Based on the knowledge graph, the basic subtitle text is adjusted to form optimized subtitles corresponding to the target video. Based on this, since the content summary can contain the main content of the target video and entity information such as character personality and character emotions, the knowledge graph can integrate the main content of the target video and various entity information. Therefore, by referencing knowledge graphs to optimize the basic subtitle text, the optimized subtitles can be tailored to the main content and entity information of the target video during the optimization process. This better reflects the plot atmosphere of the target video, ensures the progressive development of the subtitle storyline, and further improves the accuracy of the optimized subtitles. It is evident that this application can integrate video audio, video footage, and video synopsis to improve the accuracy of the final optimized subtitles and enhance the viewing experience for video viewers. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 This is a flowchart of a subtitle generation method disclosed in an embodiment of this application;

[0045] Figure 2 This is a structural block diagram of a subtitle generation device disclosed in an embodiment of this application;

[0046] Figure 3 This is a hardware structure block diagram of a subtitle generation device disclosed in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] This application provides a subtitle generation method, which can be applied to various video playback platforms or smart TV operating systems, as well as to various computer terminals or smart terminals. The executing entity can be the processor or server of the computer terminal or smart terminal. The method flowchart is shown below. Figure 1 As shown, it specifically includes:

[0049] Next, combine Figure 1 The subtitle generation method described in this application is detailed, including the following steps:

[0050] Step S1: Combine the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video.

[0051] Specifically, it can obtain the audio and visual content corresponding to the target video.

[0052] Speech recognition technology can be used to convert audio content into first subtitle text, and optical vision technology can be used to identify the second subtitle text contained in the target video;

[0053] The first and second subtitle texts are cross-corrected to obtain the base subtitle text.

[0054] The target video can be any type of playable video, such as any segment of a TV series, any episode of a variety show, or any segment of an animated film.

[0055] Step S2: Generate a knowledge graph containing a summary of the target video content.

[0056] Specifically, a summary of the target video's content can be generated by referring to the target video's audio and visual content, as well as data information reflecting the target video's playback content, such as comment data and introductory information.

[0057] The content outlines can be organized into a knowledge graph.

[0058] Knowledge graphs can be used to represent multiple target entities contained in a target video, the entity attributes of each target entity, and the entity relationships between the target entities.

[0059] Among them, the same type of target entity can correspond to multiple different types of entity attributes.

[0060] For example, when the target entity is of the "person" type, its entity attributes can correspond to entity attributes such as "person's emotions", "person's personality", "person's relationships", and "person's behavior".

[0061] When the target entity is of type event, its entity attributes can include entity attributes such as time, location, and key plot developments.

[0062] Different types of target videos can contain the same type of target entity.

[0063] For example, target videos of the type TV series and target videos of the type variety show can both correspond to target entities of the character category.

[0064] The same target video can contain multiple different types of target entities.

[0065] For example, the target entity types in some target videos can include people and events.

[0066] Step S3: Adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0067] Specifically, by referring to knowledge graphs, the basic subtitle text can be fine-tuned from multiple perspectives to obtain optimized subtitles for the target video.

[0068] For example, you can refer to knowledge graphs to add speaker identifiers to each line of dialogue in the basic subtitle text;

[0069] You can also refer to the emotions of each character in the knowledge graph and add words that reflect the emotions of the corresponding characters to the corresponding subtitle segments in the basic subtitle text;

[0070] You can also refer to the personalities of each character in the knowledge graph and add words that reflect the personality of the corresponding character to the corresponding subtitle fragments in the basic subtitle text.

[0071] As can be seen from the above technical solutions, the subtitle generation method provided in this application can combine the audio and visual content corresponding to the target video to generate basic subtitle text corresponding to the target video. Based on this, this application can utilize audio and visuals to generate basic subtitle text. Thus, by referring to the sound effects of the target video and its matching visuals, the method can identify key information such as the emotions and actions of the sound effect generation object, avoid missing key details of the target video, and improve the accuracy of the basic subtitle text. In order to improve the fit between the final generated subtitle text and the target video, this application can generate a knowledge graph containing a summary of the target video content. Based on the knowledge graph, the basic subtitle text is adjusted to form optimized subtitles corresponding to the target video. Based on this, since the content summary can contain the main content of the target video and entity information such as character personality and character emotions, the knowledge graph can integrate the main content of the target video and various entity information. Therefore, by referencing knowledge graphs to optimize the basic subtitle text, the optimized subtitles can be tailored to the main content and entity information of the target video during the optimization process. This better reflects the plot atmosphere of the target video, ensures the progressive development of the subtitle storyline, and further improves the accuracy of the optimized subtitles. It is evident that this application can integrate video audio, video footage, and video synopsis to improve the accuracy of the final optimized subtitles and enhance the viewing experience for video viewers.

[0072] Furthermore, to ensure the reliability of the optimized subtitles, the final optimized subtitles can be compared with the original subtitles of the target video to evaluate the quality of the optimized subtitles.

[0073] Furthermore, evaluation metrics such as BLEU and ROUGE, as well as embedding models such as METEOR, BERT, and GPT, can be used to calculate the semantic similarity between the optimized subtitles and the original subtitles, thereby evaluating the quality of the optimized subtitles.

[0074] It can also receive user feedback and further optimize the subtitle generation method of this application based on the user feedback.

[0075] Optimized subtitles can be embedded into the target video using an embedding method.

[0076] In some embodiments of this application, the process of generating basic subtitle text corresponding to the target video by combining the audio and visual content corresponding to the target video is described in detail, and the steps are as follows:

[0077] S10. Using speech recognition technology, the audio content of the target video is converted into first subtitle text in the target language.

[0078] Specifically, ASR technology can be used to transcribe audio content into text;

[0079] Determine the target language;

[0080] The target language can be set according to actual needs; it can be any dialect or any official language.

[0081] If the target language and the target video language are not the same, translation technology is used to translate the text into the first subtitle text;

[0082] If the target language and the target video language are the same, then the text will be used as the first subtitle text.

[0083] S11. Using optical character recognition technology, identify and extract the visual content of the target video, and generate second subtitle text in the target language.

[0084] Specifically, OCR technology can be used to extract text content appearing in the target video, such as subtitles and signs;

[0085] If the target language and the target video language are not the same, translation technology is used to translate the text content into a second subtitle text;

[0086] If the target language and the target video language are the same, the text content will be used as the second subtitle text.

[0087] The first subtitle text refers to the subtitle text generated based on the target video audio content; the second subtitle text refers to the subtitle text generated based on the target video visual content.

[0088] The information contained in the first subtitle text and the second subtitle text may be exactly the same, or there may be some differences.

[0089] S12. The first subtitle text and the second subtitle text are mutually calibrated to obtain the basic subtitle text expressed in the target language.

[0090] Specifically, the first subtitle text and the second subtitle text can be referenced to each other, and the first subtitle text and the second subtitle text can be mutually corrected to obtain the basic subtitle text.

[0091] As can be seen from the above technical solution, this embodiment provides an optional method for generating basic subtitle text corresponding to a target video. Through the above method, this application can combine speech recognition technology and optical character recognition technology to generate basic subtitle text corresponding to the target language, further better integrating audio and video to construct basic subtitle text. At the same time, the generated basic subtitle text can also be expressed in the target language. Based on this, the subtitles generated by this application may not necessarily correspond to the same language as the target video. This application can be used for video translation to generate subtitle text in a specific language, further improving the applicability of this application.

[0092] In some embodiments of this application, the process of step S2, generating a knowledge graph containing a summary of the target video content, is described in detail below:

[0093] S20. Obtain the associated videos of the target video, and obtain the video summary of the associated videos and the content summary of the target video.

[0094] Specifically, all videos related to the target video can be obtained as associated videos;

[0095] Key information can be extracted from the audio, visuals, reviews, and video descriptions of each associated video, and then organized and combined to obtain a video summary of the associated video.

[0096] S21. Combining the video summary and the content outline, construct a global knowledge graph using the target language representation.

[0097] Specifically, it can extract all entities from the content outline and each video summary, and sort out all relationships in the content outline and each video summary;

[0098] A global knowledge graph is constructed based on each entity and its relationships.

[0099] S22. Based on the content summary and referring to the global knowledge graph, construct a knowledge graph that applies the target language representation and corresponds to the target video.

[0100] Specifically, based on the content outline and with reference information from the global knowledge graph, a knowledge graph that is expressed in the target language and corresponds to the target video can be constructed.

[0101] As can be seen from the above technical solution, this embodiment provides an optional method for generating a knowledge graph containing a summary of the target video content. Through the above method, a knowledge graph can be constructed by combining the summary of the target video content and the video summaries of related videos. This further improves the reliability and continuity of the knowledge graph.

[0102] In some embodiments of this application, the process of obtaining the associated video of the target video in step S20 is described in detail, and the steps are as follows:

[0103] S200: Other videos that have a logical relationship with the target video are designated as associated videos of the target video.

[0104] Specifically, logical relationships can be any one or more of the following: content relationships, narrative logic relationships, character development relationships, and theme presentation relationships.

[0105] Content relationships can include plot coherence, story continuity, and scene connection.

[0106] Narrative logic relationships can include the relationship between foreshadowing and revelation of clues, as well as the relationship between the main plot and the overall narrative.

[0107] Character development relationships can include character growth relationships, character development relationships, and character succession relationships.

[0108] As can be seen from the above technical solution, this embodiment provides an optional method for obtaining associated videos. Through the above method, the relationship between the target video and other videos can be analyzed, thereby determining the associated videos of the target video.

[0109] In some embodiments of this application, the process of obtaining the content outline of the target video in step S20 is described in detail, and the steps are as follows:

[0110] S201. Extract a set of key information from the audio and visual content of the target video, and organize the set of key information to form a content outline of the corresponding target video.

[0111] Specifically, the key information types in the content summary can be determined based on the type of the target video;

[0112] Extract key information matching the key information type from the audio content, visual content, comment data, and introductory information of the target video to form a key information set.

[0113] For example, when the target video is a sitcom, key information can be extracted from the audio content, visual content, comment data, and introductory information to form a set of key information, including character identifiers, character relationship information, personality traits, character emotions, character interaction scenes, and / or event development information.

[0114] When the target video is a singing variety show, key information can be extracted from the audio content, visual content, comment data, and introductory information to form a set of key information, including singer identification, song selection, singer emotions, singer relationships, and / or singing events.

[0115] As can be seen from the above technical solution, this embodiment provides an optional method for obtaining the content outline of the target video. Through the above method, key information can be extracted in a targeted manner, a content outline can be constructed, and the reliability of the knowledge graph of this application can be further improved.

[0116] In some embodiments of this application, the process of step S22, which involves constructing a knowledge graph that applies the target language representation and corresponds to the target video based on the content outline and referring to the global knowledge graph, is described in detail below:

[0117] S220. Extract all entities and entity relationships from the content summary to construct a preliminary graph.

[0118] Specifically, it can perform word segmentation, part-of-speech tagging, and named entity recognition on the content outline to extract entities and entity relationships;

[0119] Using each entity as a node and referring to the relationships between the entities, edges are constructed to form a preliminary graph.

[0120] S221. Based on the content summary and the video summary, determine the target language terminology database corresponding to the target video.

[0121] Specifically, one can analyze the professional fields involved in the content outline and video summary to determine the target language terminology database for expression in the target language.

[0122] S222. Combining the target language terminology database, the content summary, and the global knowledge graph, the preliminary graph is optimized to form the knowledge graph of the target video.

[0123] Specifically, using a target language terminology database, non-standard terms in the preliminary atlas are optimized;

[0124] Based on the content outline and the overall knowledge graph, the optimized preliminary knowledge graph is further refined to obtain the knowledge graph of the target video.

[0125] Knowledge graphs can be used to represent multiple target entities contained in a target video, the entity attributes of each target entity in the target video, and the entity relationships between the target entities. They can also include the entity attributes of each target entity in each associated video and the entity relationships between the target entities.

[0126] As can be seen from the above technical solution, this embodiment provides an optional method for constructing a target language representation knowledge graph based on content outline and global knowledge graph. Through the above method, the knowledge graph can be further optimized and adjusted by using the target language terminology database, which further ensures the professionalism of the optimized subtitles generated in this application and further improves the viewing experience of the audience.

[0127] In some embodiments of this application, the process of adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video is described in detail, and the steps are as follows:

[0128] S30. Obtain the trained multimodal model.

[0129] Specifically, a trained VLM model can be obtained.

[0130] The VLM model can include a matching layer and a basic caption text processing layer.

[0131] S31. Using the multimodal model and referring to the knowledge graph, optimize and adjust the basic subtitle text to obtain the optimized subtitles output by the multimodal model.

[0132] Specifically, the knowledge graph and basic caption text can be input into the multimodal model, and the matching layer can be used to match each statement in the basic caption text with each node in the knowledge graph.

[0133] By utilizing the basic subtitle text processing layer and based on matching nodes, detailed adjustments are made to each sentence to obtain optimized subtitles.

[0134] As can be seen from the above technical solution, this embodiment provides an optional method for adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video. The above method can utilize a trained multimodal model to complete the adjustment of the basic subtitle text, further ensuring the accuracy of the optimized subtitles.

[0135] In some embodiments of this application, the process of obtaining the trained multimodal model in step S30 is described in detail, and the steps are as follows:

[0136] S300. Obtain the initial multimodal model and multiple training samples. Each training sample contains the training knowledge graph, basic training subtitles, and optimized training subtitles for the corresponding training video.

[0137] Specifically, an initial multimodal model can be constructed, and multiple training samples containing the corresponding training videos, basic training subtitles, and optimized training subtitles can be obtained.

[0138] Among them, the training optimized subtitles and the training basic subtitles can be expressed in different types of languages.

[0139] Training and optimizing subtitles can provide subtitles that have been adjusted by professionals.

[0140] S301. The initial multimodal model is trained sequentially using each training sample to obtain the predicted text corresponding to each training sample output by the initial multimodal model.

[0141] Specifically, the initial multimodal model can be trained based on each training sample to obtain the predicted text corresponding to each training sample.

[0142] S302. Calculate the similarity between the predicted text and the training optimized subtitles corresponding to the same training sample.

[0143] Specifically, word segmentation, part-of-speech tagging, and named entity recognition can be performed on the predicted text and the training optimized captions, and the similarity between each predicted text and its corresponding training optimized captions can be calculated.

[0144] The similarity between each predicted text and its corresponding trained and optimized subtitles can be calculated in several ways.

[0145] For example, the similarity between each predicted text and its corresponding trained and optimized captions can be calculated using the Euclidean distance calculation expression.

[0146] Alternatively, the similarity between each predicted text and its corresponding trained and optimized captions can be calculated using the cosine distance expression.

[0147] S303. Based on each similarity, update the parameters of the initial multimodal model until the latest initial multimodal model meets the preset conditions, and use the final initial multimodal model as the trained multimodal model.

[0148] Specifically, the initial multimodal model is iteratively updated based on each similarity until the similarity between the predicted text output by the latest initial multimodal model and the corresponding trained and optimized subtitles is less than a preset similarity threshold. The final initial multimodal model is then used as the trained multimodal model.

[0149] As can be seen from the above technical solution, this embodiment provides an optional method for training a multimodal model. By referring to the above method, the multimodal model can be trained with reference to similarity, ensuring that the multimodal model can make detailed adjustments while ensuring the accuracy of the subtitle text, thereby improving the reliability of the optimized subtitles.

[0150] Next, we will combine Figure 2 The subtitle generation apparatus provided in this application will be described in detail. The subtitle generation apparatus described below can be compared with the subtitle generation method described above.

[0151] See Figure 2 It can be observed that the subtitle generation device may include:

[0152] Module 10 is used to combine the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video.

[0153] Generation module 20 is used to generate a knowledge graph containing a summary of the target video content;

[0154] The adjustment module 30 is used to adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0155] Furthermore, module 10 may include:

[0156] The first combining unit is used to convert the audio content of the target video into first subtitle text in the target language using speech recognition technology;

[0157] The second combining unit is used to identify and extract the visual content of the target video using optical character recognition technology, and generate second subtitle text in the target language;

[0158] The third combining unit is used to perform mutual calibration on the first subtitle text and the second subtitle text to obtain a basic subtitle text expressed in the target language.

[0159] Furthermore, the generation module 20 may include:

[0160] The associated video acquisition unit is used to acquire associated videos of the target video, and to acquire video summaries of the associated videos and content summaries of the target video;

[0161] A global knowledge graph construction unit is used to combine the video summary and the content outline to construct a global knowledge graph represented by the target language.

[0162] The knowledge graph construction unit is used to construct a knowledge graph that applies the target language representation and corresponds to the target video based on the content outline and with reference to the global knowledge graph.

[0163] Furthermore, the associated video acquisition unit may include:

[0164] A logical association recognition component is used to identify other videos that have a logical association with the target video as associated videos of the target video.

[0165] Furthermore, the associated video acquisition unit may also include:

[0166] A key information set extraction component is used to extract a key information set from the audio and visual content of the target video, and to organize the key information set to form a content outline of the corresponding target video.

[0167] The key information set includes any combination of the following: character identifiers, character relationship information, personality trait information, character emotional information, character interaction scenarios, and event development information.

[0168] Furthermore, knowledge graph construction units may include:

[0169] A preliminary graph construction component is used to extract all entities and entity relationships from the content outline and construct a preliminary graph.

[0170] A target language terminology database determination component is used to determine the target language terminology database corresponding to the target video based on the content summary and the video summary;

[0171] A global knowledge graph utilization component is used to combine the target language terminology database, the content summary, and the global knowledge graph to optimize the preliminary graph and form a knowledge graph for the target video.

[0172] Furthermore, the adjustment module 30 may include:

[0173] The multimodal model acquisition unit is used to acquire the trained multimodal model.

[0174] The basic subtitle text optimization unit is used to optimize and adjust the basic subtitle text using the multimodal model and referring to the knowledge graph, so as to obtain the optimized subtitles output by the multimodal model.

[0175] Furthermore, the multimodal model acquisition unit may include:

[0176] The first multimodal model acquisition component is used to acquire an initial multimodal model and multiple training samples. Each training sample contains a training knowledge graph, basic training subtitles, and optimized training subtitles for the corresponding training video.

[0177] The second multimodal model acquisition component is used to train the initial multimodal model sequentially using each training sample to obtain the predicted text corresponding to each training sample output by the initial multimodal model.

[0178] The third multimodal model acquisition component is used to calculate the similarity between predicted text and training optimized captions corresponding to the same training sample.

[0179] The fourth multimodal model acquisition component is used to update the parameters of the initial multimodal model based on each similarity until the latest initial multimodal model meets the preset conditions, and the final initial multimodal model is used as the trained multimodal model.

[0180] The subtitle generation device provided in this application embodiment can be applied to subtitle generation equipment, such as PC terminals, cloud platforms, servers, and server clusters. Optionally, Figure 3 The hardware structure block diagram of the subtitle generation device is shown below. Figure 3 The hardware structure of the subtitle generation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0181] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0182] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0183] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0184] The memory stores a program, which the processor can call. The program is used for:

[0185] By combining the audio and visual content corresponding to the target video, basic subtitle text corresponding to the target video is generated;

[0186] Generate a knowledge graph containing a summary of the target video content;

[0187] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0188] Optionally, the refined and extended functions of the program can be referred to the above description.

[0189] This application embodiment also provides a readable storage medium that can store a program suitable for execution by a processor, the program being used for:

[0190] By combining the audio and visual content corresponding to the target video, basic subtitle text corresponding to the target video is generated;

[0191] Generate a knowledge graph containing a summary of the target video content;

[0192] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0193] Optionally, the refined and extended functions of the program can be referred to the above description.

[0194] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0195] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0196] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. The various embodiments of this application can be combined with each other. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating subtitles, characterized in that, include: By combining the audio and visual content corresponding to the target video, basic subtitle text corresponding to the target video is generated; Obtain the associated videos of the target video, and obtain the video summary of the associated videos and the content summary of the target video; By combining the video summary and the content outline, a global knowledge graph representing the target language is constructed. Extract all entities and entity relationships from the aforementioned content outline to construct a preliminary graph; Based on the content summary and the video summary, the target language terminology database corresponding to the target video is determined; By combining the target language terminology database, the content summary, and the global knowledge graph, the preliminary graph is optimized to form the knowledge graph of the target video. The knowledge graph is used to indicate the multiple target entities contained in the target video, the entity attributes of each target entity, and the entity relationships between the target entities. The same type of target entity corresponds to multiple different types of entity attributes. Based on the knowledge graph, characters reflecting different entity attributes of the target entity are added to the basic subtitle text to form optimized subtitles corresponding to the target video.

2. The subtitle generation method according to claim 1, characterized in that, The step of combining the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video includes: Using speech recognition technology, the audio content of the target video is converted into first subtitle text in the target language; Optical character recognition technology is used to identify and extract the visual content of the target video, and generate second subtitle text in the target language; The first subtitle text and the second subtitle text are mutually calibrated to obtain the basic subtitle text expressed in the target language.

3. The subtitle generation method according to claim 1, characterized in that, Obtaining the associated videos of the target video includes: Other videos that have a logical relationship with the target video are considered as associated videos of the target video.

4. The subtitle generation method according to claim 1, characterized in that, Obtaining a summary of the target video's content includes: Extract a set of key information from the audio and visual content of the target video, and organize the set of key information to form a content outline of the corresponding target video; The key information set includes any combination of the following: character identifiers, character relationship information, personality trait information, character emotional information, character interaction scenarios, and event development information.

5. The subtitle generation method according to claim 1, characterized in that, The step of adding characters reflecting different entity attributes of the target entity to the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video includes: Obtain a trained multimodal model; Using the multimodal model and referring to the knowledge graph, characters reflecting different entity attributes of the target entity are added to the basic subtitle text to obtain optimized subtitles output by the multimodal model.

6. The subtitle generation method according to claim 5, characterized in that, The process of obtaining the trained multimodal model includes: Obtain an initial multimodal model and multiple training samples. Each training sample contains a training knowledge graph, basic training subtitles, and optimized training subtitles for the corresponding training video. The initial multimodal model is trained sequentially using each training sample to obtain the predicted text corresponding to each training sample output by the initial multimodal model; Calculate the similarity between the predicted text and the trained optimized captions corresponding to the same training sample; Based on each similarity, the parameters of the initial multimodal model are updated until the latest initial multimodal model meets the preset conditions. The final initial multimodal model is then used as the trained multimodal model.

7. A subtitle generation device, characterized in that, include: The module is used to combine the audio and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video; A generation module is used to obtain associated videos of the target video, and obtain video summaries of the associated videos and content summaries of the target video; combine the video summaries and content summaries to construct a global knowledge graph represented by the target language; extract all entities and entity relationships from the content summaries to construct a preliminary graph; determine the target language terminology database corresponding to the target video based on the content summaries and video summaries; combine the target language terminology database, the content summaries, and the global knowledge graph to optimize the preliminary graph to form a knowledge graph of the target video. The knowledge graph is used to indicate multiple target entities contained in the target video, the entity attributes of each target entity, and the entity relationships between the target entities. Target entities of the same type correspond to multiple different types of entity attributes. The adjustment module is used to add characters reflecting different entity attributes of the target entity to the basic subtitle text based on the knowledge graph, thereby forming optimized subtitles corresponding to the target video.

8. A subtitle generation device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the subtitle generation method as described in any one of claims 1-6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the subtitle generation method as described in any one of claims 1-6.

Images