Subtitle generation method, device and equipment and readable storage medium

Abstract

The invention discloses a subtitle optimization method, device and equipment and a readable storage medium, and the method comprises the steps: generating a basic subtitle text corresponding to a target video in combination with audio content and visual content corresponding to the target video; on the basis, omission of key details of the target video can be avoided by using video sound effects and pictures; in order to improve the integrating degree of the finally generated subtitle text and the target video, the application can generate a knowledge graph containing the content infarction of the target video; adjusting the basic caption text based on the knowledge graph to form an optimized caption corresponding to the target video; based on this, in the adjustment and optimization process, targeted adjustment can be performed on the basic caption text according to the main content and entity information of the target video, and layer-by-layer progressive of the caption drama is ensured; visibly, according to the method and the device, the video audio, the video picture and the video probability can be integrated, the accuracy of the finally obtained optimized subtitles is improved, and the watching experience of video audiences is improved.

Description

Technical Field

[0001] The present application relates to the technical field of subtitles, and more specifically, to a subtitle generation method, device, equipment and readable storage medium. Background Art

[0002] In modern society, watching videos has become one of the most common entertainment activities for people. During the viewing process, viewers can further understand the video content through video subtitles. However, existing subtitle generation technology mainly relies on a general speech conversion model to convert video audio into video subtitles. It lacks consideration of situational factors such as the characters and plot in the video, resulting in the inability of the generated subtitle text to accurately fit the specific situation created by the video. Inaccurate emotional communication often occurs, which reduces the accuracy of the subtitle text and affects the audience's viewing experience. Summary of the invention

[0003] In view of this, the present application provides a subtitle generation method, apparatus, device and readable storage medium, which are used to solve the shortcoming of low accuracy of subtitle text generated in the prior art.

[0004] In order to achieve the above objectives, the proposed solution is as follows:

[0005] A subtitle generation method, comprising:

[0006] Combining the audio content and visual content corresponding to the target video, generating basic subtitle text corresponding to the target video;

[0007] Generate a knowledge graph containing a summary of the target video content;

[0008] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0009] Optionally, combining the audio content and visual content corresponding to the target video to generate a basic subtitle text corresponding to the target video includes:

[0010] Using speech recognition technology, converting the audio content of the target video into a first subtitle text in a target language;

[0011] Using optical character recognition technology to identify and extract visual content of the target video and generate a second subtitle text in the target language;

[0012] The first subtitle text and the second subtitle text are mutually calibrated to obtain a basic subtitle text expressed in the target language.

[0013] Optionally, generating a knowledge graph containing a content summary of the target video includes:

[0014] Obtaining the associated videos of the target video, and obtaining the video summary of the associated videos and the content summary of the target video;

[0015] Combining the video summary and the content summary, constructing a global knowledge graph expressed in the target language;

[0016] Based on the content outline and with reference to the global knowledge graph, a knowledge graph represented in the target language and corresponding to the target video is constructed.

[0017] Optionally, obtaining an associated video of the target video includes:

[0018] Other videos that are logically associated with the target video are used as associated videos of the target video.

[0019] Optionally, obtaining a content summary of the target video includes:

[0020] Extracting a key information set from the audio content and visual content of the target video, and organizing the key information set to form a content summary of the corresponding target video;

[0021] The key information set includes any of the following information: character identification, character relationship information, personality trait information, character emotion information, character interaction scene and thing development information.

[0022] Optionally, the step of constructing a knowledge graph corresponding to the target video and using the target language expression based on the content summary and referring to the global knowledge graph includes:

[0023] Extract all entities and entity relationships from the content outline to construct a preliminary graph;

[0024] Determining a target language term library corresponding to the target video based on the content summary and the video summary;

[0025] The preliminary graph is optimized by combining the target language terminology library, the content summary and the global knowledge graph to form a knowledge graph of the target video.

[0026] Optionally, the adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video includes:

[0027] Get the trained multimodal model;

[0028] The multimodal model is used to refer to the knowledge graph to optimize and adjust the basic subtitle text to obtain optimized subtitles output by the multimodal model.

[0029] Optionally, the acquiring a trained multimodal model includes:

[0030] Obtain an initial multimodal model and multiple training samples, each training sample including a training knowledge graph, training basic subtitles, and training optimized subtitles corresponding to a training video;

[0031] Using each training sample in turn to train the initial multimodal model, to obtain a predicted text corresponding to each training sample output by the initial multimodal model;

[0032] Calculate the similarity between the predicted text and the trained optimized subtitles corresponding to the same training sample;

[0033] Based on each similarity, the parameters of the initial multimodal model are updated until the latest initial multimodal model meets the preset conditions, and the final initial multimodal model is used as the trained multimodal model.

[0034] A subtitle optimization device, comprising:

[0035] A combining module, used to combine the audio content and the visual content corresponding to the target video to generate a basic subtitle text corresponding to the target video;

[0036] A generation module, used to generate a knowledge graph containing a summary of the target video content;

[0037] An adjustment module is used to adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0038] A subtitle optimization device, comprising a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to execute the program to implement each step of the above-mentioned subtitle optimization method.

[0041] A readable storage medium stores a computer program thereon, characterized in that when the computer program is executed by a processor, each step of the above-mentioned subtitle optimization method is implemented.

[0042] It can be seen from the above technical solutions that the subtitle generation method provided by the present application can combine the audio content and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video; based on this, the present application can use audio and vision to generate basic subtitle text; thus, the sound effects of the target video and its matching visual images can be referred to to achieve the purpose of identifying key information such as emotions and actions of the sound effect generation object, avoid missing key details of the target video, and improve the accuracy of the basic subtitle text; in order to improve the fit between the final generated subtitle text and the target video, the present application can generate a knowledge graph containing the content outline of the target video; based on the knowledge graph, the basic subtitle text is adjusted to form optimized subtitles corresponding to the target video; based on this, since the content outline can contain the main content of the target video and entity information such as character personality and character emotions, the knowledge graph can integrate the main content of the target video and various entity information. Therefore, by adjusting the basic subtitle text with reference to the knowledge graph to obtain optimized subtitles, during the optimization process, the basic subtitle text can be adjusted in a targeted manner according to the main content and entity information of the target video, which can better reflect the plot atmosphere of the target video, and can better ensure the progressiveness of the subtitle plot, thereby further improving the accuracy of the optimized subtitles. It can be seen that the present application can integrate video audio, video pictures and video outlines to improve the accuracy of the final optimized subtitles and improve the viewing experience of video viewers. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A flow chart of a subtitle generation method disclosed in an embodiment of the present application;

[0045] Figure 2 A structural block diagram of a subtitle generating device disclosed in an embodiment of the present application;

[0046] Figure 3 The following is a hardware structure block diagram of a subtitle generation device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The present application provides a subtitle generation method, which can be applied to various video playback platforms or various smart TV operating systems, and can also be applied to various computer terminals or smart terminals. The execution subject can be a processor or server of the computer terminal or smart terminal. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0049] Next, combine Figure 1 The subtitle generation method of the present application is described in detail, including the following steps:

[0050] Step S1: Combine the audio content and visual content corresponding to the target video to generate basic subtitle text corresponding to the target video.

[0051] Specifically, the audio content and visual content corresponding to the target video may be obtained.

[0052] The audio content may be converted into a first subtitle text using speech recognition technology, and the second subtitle text contained in the target video may be recognized using optical vision technology;

[0053] The first subtitle text and the second subtitle text are mutually corrected to obtain a basic subtitle text.

[0054] The target video may be any type of playable video, such as any segment of a TV series, any episode of a variety show, or any segment of a cartoon.

[0055] Step S2: Generate a knowledge graph containing a summary of the target video content.

[0056] Specifically, the content summary of the target video can be generated by referring to the audio content and visual content of the target video, and the data information reflecting the target video playback content such as the comment data and introduction information corresponding to the target video;

[0057] The content outline can be organized into a knowledge graph.

[0058] Among them, the knowledge graph can be used to indicate the multiple target entities contained in the target video, the entity attributes of each target entity, and the entity relationships between the target entities.

[0059] Among them, the same type of target entity can correspond to multiple different types of entity attributes.

[0060] For example, when the type of the target entity is a person, its entity attributes may correspond to entity attributes such as person emotions, person personalities, person relationships, and person behaviors.

[0061] When the type of the target entity is an event class, its entity attributes may correspond to entity attributes such as time, location, and key plot development.

[0062] The same type of target entity can exist for different types of target videos.

[0063] For example, a target video of the type of TV series and a target video of the type of variety show can both correspond to a character-type target entity.

[0064] The same target video can contain multiple different types of target entities.

[0065] For example, target entity types in some target videos may include person classes and event classes.

[0066] Step S3: adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0067] Specifically, the basic subtitle text can be adjusted in detail from multiple angles with reference to the knowledge graph, and optimized subtitles corresponding to the target video can be obtained after the adjustments.

[0068] For example, we can refer to the knowledge graph and add speaker identification to each sentence of the basic subtitle text;

[0069] You can also refer to the emotions of each character in the knowledge graph and add words that reflect the emotions of the corresponding characters in the subtitle fragments corresponding to the basic subtitle text;

[0070] You can also refer to the personalities of each character in the knowledge graph and add words that reflect the corresponding character's personality in the subtitle segment corresponding to the basic subtitle text.

[0071] It can be seen from the above technical solutions that the subtitle generation method provided by the present application can combine the audio content and visual content corresponding to the target video to generate the basic subtitle text corresponding to the target video; based on this, the present application can use audio and vision to generate basic subtitle text; thus, the sound effects of the target video and its matching vision can be used to achieve the purpose of identifying key information such as emotions and actions of the sound effect generation object, avoid missing key details of the target video, and improve the accuracy of the basic subtitle text; in order to improve the fit between the final generated subtitle text and the target video, the present application can generate a knowledge graph containing the content outline of the target video; based on the knowledge graph, the basic subtitle text is adjusted to form optimized subtitles corresponding to the target video; based on this, since the content outline can include the main content of the target video and entity information such as character personality and character emotions, the knowledge graph can integrate the main content of the target video and various entity information. Therefore, by adjusting the basic subtitle text with reference to the knowledge graph to obtain optimized subtitles, during the optimization process, the basic subtitle text can be adjusted in a targeted manner according to the main content and entity information of the target video, which can better reflect the plot atmosphere of the target video, and can better ensure the progressiveness of the subtitle plot, thereby further improving the accuracy of the optimized subtitles. It can be seen that the present application can integrate video audio, video pictures and video outlines to improve the accuracy of the final optimized subtitles and improve the viewing experience of video viewers.

[0072] Furthermore, in order to ensure the reliability of the optimized subtitles, the finally obtained optimized subtitles may be compared with the original subtitles of the target video to evaluate the quality of the optimized subtitles.

[0073] Furthermore, evaluation indicators such as BLEU and ROUGE, as well as embedding models such as METEOR, BERT and GPT, can be used to calculate the semantic similarity between the optimized subtitles and the original subtitles and evaluate the quality of the optimized subtitles.

[0074] User feedback results may also be received, and the subtitle generation method of the present application may be further optimized based on the user feedback results.

[0075] The optimized subtitles can be embedded into the target video in an embedded manner.

[0076] In some embodiments of the present application, the process of generating a basic subtitle text corresponding to the target video by combining the audio content and visual content corresponding to the target video in step S1 is described in detail, and the steps are as follows:

[0077] S10: Using speech recognition technology, convert the audio content of the target video into a first subtitle text in a target language.

[0078] Specifically, ASR technology can be used to transcribe audio content into text;

[0079] Determine the target language;

[0080] The target language can be set according to actual needs and can be any dialect or any official language.

[0081] If the target language and the language of the target video are not the same language, a translation technique is used to translate the text into a first subtitle text;

[0082] If the target language and the language of the target video are the same language, the text is used as the first subtitle text.

[0083] S11. Using optical character recognition technology, identify and extract the visual content of the target video, and generate a second subtitle text in the target language.

[0084] Specifically, OCR technology can be used to extract text content appearing in the target video, such as subtitles, signs, etc.;

[0085] If the target language and the language of the target video are not the same language, translation technology is used to translate the text content into a second subtitle text;

[0086] If the target language and the language of the target video are the same language, the text content is used as the second subtitle text.

[0087] Among them, the first subtitle text refers to the subtitle text generated based on the audio content of the target video; the second subtitle text refers to the subtitle text produced based on the visual content of the target video.

[0088] The information included in the first subtitle text and the second subtitle text may be completely the same, or there may be some differences.

[0089] S12: calibrate the first subtitle text and the second subtitle text to obtain a basic subtitle text expressed in the target language.

[0090] Specifically, the first subtitle text and the second subtitle text may be referenced to each other, and the first subtitle text and the second subtitle text may be corrected to obtain a basic subtitle text after the correction.

[0091] It can be seen from the above technical solution that this embodiment provides an optional method for generating basic subtitle text corresponding to the target video. Through the above method, the present application can combine speech recognition technology and optical character recognition technology to generate basic subtitle text corresponding to the target language, further better integrate audio and picture, and construct basic subtitle text. At the same time, the generated basic subtitle text can also be expressed in the target language. Based on this, the subtitles generated by this application do not necessarily correspond to the same language as the target video. This application can be used for video translation to generate subtitle text in a specific language, further improving the wide application of this application.

[0092] In some embodiments of the present application, the process of step S2, generating a knowledge graph containing a summary of the target video content, is described in detail, and the steps are as follows:

[0093] S20, obtaining the associated videos of the target video, and obtaining the video summary of the associated videos and the content summary of the target video.

[0094] Specifically, all videos related to the target video may be obtained as associated videos;

[0095] Various key information can be extracted from the audio, visual images, comments and video introduction of each related video, and the various key information can be sorted and combined to obtain a video summary of the related video.

[0096] S21. Combining the video summary and the content outline, construct a global knowledge graph expressed in the target language.

[0097] Specifically, all entities in the content summary and each video summary can be refined, and all relationships in the content summary and each video summary can be sorted out;

[0098] Build a global knowledge graph based on each entity and entity relationship.

[0099] S22. Based on the content summary and with reference to the global knowledge graph, a knowledge graph is constructed that is expressed in the target language and corresponds to the target video.

[0100] Specifically, based on the content summary and using the global knowledge graph as reference information, a knowledge graph expressed in the target language and corresponding to the target video can be constructed.

[0101] It can be seen from the above technical solution that this embodiment provides an optional method for generating a knowledge graph containing a content summary of a target video. Through the above method, the content summary of the target video and the video summary of associated videos related to the target video can be integrated to construct a knowledge graph, further improving the reliability of the knowledge graph and the continuity of the knowledge graph.

[0102] In some embodiments of the present application, the process of obtaining the associated video of the target video in step S20 is described in detail, and the steps are as follows:

[0103] S200: Use other videos that are logically associated with the target video as associated videos of the target video.

[0104] Specifically, the logical association relationship can be any one or more of the following relationships: content association relationship, narrative logic relationship, character creation relationship and theme presentation relationship.

[0105] Content relevance relationships may include plot coherence, story continuity, and scene connection, etc.

[0106] Narrative logical relationships can include the relationship between clues and revelation and the relationship between the main line.

[0107] Character creation relationships can include character growth relationships, character development relationships, and character inheritance relationships.

[0108] It can be seen from the above technical solution that this embodiment provides an optional method for obtaining related videos. The above method can be used to analyze the association relationship between the target video and other videos, thereby determining the related videos of the target video.

[0109] In some embodiments of the present application, the process of obtaining the content summary of the target video in step S20 is described in detail, and the steps are as follows:

[0110] S201, extracting a key information set from the audio content and visual content of the target video, and organizing the key information set to form a content summary of the corresponding target video.

[0111] Specifically, the key information type of the content summary can be determined according to the type of the target video;

[0112] Key information matching the key information type is extracted from the audio content, visual content, comment data and introduction information of the target video to form a key information set.

[0113] For example, when the target video is a situational drama, character identification, character relationship information, personality trait information, character emotion information, character interaction scenes and / or event development information can be extracted from the audio content, visual content, commentary data and introduction information to form a key information set.

[0114] When the target video is a singing variety show, the key information set consisting of singer identification, song tracks, singer emotions, singer relationships and / or singing events can be extracted from the audio content, visual content, comment data and introduction information.

[0115] It can be seen from the above technical solution that this embodiment provides an optional method for obtaining the content summary of the target video. Through the above method, key information can be extracted in a targeted manner to construct a content summary, thereby further improving the reliability of the knowledge graph of this application.

[0116] In some embodiments of the present application, the process of step S22, based on the content summary and referring to the global knowledge graph, constructing a knowledge graph using the target language representation and corresponding to the target video is described in detail, and the steps are as follows:

[0117] S220: extract all entities and entity relationships from the content summary to construct a preliminary graph.

[0118] Specifically, the content summary can be segmented, POS tagged, and named entity recognized to extract entities and entity relationships;

[0119] Take each entity as a node, build edges with reference to the relationship between each entity, and form a preliminary graph.

[0120] S221. Determine a target language term library corresponding to the target video based on the content summary and the video summary.

[0121] Specifically, the professional fields involved in the content outline and the video summary can be analyzed to determine the target language terminology library that should be expressed in the target language.

[0122] S222. Optimize the preliminary graph by combining the target language terminology library, the content summary and the global knowledge graph to form a knowledge graph for the target video.

[0123] Specifically, the non-standard words in the preliminary map are optimized using the target language terminology library;

[0124] With reference to the content outline and the global knowledge graph, the optimized preliminary graph is adjusted in detail to obtain the knowledge graph of the target video.

[0125] The knowledge graph can be used to indicate multiple target entities contained in the target video, the entity attributes of each target entity in the target video, and the entity relationships between the target entities. In addition, it can also include the entity attributes of each target entity in each associated video and the entity relationships between the target entities.

[0126] It can be seen from the above technical solution that this embodiment provides an optional method of constructing a knowledge graph representing the target language of the application based on the content outline and the global knowledge graph. Through the above method, the knowledge graph can be further optimized and adjusted through the target language terminology library, thereby further ensuring the professionalism of the optimized subtitles generated by this application and further improving the audience's viewing experience.

[0127] In some embodiments of the present application, the process of adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video in step S3 is described in detail, and the steps are as follows:

[0128] S30: Obtain a trained multimodal model.

[0129] Specifically, a trained VLM model may be obtained.

[0130] The VLM model may include a matching layer and a basic subtitle text processing layer.

[0131] S31. Utilizing the multimodal model and referring to the knowledge graph, the basic subtitle text is optimized and adjusted to obtain optimized subtitles output by the multimodal model.

[0132] Specifically, the knowledge graph and the basic subtitle text can be input into the multimodal model, and the matching layer can be used to match each sentence in the basic subtitle text with each node of the knowledge graph;

[0133] Using the basic subtitle text processing layer, detailed adjustments are made to each sentence based on matching nodes to obtain optimized subtitles.

[0134] It can be seen from the above technical solution that this embodiment provides an optional method for adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video. Through the above method, the trained multimodal model can be used to complete the adjustment of the basic subtitle text, thereby further ensuring the accuracy of the optimized subtitles.

[0135] In some embodiments of the present application, the process of obtaining a trained multimodal model in step S30 is described in detail, and the steps are as follows:

[0136] S300, obtaining an initial multimodal model and a plurality of training samples, each training sample comprising a training knowledge graph, training basic subtitles and training optimized subtitles corresponding to a training video.

[0137] Specifically, an initial multimodal model may be constructed, and a plurality of training samples including a training knowledge graph corresponding to a training video, training basic subtitles, and training optimized subtitles may be obtained.

[0138] Among them, training optimized subtitles and training basic subtitles can be expressed in different types of languages.

[0139] The trained optimized subtitles can be subtitles adjusted by professionals.

[0140] S301. Train the initial multimodal model using each training sample in turn to obtain a predicted text corresponding to each training sample output by the initial multimodal model.

[0141] Specifically, the initial multimodal model can be trained based on each training sample to obtain the predicted text corresponding to each training sample.

[0142] S302: Calculate the similarity between the predicted text and the training optimized subtitles corresponding to the same training sample.

[0143] Specifically, the predicted text and the trained optimized subtitles may be segmented, POS tagged, and named entity recognized, and the similarity between each predicted text and its corresponding trained optimized subtitle may be calculated.

[0144] There are many ways to calculate the similarity between each predicted text and its corresponding trained optimized subtitle.

[0145] For example, the similarity between each predicted text and its corresponding trained optimized subtitle can be calculated by using the Euclidean distance calculation expression.

[0146] The similarity between each predicted text and its corresponding trained optimized subtitle can also be calculated by using the cosine distance calculation expression.

[0147] S303: Based on each similarity, update the parameters of the initial multimodal model until the latest initial multimodal model meets the preset conditions, and use the final initial multimodal model as the trained multimodal model.

[0148] Specifically, the initial multimodal model is iteratively updated based on each similarity in turn until the similarity between the predicted text output by the latest initial multimodal model and the corresponding training optimized subtitles is less than a preset similarity threshold, and the final initial multimodal model is used as the trained multimodal model.

[0149] It can be seen from the above technical solution that this embodiment provides an optional method for training a multimodal model. Through the above method, the multimodal model can be trained with reference to the similarity, ensuring that the multimodal model can make detail adjustments while ensuring the accuracy of the subtitle text, thereby improving the reliability of the optimized subtitles.

[0150] Next, we will combine Figure 2 The subtitle optimization device provided in the present application is introduced in detail. The subtitle optimization device provided below can be compared with the subtitle optimization method provided above.

[0151] See also Figure 2 It can be found that the subtitle optimization device may include:

[0152] A combining module 10 is used to combine the audio content and the visual content corresponding to the target video to generate a basic subtitle text corresponding to the target video;

[0153] A generation module 20, used to generate a knowledge graph containing a content summary of the target video;

[0154] The adjustment module 30 is used to adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0155] Further, the combination module 10 may include:

[0156] A first combining unit, configured to convert the audio content of the target video into a first subtitle text in a target language by using a speech recognition technology;

[0157] A second combining unit is used to use optical character recognition technology to recognize and extract the visual content of the target video and generate a second subtitle text in the target language;

[0158] The third combining unit is used to calibrate the first subtitle text and the second subtitle text to obtain a basic subtitle text expressed in the target language.

[0159] Further, the generating module 20 may include:

[0160] An associated video acquisition unit, used to acquire associated videos of the target video, and acquire video summaries of the associated videos and a content summary of the target video;

[0161] A global knowledge graph construction unit, used to construct a global knowledge graph expressed in a target language by combining the video summary and the content summary;

[0162] A knowledge graph construction unit is used to construct a knowledge graph represented by the target language and corresponding to the target video based on the content outline and with reference to the global knowledge graph.

[0163] Further, the associated video acquisition unit may include:

[0164] The logical association relationship identification component is used to take other videos that have a logical association relationship with the target video as associated videos of the target video.

[0165] Furthermore, the associated video acquisition unit may further include:

[0166] A key information set extraction component is used to extract a key information set from the audio content and visual content of the target video, and to organize the key information set to form a content summary of the corresponding target video;

[0167] The key information set includes any of the following information: character identification, character relationship information, personality trait information, character emotion information, character interaction scene and thing development information.

[0168] Furthermore, the knowledge graph construction unit may include:

[0169] A preliminary graph building component, used to extract all entities and entity relationships from the content outline to build a preliminary graph;

[0170] A target language term library determination component, used to determine a target language term library corresponding to the target video based on the content summary and the video summary;

[0171] The global knowledge graph utilizes a component for optimizing the preliminary graph by combining the target language terminology library, the content outline and the global knowledge graph to form a knowledge graph of the target video.

[0172] Further, the adjustment module 30 may include:

[0173] A multimodal model acquisition unit, used to acquire a trained multimodal model;

[0174] The basic subtitle text optimization unit is used to optimize and adjust the basic subtitle text by using the multimodal model and referring to the knowledge graph to obtain optimized subtitles output by the multimodal model.

[0175] Furthermore, the multimodal model acquisition unit may include:

[0176] A first multimodal model acquisition component is used to acquire an initial multimodal model and a plurality of training samples, each training sample comprising a training knowledge graph, training basic subtitles, and training optimized subtitles corresponding to a training video;

[0177] A second multimodal model acquisition component, used to train the initial multimodal model using each training sample in turn to obtain a predicted text corresponding to each training sample output by the initial multimodal model;

[0178] A third multimodal model acquisition component, used to calculate the similarity between the predicted text and the training optimized subtitle corresponding to the same training sample;

[0179] The fourth multimodal model acquisition component is used to update the parameters of the initial multimodal model based on each similarity until the latest initial multimodal model meets the preset conditions, and the final initial multimodal model is used as the trained multimodal model.

[0180] The subtitle optimization device provided in the embodiment of the present application can be applied to subtitle optimization devices, such as PC terminals, cloud platforms, servers and server clusters. Figure 3 The hardware structure diagram of the subtitle optimization device is shown in FIG. Figure 3 ,The hardware structure of the subtitle optimization device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0181] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0182] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0183] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0184] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0185] Combining the audio content and visual content corresponding to the target video, generating basic subtitle text corresponding to the target video;

[0186] Generate a knowledge graph containing a summary of the target video content;

[0187] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0188] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0189] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0190] Combining the audio content and visual content corresponding to the target video, generating basic subtitle text corresponding to the target video;

[0191] Generate a knowledge graph containing a summary of the target video content;

[0192] The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

[0193] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0195] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0196] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A subtitle generation method, characterized in that: include: Combining the audio content and visual content corresponding to the target video, generating basic subtitle text corresponding to the target video; Generate a knowledge graph containing a summary of the target video content; The basic subtitle text is adjusted based on the knowledge graph to form optimized subtitles corresponding to the target video.

2. The subtitle generation method according to claim 1, characterized in that: The step of combining the audio content and the visual content corresponding to the target video to generate a basic subtitle text corresponding to the target video includes: Using speech recognition technology, converting the audio content of the target video into a first subtitle text in a target language; Using optical character recognition technology to identify and extract visual content of the target video and generate a second subtitle text in the target language; The first subtitle text and the second subtitle text are mutually calibrated to obtain a basic subtitle text expressed in the target language.

3. The subtitle generation method according to claim 1, characterized in that: The generating of a knowledge graph containing a content summary of the target video includes: Obtaining the associated videos of the target video, and obtaining the video summary of the associated videos and the content summary of the target video; Combining the video summary and the content summary, constructing a global knowledge graph expressed in the target language; Based on the content outline and with reference to the global knowledge graph, a knowledge graph represented in the target language and corresponding to the target video is constructed.

4. The subtitle generation method according to claim 3, characterized in that: Obtaining the associated videos of the target video, including: Other videos that are logically associated with the target video are used as associated videos of the target video.

5. The subtitle generation method according to claim 3, characterized in that: Obtaining a content summary of the target video, including: Extracting a key information set from the audio content and visual content of the target video, and organizing the key information set to form a content summary of the corresponding target video; The key information set includes any of the following information: character identification, character relationship information, personality trait information, character emotion information, character interaction scene and thing development information.

6. The subtitle generation method according to claim 3, characterized in that: The step of constructing a knowledge graph corresponding to the target video and using the target language representation based on the content summary and referring to the global knowledge graph includes: Extract all entities and entity relationships from the content outline to construct a preliminary graph; Determining a target language term library corresponding to the target video based on the content summary and the video summary; The preliminary graph is optimized by combining the target language terminology library, the content summary and the global knowledge graph to form a knowledge graph of the target video.

7. The subtitle generation method according to claim 1, characterized in that: The adjusting the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video includes: Get the trained multimodal model; The multimodal model is used to refer to the knowledge graph to optimize and adjust the basic subtitle text to obtain optimized subtitles output by the multimodal model.

8. The subtitle generation method according to claim 7, characterized in that: The obtaining of the trained multimodal model comprises: Obtain an initial multimodal model and multiple training samples, each training sample including a training knowledge graph, training basic subtitles, and training optimized subtitles corresponding to a training video; Using each training sample in turn to train the initial multimodal model, to obtain a predicted text corresponding to each training sample output by the initial multimodal model; Calculate the similarity between the predicted text and the trained optimized subtitles corresponding to the same training sample; Based on each similarity, the parameters of the initial multimodal model are updated until the latest initial multimodal model meets the preset conditions, and the final initial multimodal model is used as the trained multimodal model.

9. A subtitle optimization device, characterized in that: include: A combining module, used to combine the audio content and the visual content corresponding to the target video to generate a basic subtitle text corresponding to the target video; A generation module, used to generate a knowledge graph containing a summary of the target video content; An adjustment module is used to adjust the basic subtitle text based on the knowledge graph to form optimized subtitles corresponding to the target video.

10. A subtitle optimization device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the subtitle generation method according to any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the subtitle generation method according to any one of claims 1 to 8 is implemented.

Images