Subtitle generation method and training method and device of subtitle generation model

Abstract

The invention provides a subtitle generation method and a training method and device of a subtitle generation model, and relates to the technical field of computers, in particular to the technical fields of subtitle generation, neural networks, large language models and the like. According to the specific implementation scheme, reason features and result features are extracted from a video; and generating subtitles of the video based on the cause feature and the result feature. According to the method, the subtitles with the causal sequential relationship can be generated based on the reason features and the result features in the video, the subtitles better conform to the video content, and the accuracy and logicality of the subtitles can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as subtitle generation, neural networks, and large language models. Background Art

[0002] Video subtitles can be generated by using video editing software, subtitle generation software or some subtitle generation platforms through manual settings, voice recognition and other methods. For example, the voice in the video can be converted into text content, and the subtitle content can be further improved on the editing page, including adjusting the timeline, merging or splitting sentences, etc. After processing, the subtitle file can be embedded into the original video. However, most video subtitles only describe the voice or visible content in the video. Summary of the invention

[0003] The present invention provides a subtitle generation method, a subtitle generation model training method and a device.

[0004] According to one aspect of the present disclosure, a subtitle generation method is provided, comprising:

[0005] Extract cause features and effect features from videos;

[0006] Based on the cause feature and the result feature, a subtitle of the video is generated.

[0007] According to another aspect of the present disclosure, a method for training a subtitle generation model is provided, comprising:

[0008] Input the video samples of the training samples into the subtitle generation model to be trained, and obtain the output sample cause features and sample result features respectively;

[0009] Based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample, the subtitle generation model to be trained is trained through a loss function to obtain a trained subtitle generation model.

[0010] According to another aspect of the present disclosure, there is provided a subtitle generating device, comprising:

[0011] An extraction module, used to extract cause features and result features from the video;

[0012] A generating module is used to generate subtitles for the video based on the cause feature and the result feature.

[0013] According to another aspect of the present disclosure, a training device for a subtitle generation model is provided, comprising:

[0014] An input module, used to input the video sample of the training sample into the subtitle generation model to be trained, and obtain output sample cause features and sample result features respectively;

[0015] The training module is used to train the subtitle generation model to be trained through a loss function based on the sample cause feature, the sample result feature, and the benchmark subtitle of the training sample to obtain a trained subtitle generation model.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] According to the present disclosure, based on the cause features and result features in the video, subtitles with causal temporal relationships can be generated. Such subtitles are more consistent with the video content and can improve the accuracy and logic of the subtitles.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a flowchart of a subtitle generation method according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of a subtitle generation method according to another embodiment of the present disclosure;

[0027] Figure 3 is a flowchart of a method for training a subtitle generation model according to an embodiment of the present disclosure;

[0028] Figure 4is a flowchart of a method for training a subtitle generation model according to another embodiment of the present disclosure;

[0029] Figure 5 is a flowchart of a method for training a subtitle generation model according to another embodiment of the present disclosure;

[0030] Figure 6 is a flowchart of a method for training a subtitle generation model according to another embodiment of the present disclosure;

[0031] Figure 7 is a structural diagram of a causal narrative subtitle generation model according to an embodiment of the present disclosure;

[0032] Figure 8 is a structural schematic diagram of a subtitle generating device according to an embodiment of the present disclosure;

[0033] Fig. 9 is a structural schematic diagram of a subtitle generating device according to another embodiment of the present disclosure;

[0034] Fig.10 is a structural schematic diagram of a training device for a subtitle generation model according to an embodiment of the present disclosure;

[0035] Fig.11 is a structural schematic diagram of a training device for a subtitle generation model according to an embodiment of the present disclosure;

[0036] Fig.12 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] While improving the accuracy and fluency of subtitles, subtitle generation methods often overlook the importance of causal temporal narrative in videos. Causal temporal narrative, which can be understood as the causal relationship between events and their unfolding in time, is the key to a deep understanding of video content. Most video subtitle datasets (such as Microsoft Research Video to Text (MSR-VTT) and Microsoft Research Video Description Corpus (MSVD)) only describe the visible content in the video and lack clear causal narrative expressions, which limits the performance of video understanding models.

[0039] Figure 1 FIG. 1 is a flow chart of a subtitle generation method 100 according to an embodiment of the present disclosure. In one implementation, the method includes:

[0040] S101, extracting cause features and result features from the video;

[0041] S102: Generate subtitles for the video based on the cause feature and the result feature.

[0042] In the disclosed embodiments, a video may include one or more events. The occurrence of an event may have specific causes and results (or effects). For example, an event occurs in a video: a car is stuck in a mud pit during driving, resulting in the inability to continue driving. In this event, the cause of the occurrence may include "the car is stuck in a mud pit", and the result of the occurrence may include "the inability to drive".

[0043] In an embodiment of the present disclosure, a video may include multiple video frame images. Cause features of an event may be extracted from some video frame images, and result features of an event may be extracted from some video frame images. There may also be some video frame images in which no event occurs, and there are neither cause features nor result features. Cause features and / or result features may include text, vectors, etc., and may be specifically set according to requirements. Based on the cause features and result features of an event, subtitles of the event may be generated. If multiple groups of cause features and result features of multiple events are extracted from a video, subtitles corresponding to each group of cause features and result features may be generated respectively, and the generated subtitles may be associated with the time when the event occurred.

[0044] In the disclosed embodiment, the identification information of the event, such as the name, number, etc., can be identified from the video, and then the cause feature and result feature of the event can be extracted, and then the subtitles of the event can be generated. Alternatively, instead of identifying the identification information of the event from the video, each group of cause features and result features can be directly extracted to generate subtitles corresponding to each group of cause features and result features.

[0045] Based on the cause features and result features in the video, subtitles with causal temporal relationships can be generated. Such subtitles are more consistent with the video content and can improve the accuracy and logic of the subtitles.

[0046] Figure 2 2 is a flow chart of a subtitle generation method 200 according to another embodiment of the present disclosure. The method 200 can be used to implement step S101 in the subtitle generation method 100. In one embodiment, the cause feature and the result feature are extracted from the video, and S101 further includes:

[0047] S201, extracting the reason feature from the video through a first encoder of a subtitle generation model;

[0048] S202: Extract the result feature from the video through a second encoder of the subtitle generation model.

[0049] In an embodiment of the present disclosure, a subtitle generation model can be obtained by training based on a neural network model. The subtitle generation model may include multiple encoders. Among them, the first encoder and the second encoder may be two independent video encoders. The two video encoders may be trained separately based on their respective loss functions, or they may be jointly trained based on the same loss function. The training samples may include video samples, and cause annotations and result annotations may be pre-added to the video frame images of the video samples (which may be referred to as video frames for short). The annotated samples are input into the model for training, and the trained model may extract cause features and result features from the video. In the embodiment of the present disclosure, the execution order of extracting cause features and extracting result features may not have a timing requirement, and the cause features may be extracted first and then the result features, or the result features may be extracted first and then the cause features, or both may be extracted in parallel.

[0050] Through the first encoder and the second encoder, causal features and result features can be accurately extracted from the video.

[0051] In one embodiment, Figure 2 As shown, the method 200 can be used to implement step S102 in the subtitle generation method 100. In one embodiment, the subtitles of the video are generated based on the cause feature and the result feature, and S102 further includes:

[0052] S203, converting the reason feature into a reason representation through a third encoder of the subtitle generation model;

[0053] S204, converting the result feature into a result representation through a fourth encoder of the subtitle generation model;

[0054] S205, connecting the cause representation and the result representation to obtain a joint representation;

[0055] S206 . Decoding the joint representation through a decoder of the subtitle generation model to obtain subtitles for the video.

[0056] In the disclosed embodiment, the third encoder and the fourth encoder of the subtitle generation model may adopt a transformer architecture, and therefore, the third encoder and the fourth encoder may be referred to as a transcoder. The encoder of the transformer architecture may convert input features into output representations. For example, the input features may include individual word segments obtained based on a text sequence, or may include the result of word segmentation vectorization processing. The output representation may include a probability distribution for representing a certain probability corresponding to the output result.

[0057] In the disclosed embodiment, after the third encoder converts the cause feature into the cause representation, and the fourth encoder converts the result feature into the result representation, the cause representation and the result representation of a certain event or a certain group can be connected to obtain a joint representation of the event or the group. For example, the cause representation hc1 and the result representation he1 of event E1 are connected to obtain a joint representation {hc1, he1}. For another example, the cause representation hc2 and the result representation he2 of group G2 are connected to obtain a joint representation {hc2, he2}. The cause representation and the result representation can be connected in the decoder to obtain a joint representation, or the cause representation and the result representation can be connected to obtain a joint representation, and then the joint representation is input into the decoder. The decoder can adopt a sequence-to-sequence model in the Transformer architecture (such as a Uni-VL model). The decoder can predict the words to be generated in the next step based on the joint representation and the previously generated words, thereby generating subtitles with a causal temporal relationship.

[0058] In the disclosed embodiment, after obtaining the subtitles corresponding to the video, the similarity between the subtitles and the video content can be calculated based on the generated subtitles using algorithms such as EMScore, and the score of the generated subtitles can be calculated based on the similarity. For an example of a specific scoring method, see the relevant description in the above embodiment. When the score of the generated subtitles is higher than a set threshold, the subtitles are used as the subtitles corresponding to the video. When the score of the generated subtitles is lower than the set threshold, the steps of the above embodiment are re-executed to generate new subtitles until the score of the new subtitles meets the threshold requirement.

[0059] Through the third encoder, the fourth encoder and the decoder, the cause feature and the result feature can be encoded and connected to generate subtitles of the video with a causal temporal relationship.

[0060] In one embodiment, the training samples of the subtitle generation model include video samples and benchmark subtitles, and the benchmark subtitles are generated based on a large language model. In the disclosed embodiment, the training samples of the subtitle generation model can be mainly used to train the first encoder and the second encoder. The annotation method of the video sample can refer to the above example, and the specific process of training the subtitle generation model based on the video sample and the benchmark subtitle can refer to the relevant description in the training method of the subtitle generation model below. The benchmark subtitles generated based on the large prediction model may include a cause part and a result part. By training the subtitle generation model with training samples consisting of video samples and benchmark subtitles, the accuracy of the subtitles generated by the subtitle generation model can be improved, and narrative subtitles with accurate causal temporal relationships can be obtained.

[0061] Figure 3 FIG. 3 is a flow chart of a method 300 for training a subtitle generation model according to an embodiment of the present disclosure. In one implementation, the method includes:

[0062] S301, inputting the video sample of the training sample into the subtitle generation model to be trained, and obtaining output sample cause features and sample result features respectively;

[0063] S302: Based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample, the subtitle generation model to be trained is trained by using a loss function to obtain a trained subtitle generation model.

[0064] In an embodiment of the present disclosure, the training sample of the subtitle generation model to be trained may include a video sample and a benchmark subtitle. The video sample may include one or more events. An event may have one or more causes, and one or more results. The benchmark subtitle may include a cause part and a result part. The subtitle generation model may be optimized based on a loss function constructed by combining sample cause features extracted from the video sample with the cause part of the benchmark subtitle. The subtitle generation model may also be optimized based on a loss function constructed by combining sample result features extracted from the video sample with the result part of the benchmark subtitle. The cause feature extraction part and the result feature extraction part of the subtitle generation model may be trained separately based on different loss functions, or the cause feature extraction part and the result feature extraction part of the subtitle generation model to be trained may be jointly trained based on the same loss function.

[0065] Through training, a subtitle generation model is obtained that can learn the causal features and result features in the video. The causal features and result features can be accurately extracted from the video, and then subtitles with causal temporal relationships can be generated.

[0066] Figure 44 is a flow chart of a method 400 for training a subtitle generation model according to another embodiment of the present disclosure. The method 400 can be used to implement step S301 in the method 300 for training a subtitle generation model. The video sample includes a cause annotation of a video frame and a result annotation of the video frame; in one embodiment, the video sample of the training sample is input into the subtitle generation model to be trained, and the output sample cause feature and sample result feature are obtained respectively, and S301 further includes:

[0067] S401, inputting the video sample into the first encoder of the subtitle generation model to be trained to obtain the output reason feature of the sample;

[0068] S402: Input the video sample into the second encoder of the subtitle generation model to be trained to obtain output result features of the sample.

[0069] In the disclosed embodiment, the cause feature extraction part of the subtitle generation model may be a first encoder, and the result feature extraction part of the subtitle generation model may be a second encoder. The first encoder and the second encoder may be two independent video encoders, such as video encoders trained based on the CLIP-ViT visual model. The video cause encoder (first encoder) and the video result encoder (second encoder) may process the cause and effect features and result features of the video frame image, respectively. In the video frame image of the video sample, cause annotations and result annotations may be pre-added. For example, there are no annotations in the 1st to 20th video frames of the video sample, the 21st to 40th video frames include cause annotations, and the 41st to 60th video frames include result annotations. In the same event, there may be multiple video frames with the same cause annotations and multiple video frames with the same result annotations; there may also be multiple video frames with different cause annotations and multiple video frames with different result annotations. For example, the 21st to 30th video frames of the video sample include cause annotation 1, the 31st to 40th video frames include cause annotation 2, and the 41st to 60th video frames include result annotation 1. This situation can be understood as: multiple causes lead to one result. It is also possible that one cause leads to one result, or one cause leads to multiple results. After training the subtitle generation model with training samples, the first encoder and the second encoder of the subtitle generation model can extract cause features and result features from the video respectively. One cause feature may be associated with one or more result features, and one result feature may be associated with one or more cause features. Cause features and result features with associated relationships can be set as a group.

[0070] By training with labeled samples, a first encoder and a second encoder are obtained that can learn the causal features and the result features in the video, and can accurately extract the causal features and the result features from the video.

[0071] In one embodiment, Figure 4 As shown, the method 400 can be used to implement step S302 in the training method 300 for a subtitle generation model. In one embodiment, based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample, the subtitle generation model is trained by a loss function to obtain a trained subtitle generation model, and S302 further includes at least one of the following:

[0072] S403, obtaining a first loss function value based on the sample reason feature and the reference subtitle reason feature, and training the first encoder based on the first loss function value to obtain a trained first encoder;

[0073] S404: Obtain a second loss function value based on the sample result feature and the result feature of the reference subtitle, and train the second encoder based on the second loss function value to obtain a trained second encoder.

[0074] In the disclosed embodiment, the first encoder and the second encoder can be trained independently using different loss functions. Based on the cause features output by the subtitle generation model and the cause features of the cause part of the benchmark subtitles in the sample, a first loss function can be constructed. The cause features output by the model and the cause features of the benchmark subtitles are substituted into the first loss function to calculate the first loss function value. In the case where the first loss function value does not meet the convergence condition, the parameters of the first encoder can be adjusted. In the case where the first loss function value meets the convergence condition, the optimization of the first encoder can be stopped to obtain the trained first encoder. Based on the result features output by the subtitle generation model and the result features of the result part of the benchmark subtitles in the sample, a second loss function can be constructed. The result features output by the model and the result features of the benchmark subtitles are substituted into the second loss function to calculate the second loss function value. In the case where the second loss function value does not meet the convergence condition, the parameters of the second encoder can be adjusted. In the case where the second loss function value meets the convergence condition, the optimization of the second encoder can be stopped to obtain the trained second encoder.

[0075] In an embodiment of the present disclosure, the first encoder and the second encoder may be jointly trained using the same loss function. A target loss function may be constructed based on the cause features and result features output by the subtitle generation model, and the cause features and result features of the cause part of the benchmark subtitles in the sample. The cause features and result features output by the model, and the cause features and result features of the benchmark subtitles are substituted into the loss function to calculate the target loss function value. In the case where the target loss function value does not meet the convergence condition, the parameters of the first encoder and / or the second encoder may be adjusted. In the case where the target loss function value meets the convergence condition, the optimization of the first encoder and / or the second encoder may be stopped to obtain the trained first encoder and / or the second encoder.

[0076] The trained first encoder is connected to the third encoder, the second encoder is connected to the fourth encoder, and the third encoder and the fourth encoder are connected to the decoder. The third encoder converts the cause feature output by the first encoder into a cause representation, and the fourth encoder converts the result feature output by the second encoder into a result representation, and the cause representation and the result representation can be connected to obtain a joint representation. The cause representation and the result representation can be connected in the decoder to obtain a joint representation, or the cause representation and the result representation can be connected to obtain a joint representation first, and then the joint representation is input into the decoder. One or more of the third encoder, the fourth encoder and the decoder can be pre-trained, and can also be trained jointly with the first encoder and / or the second encoder.

[0077] The contrast loss can be used to optimize the encoders of the video causes and results of the subtitle generation model, which can learn and approach the text description embedding corresponding to the benchmark subtitles in the feature space, thereby improving the accuracy of subtitle generation by the model.

[0078] Figure 5 5 is a flow chart of a method 500 for training a subtitle generation model according to another embodiment of the present disclosure, and the method 500 can be used to implement step S302 in the method 300 for training a subtitle generation model. In one embodiment, based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample, the subtitle generation model to be trained is trained by a loss function to obtain a trained subtitle generation model, and S302 further includes:

[0079] S501, converting the sample reason feature into a sample reason representation through a third encoder of the subtitle generation model to be trained;

[0080] S502, converting the sample result feature into a sample result representation through a fourth encoder of the subtitle generation model to be trained;

[0081] S503, connecting the sample cause representation and the sample result representation to obtain a sample joint representation;

[0082] S504, decoding the sample joint representation by a decoder of the subtitle generation model to be trained to obtain sample subtitles of the video;

[0083] S505: Obtain a third loss function value based on the sample subtitle and the reference subtitle, and train the subtitle generation model to be trained based on the third loss function value to obtain the trained subtitle generation model.

[0084] In the embodiment of the present disclosure, in addition to the first encoder and the second encoder, the subtitle generation model may further include: a third encoder for converting the input cause feature into a cause representation; a fourth encoder for converting the input result feature into a result representation; and a decoder for decoding the joint representation of the input cause representation and the result representation into a subtitle. After executing S401 and S402, S501 to S505 may be executed.

[0085] The third encoder and the fourth encoder of the subtitle generation model can adopt a transformer architecture. The input feature can be converted into an output representation by the encoder of the transformer architecture. After the third encoder converts the sample cause feature into a sample cause representation, and the fourth encoder converts the sample result feature into a sample result representation, the sample cause representation and the sample result representation of a certain event or a certain group can be connected to obtain a sample joint representation of the event or the group. The decoder can adopt a sequence-to-sequence model (such as a Uni-VL model) in the transformer architecture. The decoder can predict the next generated word based on the sample joint representation and the previously generated words, thereby generating sample subtitles with a causal temporal relationship. By comparing the sample subtitles with the baseline subtitles of the sample video, the accuracy of the subtitle generation model can be determined. If the accuracy is not high, the first encoder and the second encoder of the subtitle generation model can be optimized, or other encoders or decoders can be optimized. If the accuracy is high, the current subtitle generation model can be used to continue to generate subtitles for the video.

[0086] Based on the input reference subtitles and the sample subtitles output by the subtitle generation model, a third loss function can be constructed. For example, the third loss function can be determined based on the similarity between the reference subtitles and the sample subtitles. The first encoder, the second encoder, the third encoder, the fourth encoder, and the decoder can be jointly trained using the same loss function. For example, the third loss function value is calculated based on the reference subtitles output by the model and the sample subtitles in the sample. In the case where the third loss function value does not meet the convergence condition, the parameters of at least one of the first encoder, the second encoder, the third encoder, the fourth encoder, and the decoder can be adjusted. In the case where the third loss function value meets the convergence condition, the optimization of at least one of the first encoder, the second encoder, the third encoder, the fourth encoder, and the decoder can be stopped to obtain a trained subtitle generation model.

[0087] Through the third encoder, fourth encoder and decoder of the subtitle generation model, the cause feature and the result feature encoding can be connected to generate subtitles with a causal temporal relationship for the video, thereby improving the relevance of the subtitles generated by the model to the video and the causal relevance of the video subtitles.

[0088] Figure 6 FIG. 6 is a flow chart of a method 600 for training a subtitle generation model according to another embodiment of the present disclosure. In one implementation, the method further includes:

[0089] S601: Input the subtitle prompt into a large language model to generate candidate subtitles, which include a cause part and a result part.

[0090] In the disclosed embodiment, the prompt input to the large language model may also be referred to as a prompt word or a prompt word template, etc. The subtitle prompt may guide the large language model to generate multiple candidate subtitles. The cause part and the result part of these candidate subtitles may be used to compare with the cause features and the result features of the video, thereby completing the training of the model. The large prediction model may generate a wealth of candidate subtitles, facilitate the construction of a wealth of training samples, and improve the accuracy of the output results of the model.

[0091] In one embodiment, the subtitle prompt includes one or more of the following:

[0092] Sample subtitles to demonstrate the cause-and-effect narrative structure that the subtitles are expected to include;

[0093] Task description, used to describe the task of generating subtitles containing causal relationships;

[0094] Format requirements, used to specify the format of output subtitles.

[0095] In the disclosed embodiment, a sample subtitle with the correct format and causal narrative structure can be displayed in the subtitle prompt to guide the large language model (LLM) to understand the output requirements. For example, a given prompt sample may include the following causal subtitles:

[0096] Example 1: "The vehicle could not continue to move because it fell into a mud pit."

[0097] Example 2: "The vehicle collided and rolled over."

[0098] Through these small samples, we can show how causal relationships are expressed, so that LLM can follow suit in subsequent generation.

[0099] There can be various format requirements for subtitle prompts, such as displaying the "cause" and "result" fields through JSON objects, or directly presenting the "cause-result" relationship in the subtitles.

[0100] An example of LLM output subtitles can be seen in the following JSON format:

[0101] {

[0102] "Cause":"The car lost control in an open field",

[0103] "Effect":"The car was severely damaged after rolling over"

[0104] }

[0105] “Cause” describes the cause of the event, and “Effect” represents the result of the event.

[0106] Through the example subtitles, task descriptions, and format requirements of the subtitle prompts, the big prediction model can be guided to generate abundant candidate subtitles, which facilitates the construction of abundant training samples and improves the accuracy of the model's output results.

[0107] In one embodiment, Figure 6 As shown, the method also includes:

[0108] S602: Determine a reference subtitle corresponding to the video sample according to a matching result between the candidate subtitle output by the large language model and the video sample, wherein the training sample includes: the video sample and the reference subtitle corresponding to the video sample.

[0109] In the disclosed embodiment, the subtitle generation model can generate many candidate subtitles based on the subtitle prompts. And different subtitle prompts can be constructed according to different scenes to generate rich candidate subtitles. Based on the matching degree between the candidate subtitles and the content of the sample video, the relevance score and the total score of a candidate subtitle can be obtained through a similarity measure (such as embedding similarity). For example, an automatic evaluation method may include: calculating the similarity score between the subtitle and the video embedding, such as EMScore, which can measure the matching degree between the subtitle and the video content in the feature space.

[0110] An example of an automatically evaluated formula is:

[0111] Score = relevance score / total score

[0112] The "relevance score" may be calculated based on the degree of matching between the subtitles and the video content, and the "total score" may be a normalization factor. The higher the score value, the more consistent and relevant the subtitles are to the video content.

[0113] During the generation process, the LLM is iteratively prompted and the generated subtitles are evaluated until the score exceeds a preset threshold, and then the subtitle is retained as part of the baseline subtitle.

[0114] The subtitles generated by LLM will be scored (for example, through threshold screening) to ensure that the subtitles are highly semantically relevant to the video events, thereby obtaining training samples that meet causal requirements.

[0115] In order to make up for the deficiencies of video subtitle generation benchmarks and models in representing the causal temporal narrative of the video, the subtitle generation method of the embodiment of the present disclosure may include a video subtitle generation method based on causal temporal narrative, which includes benchmark subtitle generation and causal effect neural network design. Among them, the benchmark subtitle generation includes: using a large language model (LLM) and a few-sample prompt technology, combined with a pre-designed prompt template, to automatically generate benchmark subtitles containing clear causal and temporal relationships. The generated subtitles are quality screened through an automatic evaluation framework to ensure high consistency and relevance with the video content.

[0116] The method for generating video subtitles based on causal temporal narrative in the disclosed embodiment can be a baseline subtitle generation method based on a large language model (LLM) and a few sample prompts. Specifically, the baseline subtitles of the disclosed embodiment can clearly encode the causal and temporal relationships in the video through the large language model. In the neural network design in the disclosed embodiment, the baseline subtitles are used as key input parameters to guide the network to learn how to capture and express the causal temporal narrative in the video through supervised learning, thereby generating more accurate and contextually relevant video subtitles.

[0117] Large language models can include causal-temporal narrative (CTN) models, where temporal order can be implicitly represented by the order of causal relationships. Specifically, the generated subtitle sequence reflects the causal relationship of video events and their temporal order. This temporal relationship is reflected in the prompt design and the encoding-decoding structure of the causal neural network, because the model generates subtitles according to the order of events in the video, thereby capturing the temporal characteristics.

[0118] 1. The benchmark subtitle generation method mainly includes the following parts:

[0119] 1. Prompt Design: Through the designed prompts, LLM is guided to generate subtitles containing causal relationships. These prompts can include a small number of sample subtitles, showing the expected output structure and key elements. The designed prompts are input into LLM, and LLM generates baseline subtitles containing causal relationships based on the input descriptive subtitles and prompts.

[0120] Prompt design is key to producing high-quality subtitles. Prompts can explicitly include one or more of the following elements:

[0121] Example subtitles: Provide several high-quality example subtitles that demonstrate the desired cause-and-effect narrative structure.

[0122] Task description: Clearly state that the task of LLM is to generate captions that contain causal relationships.

[0123] Format requirements: Specifies the format of the output subtitles, such as a JSON object, containing "reason" and "result" fields.

[0124] The format requirements of the prompt may include the expected output structure, such as the format and organization of the subtitles. For example, the "cause" and "result" fields may be displayed through JSON objects, or the "cause-result" relationship may be directly presented in the subtitles. Key elements include the expression of causal relationships, temporal consistency, and simplicity in the subtitles. For example, the output subtitles may be in the following JSON format:

[0125] {

[0126] "Cause":"The car lost control in an open field",

[0127] "Effect":"The car was severely damaged after rolling over"

[0128] }

[0129] “Cause” describes the cause of the event, and “Effect” represents the result of the event.

[0130] 2. LLM generates subtitles

[0131] LLM generates reference subtitles based on the input descriptive subtitles and prompts. In the disclosed embodiment, LLM is used for subtitle generation. LLM uses its natural language understanding and generation capabilities, combined with few-sample learning techniques, to generate subtitles that meet the requirements. Few-sample learning here means that the model only needs a small number of example prompts to generate subtitles with causal relationships. These samples can be several example subtitles that show the correct format and structure, which are used to guide LLM to understand the output requirements. For example, a given prompt sample can include the following causal subtitles:

[0132] Example 1: "Because the student's alarm clock did not ring, he missed the school bus."

[0133] Example 2: "She didn't use moisturizer after bathing, which caused dry skin."

[0134] These small samples demonstrate how causal relationships are expressed, allowing LLM to emulate them in subsequent generation.

[0135] 3. Automatic evaluation: Using automatic evaluation indicators, the generated subtitles can maintain consistency and relevance with the video content, and subtitles with scores above the threshold can be retained.

[0136] In order to ensure that the generated subtitles are consistent and relevant to the video content, a self-assessment method is used. An example of a calculation formula (simplified version) includes: Score = relevance score / total score

[0137] Among them, the "relevance score" is calculated based on the degree of match between the subtitles and the video content. The relevance score and the total score are usually based on the match between the subtitles and the video content, obtained through some similarity measurement (such as embedding similarity). For example, an automatic evaluation method includes: calculating the similarity score between the subtitles and the video embedding, such as EMScore, which can measure the match between the subtitles and the video content in the feature space. The subtitles generated by LLM are automatically scored (for example, through threshold screening) to ensure that the subtitles are highly semantically relevant to the video events. The "total score" is a normalization factor. The higher the score value, the more consistent and relevant the subtitles are to the video content.

[0138] During the generation process, the LLM is iteratively prompted and the generated subtitles are evaluated until the score exceeds a preset threshold, and then the subtitle is retained as part of the baseline subtitle.

[0139] 2. Neural network design mainly includes the following contents:

[0140] The disclosed embodiment proposes a neural network specifically for learning causal temporal narrative in video content (referred to as causal neural network). The design of the network includes two stages: the first stage learns the cause and effect features in the video respectively through two independent encoders; the second stage combines these features and generates the final video subtitles through a decoder. For example, in a causal neural network, a "cause" encoder and a "result" encoder can be included, each processing the cause and result parts in the video. In general, these two encoders can be independent variants of the contrastive language-image pretraining-vision transformer (CLIP-ViT) visual model, which will process the cause and result features of the video frame respectively. Through contrastive loss optimization (the loss optimization can be calculated using contrastive learning methods), the cause and result encoders are able to learn and approach their corresponding text description embeddings in the feature space.

[0141] Stage 1: Causal Visual Encoding

[0142] In the first stage, the neural network extracts the cause and effect features in the video through two independent video encoders (e.g., visual encoders). The two encoders are pre-trained on the cause and effect parts of the benchmark subtitles. The two encoders are optimized by contrastive loss functions to ensure that the video embedding is close to the corresponding text embedding in the feature space.

[0143] For example, cause extraction and result features can be extracted based on video frame images, and visual events in the video can be captured by a visual encoder. The encoder can learn how to identify causes and results through pre-training. The training samples need to be labeled with cause video frames, result video frames, etc., so that the network can perform targeted learning. The loss function can be a contrast loss (such as cosine (cos) similarity calculated using CLIP), which is used to optimize the similarity of causes and results embedded in the feature space.

[0144] Stage 2: Subtitle Generation

[0145] In the second stage, the neural network can use the sequence-to-sequence Transformer architecture to generate the final subtitles. This stage can include two independent encoders, namely the cause encoder (Encoder cause, Enc_cause) and the result encoder (Encoder effect, Enc_effect). The cause encoder and the result encoder are used to process the cause and effect video features extracted in the first stage, respectively, and a decoder (Decoder, Dec). The decoder can be used to generate subtitles.

[0146] Cause encoder: Transform the cause video feature T cause Encoded as reason representation h cause

[0147] Result encoder: Transform the result video feature T effect The result is encoded as h effect

[0148] The outputs of these two encoders are concatenated to form a single representation h concat =[h cause ;h effect ], and then input to the decoder. The decoder generates a word at each time step based on the previous generated word and the current representation h concat To predict the next word

[0149] In the subtitle generation stage, two encoders (Enc_cause and Enc_effect) process the cause and effect features extracted in the first stage respectively, and concatenate the outputs of the two encoders into a joint representation. The decoder outputs each word step by step according to the causal relationship when generating subtitles. The decoder uses the previously generated vocabulary and the current joint representation to gradually predict the next word of the subtitle, ensuring the temporal consistency of the subtitle content and the video events.

[0150] For example, the decoder can adopt the sequence-to-sequence model (Uni-VL model) in the Transformer architecture. This type of model can handle the generation of long texts well and is suitable for video subtitle generation tasks. The decoder can partially initialize the weights of the pre-trained Uni-VL model, making the decoder more efficient in processing visual and language information. This architecture allows the decoder to take into account the context before and after the subtitles when generating each word, ensuring the coherence and causality of the subtitles. The design of the decoder relies on the self-attention mechanism of the Transformer, which can capture the long-range dependencies in the input sequence (cause and effect features) and generate an output sequence that conforms to causal logic. The decoder is based on the causal features (h concat ) and the words generated in the previous step, gradually predicting and generating the next word, thereby generating subtitles that conform to the causal narrative.

[0151] Figure 7 FIG. 1 is a schematic diagram of a causal narrative subtitle generation model according to an embodiment of the present disclosure. Figure 7 As shown, the model can be roughly divided into two main parts:

[0152] (1) Find out the causes and consequences

[0153] The model first decomposes the video content into two parts: "cause" and "result", and puts them into two specially designed visual cause encoders and visual result encoders respectively. For example, in a video, the model will recognize that "the car is driving fast on the mud" is the cause, and "the car is seriously damaged after rolling over" is the result.

[0154] One of the two encoders focuses on finding the factors that led to the event (the cause), while the other focuses on recording the results (the effect) caused by these factors.

[0155] (2) Generate subtitles with causal relationships

[0156] After obtaining the cause feature and the result feature, the model inputs the two features into the cause text encoding converter (cause encoder) and the result text encoding converter (result encoder) respectively to obtain the cause representation and the result representation. The cause representation and the result representation are then merged to obtain the joint representation, which is then input into the decoder. The decoder will gradually generate subtitles that describe the entire video, such as "the car lost control and overturned, causing serious damage", which forms a clear causal narrative.

[0157] Using this method, the model can not only describe the events in the video, but also tell us the causes and consequences of the events, increasing the coherence and comprehension of the subtitles.

[0158] By first separating the cause and result features of video events, and then using these features to generate causally coherent subtitles, the video description is made more consistent with the audience's understanding logic. Video subtitles can not only explain "what happened", but also explain "why it happened".

[0159] The neural network effectively captures the causal and temporal narratives in videos through a two-stage architecture. The first stage learns cause and effect features through a specialized video encoder, and the second stage uses these features to generate video captions with causal temporal narratives.

[0160] Figure 8 8 is a schematic diagram of a subtitle generating device 800 according to an embodiment of the present disclosure. In one implementation, the device includes:

[0161] Extraction module 801, used to extract cause features and result features from the video;

[0162] The generating module 802 is used to generate subtitles of the video based on the cause feature and the result feature.

[0163] Fig. 91 is a schematic diagram of the structure of a subtitle generating device 900 according to another embodiment of the present disclosure. The device 900 includes: an extraction module 901 and a generation module 902. The functions of the above modules can refer to the functions of the modules of the subtitle generating device in the above embodiment. In one embodiment, the extraction module 901 includes:

[0164] A first extraction submodule 9011 is used to extract the reason feature from the video through a first encoder of a subtitle generation model;

[0165] The second extraction submodule 9012 is used to extract the result feature from the video through the second encoder of the subtitle generation model.

[0166] In one embodiment, Fig. 9 As shown, the generating module 902 includes:

[0167] A first conversion submodule 9021, configured to convert the reason feature into a reason representation through a third encoder of the subtitle generation model;

[0168] A second conversion submodule 9022, configured to convert the result feature into a result representation through a fourth encoder of the subtitle generation model;

[0169] A connection submodule 9023, used for connecting the cause representation and the result representation to obtain a joint representation;

[0170] The decoding submodule 9024 is used to decode the joint representation through the decoder of the subtitle generation model to obtain the subtitles of the video.

[0171] In one implementation, the training samples of the subtitle generation model include video samples and reference subtitles, where the reference subtitles are generated based on a large language model.

[0172] Fig.10 1 is a schematic diagram of a structure of a training device 1000 for a subtitle generation model according to an embodiment of the present disclosure. In one implementation, the device includes:

[0173] An input module 1001 is used to input a video sample of a training sample into a subtitle generation model to be trained, and obtain output sample cause features and sample result features respectively;

[0174] The training module 1002 is used to train the subtitle generation model to be trained based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample through a loss function to obtain a trained subtitle generation model.

[0175] Fig.111 is a schematic diagram of the structure of a training device 1100 for a subtitle generation model according to an embodiment of the present disclosure. The device may include: an input module 1101 and a training module 1102. The functions of the above modules can refer to the functions of the modules of the training device for the subtitle generation model in the above embodiment. In one embodiment, the video sample includes a cause annotation of a video frame and a result annotation of the video frame. The input module 1101 includes:

[0176] The first input submodule 11011 is used to input the video sample into the first encoder of the subtitle generation model to be trained to obtain the output reason feature of the sample;

[0177] The second input submodule 11012 is used to input the video sample into the second encoder of the subtitle generation model to be trained to obtain the output result features of the sample.

[0178] In one embodiment, Fig.11 As shown, the training module 1102 includes at least one of the following:

[0179] A first training submodule 11021 is used to obtain a first loss function value based on the sample reason feature and the reference subtitle reason feature, and train the first encoder based on the first loss function value to obtain a trained first encoder;

[0180] The second training submodule 11022 is used to obtain a second loss function value based on the sample result feature and the result feature of the reference subtitle, and train the second encoder based on the second loss function value to obtain a trained second encoder.

[0181] In one embodiment, Fig.11 As shown, the training module 1102 also includes:

[0182] A first conversion submodule 11023, configured to convert the sample reason feature into a sample reason representation through a third encoder of the subtitle generation model to be trained;

[0183] A second conversion submodule 11024, configured to convert the sample result feature into a sample result representation through a fourth encoder of the subtitle generation model to be trained;

[0184] A connection submodule 11025, used to connect the sample cause representation and the sample result representation to obtain a sample joint representation;

[0185] A decoding submodule 11026 is used to decode the sample joint representation through a decoder of the subtitle generation model to be trained to obtain sample subtitles of the video;

[0186] The third training submodule 11027 is used to obtain a third loss function value based on the sample subtitle and the reference subtitle, and train the subtitle generation model to be trained based on the third loss function value to obtain the trained subtitle generation model.

[0187] In one embodiment, Fig.11 As shown, the device also includes:

[0188] The generating module 1103 is used to input the subtitle prompt into the large language model to generate candidate subtitles, and the candidate subtitles include a cause part and a result part.

[0189] In one embodiment, the subtitle prompt includes one or more of the following:

[0190] Sample subtitles to demonstrate the cause-and-effect narrative structure that the subtitles are expected to include;

[0191] Task description, used to describe the task of generating subtitles containing causal relationships;

[0192] Format requirements, used to specify the format of output subtitles.

[0193] In one embodiment, Fig.11 As shown, the device also includes:

[0194] The matching module 1104 is used to determine the reference subtitles corresponding to the video sample according to the matching results between the candidate subtitles output by the large language model and the video sample, and the training sample includes: the video sample and the reference subtitles corresponding to the video sample.

[0195] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0196] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0197] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0198] Fig.12A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0199] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0200] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0201] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as a subtitle generation method and / or a training method for a subtitle generation model. For example, in some embodiments, the subtitle generation method and / or the training method for a subtitle generation model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the subtitle generation method and / or the training method for the subtitle generation model described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the subtitle generation method and / or the subtitle generation model training method in any other appropriate manner (for example, by means of firmware).

[0202] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0203] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0204] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0206] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0207] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0208] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0209] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A subtitle generation method, comprising: Extract cause features and effect features from videos; Based on the cause feature and the result feature, subtitles for the video are generated.

2. The method according to claim 1, wherein: The extracting of cause features and result features from the video includes: Extracting the reason feature from the video through a first encoder of a caption generation model; The result features are extracted from the video by a second encoder of the caption generation model.

3. The method according to claim 2, wherein: The step of generating subtitles for the video based on the cause feature and the result feature includes: Converting the reason feature into a reason representation through a third encoder of the caption generation model; Converting the result features into a result representation through a fourth encoder of the caption generation model; Connecting the cause representation and the result representation to obtain a joint representation; The joint representation is decoded by a decoder of the subtitle generation model to obtain subtitles of the video.

4. The method according to any one of claims 1 to 3, wherein: The training samples of the subtitle generation model include video samples and reference subtitles, and the reference subtitles are generated based on a large language model.

5. A training method for a subtitle generation model, comprising: Input the video samples of the training samples into the subtitle generation model to be trained, and obtain the output sample cause features and sample result features respectively; Based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample, the subtitle generation model to be trained is trained through a loss function to obtain a trained subtitle generation model.

6. The method according to claim 5, wherein: The video sample includes a cause annotation of the video frame and a result annotation of the video frame; The step of inputting the video sample of the training sample into the subtitle generation model to be trained, and obtaining the output sample cause feature and sample result feature respectively, comprises: Inputting the video sample into the first encoder of the subtitle generation model to be trained to obtain the output sample reason feature; The video sample is input into the second encoder of the subtitle generation model to be trained to obtain the output sample result feature.

7. The method according to claim 6, wherein: The subtitle generation model is trained based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample by using a loss function to obtain a trained subtitle generation model, including at least one of the following: Obtaining a first loss function value based on the sample reason feature and the reference subtitle reason feature, and training the first encoder based on the first loss function value to obtain a trained first encoder; A second loss function value is obtained based on the sample result feature and the result feature of the reference subtitle, and the second encoder is trained based on the second loss function value to obtain a trained second encoder.

8. The method according to any one of claims 5 to 7, wherein: The method further comprises: training the subtitle generation model to be trained based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample by using a loss function to obtain a trained subtitle generation model; Converting the sample reason feature into a sample reason representation by a third encoder of the subtitle generation model to be trained; Converting the sample result feature into a sample result representation by a fourth encoder of the subtitle generation model to be trained; Connecting the sample cause representation and the sample result representation to obtain a sample joint representation; Decoding the sample joint representation by a decoder of the subtitle generation model to be trained to obtain sample subtitles of the video; A third loss function value is obtained based on the sample subtitles and the reference subtitles, and the subtitle generation model to be trained is trained based on the third loss function value to obtain the trained subtitle generation model.

9. The method according to any one of claims 5 to 8, further comprising: The subtitle prompt is input into a large language model to generate candidate subtitles, wherein the candidate subtitles include a cause part and a result part.

10. The method according to claim 9, wherein: The subtitle prompt includes one or more of the following: sample subtitles, task description, and format requirements.

11. The method according to claim 9 or 10, further comprising: According to the matching result between the candidate subtitles output by the large language model and the video sample, a reference subtitle corresponding to the video sample is determined, and the training sample includes: the video sample and the reference subtitle corresponding to the video sample.

12. A subtitle generating device, comprising: An extraction module, used to extract cause features and result features from the video; A generating module is used to generate subtitles of the video based on the cause feature and the result feature.

13. The device according to claim 12, wherein: The extraction module comprises: A first extraction submodule, configured to extract the reason feature from the video through a first encoder of a subtitle generation model; The second extraction submodule is used to extract the result features from the video through a second encoder of the subtitle generation model.

14. The device according to claim 13, wherein: The generating module comprises: A first conversion submodule, configured to convert the reason feature into a reason representation through a third encoder of the subtitle generation model; A second conversion submodule, configured to convert the result feature into a result representation through a fourth encoder of the subtitle generation model; A connection submodule, used for connecting the cause representation and the result representation to obtain a joint representation; A decoding submodule is used to decode the joint representation through a decoder of the subtitle generation model to obtain subtitles of the video.

15. The device according to any one of claims 12 to 14, wherein: The training samples of the subtitle generation model include video samples and reference subtitles, and the reference subtitles are generated based on a large language model.

16. A training device for a subtitle generation model, comprising: An input module, used to input the video sample of the training sample into the subtitle generation model to be trained, and obtain output sample cause features and sample result features respectively; The training module is used to train the subtitle generation model to be trained based on the sample cause feature, the sample result feature, and the reference subtitle of the training sample through a loss function to obtain a trained subtitle generation model.

17. The device according to claim 16, wherein: The video sample includes a cause annotation of the video frame and a result annotation of the video frame; The input module comprises: A first input submodule, used for inputting the video sample into a first encoder of the subtitle generation model to be trained, to obtain the output sample cause feature; The second input submodule is used to input the video sample into the second encoder of the subtitle generation model to be trained to obtain the output sample result feature.

18. The device according to claim 17, wherein: The training module includes at least one of the following: A first training submodule, configured to obtain a first loss function value based on the sample reason feature and the reference subtitle reason feature, and train the first encoder based on the first loss function value to obtain a trained first encoder; The second training submodule is used to obtain a second loss function value based on the sample result feature and the result feature of the reference subtitle, and train the second encoder based on the second loss function value to obtain a trained second encoder.

19. The device according to any one of claims 16 to 18, wherein: The training module also includes: A first conversion submodule, configured to convert the sample reason feature into a sample reason representation through a third encoder of the subtitle generation model to be trained; A second conversion submodule, configured to convert the sample result feature into a sample result representation through a fourth encoder of the subtitle generation model to be trained; A connection submodule, used for connecting the sample cause representation and the sample result representation to obtain a sample joint representation; A decoding submodule, configured to decode the sample joint representation through a decoder of the subtitle generation model to be trained to obtain sample subtitles of the video; The third training submodule is used to obtain a third loss function value based on the sample subtitles and the reference subtitles, and train the subtitle generation model to be trained based on the third loss function value to obtain the trained subtitle generation model.

20. The device according to any one of claims 16 to 19, further comprising: The generation module is used to input the subtitle prompt into the large language model to generate candidate subtitles, wherein the candidate subtitles include a cause part and a result part.

21. The device according to claim 20, wherein: The subtitle prompt includes one or more of the following: sample subtitles, task description, and format requirements.

22. The device according to claim 20 or 21, further comprising: A matching module is used to determine a reference subtitle corresponding to the video sample according to a matching result between the candidate subtitle output by the large language model and the video sample, wherein the training sample includes: the video sample and the reference subtitle corresponding to the video sample.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images