Video subtitle generation system and method
By using cross-modal encoder and subtitle decoder to handle video and audio embeddings in video subtitle generation, the problem of existing models ignoring audio modality is solved, improving the quality and accessibility of video subtitles.
Patent Information
- Application Number
- CN202410555611.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-08
AI Technical Summary
Existing deep learning models ignore the audio mode when generating video subtitles, resulting in the lack of important information such as acoustic effects and speaker identity, reducing the accessibility and viewing experience of the video.
By extracting video embeddings and audio embeddings in video frames, input them to the cross-modal encoder for multi-modal embedding generation, and combining with the subtitle decoder to generate video subtitles, ensuring that the model can capture and describe audio elements.
Improves accessibility and overall viewing experience of video subtitles, improving the quality of subtitles by effectively capturing and describing audio elements in the video, such as acoustic effects and speaker identity.
Smart Images

Figure CN118921538A_ABST
Abstract
Description
Background Art
[0001] Large-scale pre-training plays a key role in improving the performance of modern deep learning models for complex vision and language tasks. One of the most challenging vision and language tasks for deep learning models is video captioning, which consists of narration that describes key elements in a video, such as the background, characters, facial expressions, audio elements, and other details that are important for understanding the video. This narration allows more viewers, including those who are blind or visually impaired, to follow the video and understand what is happening on the screen.
[0002] According to one technical approach, a large video dataset consisting of videos annotated with ground-truth text labels can potentially be generated and used to pre-train a deep learning model using supervised learning, such that the trained model will be able to output descriptive narration of an inference-time input video that describes elements in the video. However, manually annotating video datasets for captioning is expensive and not scalable. As a result, existing video captioning datasets are often limited in size. To address this challenge, deep learning models have recently been developed and trained using datasets collected from instructional videos, where transcripts generated by automatic speech recognition are used as text supervision. This establishes a new trend of pre-training large-scale video datasets with text transcripts for video captioning.
[0003] However, these training methods do not train deep learning models to describe audio elements that are not included in the text transcript. As a result, existing deep learning models may generate video description subtitles that miss important information or context conveyed by these audio elements, which may include, for example, acoustic effects, music, and speaker identity. This may significantly reduce the accessibility of the video and the overall viewing experience. Summary of the invention
[0004] In view of the above situation, a computing system for video subtitle generation is provided. The system includes a processor and a memory of a computing device. The processor is configured to use a portion of the memory to execute a program to receive an input video, extract a video frame from the input video, extract a video embedding and an audio embedding including a local video word unit and a local audio word unit from the video frame, respectively, input the local video word unit and the local audio word unit into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding, and generate a video subtitle based on the multimodal embedding using a subtitle decoder.
[0005] This summary is provided to introduce a selection of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 A schematic diagram of a training computing system and an inference computing system according to an example of the present disclosure is illustrated.
[0007] Figure 2 Picture shows Figure 1 Detailed schematic diagram of the inference computing system.
[0008] Figure 3 Picture shows Figure 1 Detailed schematic diagram of the caption generation module of the inference computing system.
[0009] Figure 4 Picture shows Figure 3 Detailed schematic diagram of the merging and fusion module of the cross-modal encoder.
[0010] Figure 5 Picture shows Figure 3 Detailed schematic diagram of the global cross-fusion module of the cross-modal encoder.
[0011] Figure 6 Picture shows Figure 1 Detailed diagram of the model training procedure of the training computing system.
[0012] Fig. 7A and Figure 7B is an example diagram of an application client showing an input video and generated video subtitle words corresponding to the input video, as shown by Figure 1 The inference computing system is executed.
[0013] Figure 8 is a flowchart of a method for generating video subtitles according to an exemplary embodiment of the present disclosure.
[0014] Fig. 9 is a flowchart of a method for training a cross-modal encoder and subtitle decoder using data and model distillation according to an example embodiment of the present disclosure.
[0015] Fig.10 is a table from an ablation study showing changes in Figure 1 and Figure 2 The impact of the method of processing audio and video tokens in the subtitle generation module on the performance of video subtitle generation.
[0016] Fig.11 An example computing environment for the present disclosure is shown. DETAILED DESCRIPTION
[0017] The popularity of portable devices with cameras has greatly facilitated the creation and broadcasting of online videos, which has increased the demand for tasks including systems that receive video and text input to generate automatic video captions (including video descriptions or descriptive voice-over subtitles). However, most current automatic video captioning methods ignore the audio modality, which deprives the generated video captions of additional information, including acoustic events and speaker identity. In view of these limitations, the present disclosure describes various methods and systems for video caption generation that can address the above-mentioned problems. Such methods and systems are described in more detail below.
[0018] See also Figure 1 , schematically depicts the process of generating subtitles for video frames of an input video 118 using a video subtitle generation process, from a training step to an inference step. Initially, a training computing system 100 executes a data distillation and model distillation module 102, the model distillation module 102 including a model trainer 104 configured to train a video encoder 106, an audio encoder 108, a cross-modal encoder 110, and a subtitle decoder 112 using training data. Then, the video encoder 106, the audio encoder 108, the cross-modal encoder 110, and the subtitle decoder 112 trained by the model trainer 104 are installed on the inference computing system 114 and used by the subtitle generation module 116 to receive and process the input video 118 to generate video subtitles 120 corresponding to the input video 118, as explained in further detail below.
[0019] See also Figure 2, an inference computing system 114 for generating video subtitles 120 corresponding to an input video 118 using a video subtitle generation process is provided. The inference computing system 114 includes a computing device 200, the computing device 200 includes a processor 202, an input / output module 204, a volatile memory 206, and a non-volatile memory 208 storing a subtitle generation program 210, the subtitle generation program 210 includes a video encoder 106, an audio encoder 108, a cross-modal encoder 110, and a subtitle decoder 112. A bus 212 can operably couple the processor 202, the input / output module 204, and the volatile memory 206 to the non-volatile memory 208. The inference computing system 114 is operably coupled to a client computing device 214 via a network 224. In some examples, the network 224 can take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and can include the Internet. Although subtitle generation program 210 is depicted as being hosted at one computing device 200 , it should be appreciated that subtitle generation program 210 may alternatively be hosted across multiple computing devices, to which computing device 200 may be communicatively coupled via a network, including network 224 .
[0020] The processor 202 is configured to store the subtitle generation program 210 in a non-volatile memory 208, which retains the data stored by the instructions even without externally applied power, such as flash memory, hard disk, read-only memory (ROM), electrically erasable programmable memory (EEPROM), etc. The instructions include one or more programs (including the subtitle generation program 210) and data used by these programs sufficient to perform the operations described herein. In response to execution by the processor 202, the instructions cause the processor 202 to execute the subtitle generation program 210, which includes the video encoder 106, the audio encoder 108, the cross-modal encoder 110, and the subtitle decoder 112.
[0021] Processor 202 is a microprocessor that includes one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a system on a chip (SOC), a field programmable gate array (FPGA), a logic circuit, or other suitable type of microprocessor configured to perform the functions described herein. Volatile memory 206 may include physical devices such as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), etc., which only temporarily store data when power is applied during program execution. Non-volatile memory 208 may include removable and / or built-in physical devices such as optical storage (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.) and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technology.
[0022] In one example, a user operating a client computing device 214 may send an input video 118 to the computing device 200. The processor 202 of the computing device 200 is configured to receive the input video 118 from the user and execute the caption generation program 210 to generate video captions 120 corresponding to the input video 118. The processor 202 then returns the video captions 120 to the client computing device 214.
[0023] The client computing device 214 may execute the application client 216 to send the input video 118 to the computing device 200 upon detecting the user input 218 and subsequently receive the video captions 120 from the computing device 200. The application client 216 may be coupled to the graphical user interface 220 of the client computing device 214 to display the graphical output 222 of the video captions 120.
[0024] Although not depicted here, it should be appreciated that the computing device 200 may be configured similarly to execute Figure 1 The training computing system 100 of the data distillation and model distillation module 102 is shown.
[0025] See also Figure 3 , describes in detail Figure 1 and Figure 2Operation of the subtitle generation module 116. An input video 118 is received. A video frame extractor 300 extracts a video frame 302 from the input video 118. A video embedding sequence 304 is extracted from the video frame 302 using a video encoder 106 (which may be a video transformer). However, it should be appreciated that the video encoder 106 may be configured using an alternative deep learning model architecture, including a convolutional neural network. The extracted video embedding 304 includes feature dimensions 304a, local video tokens 304b, and global video tokens 304c. The global video token 304c is a global representation of the video frame 302 of the input video 118 that captures high-level concepts, such as people gathering on a street.
[0026] An audio embedding sequence 306 is extracted from an associated audio spectrogram of a video frame 302 using an audio encoder 108 (which may be an audio transformer). However, it should be appreciated that the audio encoder 108 may be configured using an alternative deep learning model architecture, including a convolutional neural network. The audio embedding 306 corresponding to the video embedding 304 may alternatively be extracted from a file separate from the input video 118. The extracted audio embedding 306 includes feature dimensions 306a, local audio tokens 306b, and global audio tokens 306c. The global audio token 306c is a global representation of the audio of the input video 118 that captures high-level concepts, such as laughter.
[0027] The video embedding 304 and the audio embedding 306 are input into the cross-modal encoder 110 to generate a multi-modal embedding 308. The cross-modal encoder 110 includes a transformer layer that implements multi-head attention (MHA) and a feed-forward block (FFB). Figure 3 In the example of , the cross-modal encoder 110 includes a merge fusion module 110a for performing merge fusion and a global cross fusion module 110b for performing global cross fusion, respectively. The number of transformer layers in the cross-modal encoder 110 is not particularly limited, and may be three or more.
[0028] Return to Figure 4, which describes the operation of the merging and fusion module 110a of the cross-modal encoder 110. The splicer 110aa of the merging and fusion module 110a splices the local video word 304b and the local audio word 306b together, and then inputs the spliced video word and audio word into the transformer block of the MHA transformer 110ab of the merging and fusion module 110a. In the transformer block, each transformer layer has two modality-specific transformers, and different modalities exchange information via cross attention. In a given transformer layer having a first transformer and a second transformer, the first transformer has a key and value obtained from the local video word 304b, and the query is obtained from the local audio word 306b. In contrast, the second transformer has a key and value obtained from the local audio word 306b, and the query is obtained from the local video word 304b. The merging and fusion module 110a then outputs a merged local video word 308a and a merged local audio word 308b.
[0029] Return to Figure 5 , which describes the operation of the global cross-fusion module 110b of the cross-modal encoder 110. The global cross-fusion module 110b includes a video MHA transformer 110ba and an audio MHA transformer 110bb. The video MHA transformer 110ba receives the local video word-element 304b and the global video word-element 304c as a query, the local video word-element 304b as a key, and the global audio word-element 306c as a value, and then outputs a fused local video word-element 308c and a fused global video word-element 308e. The audio MHA transformer 110bb receives the local audio word-element 306b and the global audio word-element 306c as a query, the local audio word-element 306b as a key, and the global video word-element 304c as a value, and then outputs a fused local audio word-element 308d and a fused global audio word-element 308f. Therefore, using the global cross-fusion module 110b, the cross-modal attention stream is aggregated into the global video word-meta 304c and the global audio word-meta 306c, and the local video word-meta 304b and the local audio word-meta 306b are used for intra-modal attention. In contrast to using the local video word-meta 304b as context, by using the global video word-meta 304c as context, high-level concepts can be used in the generation of video captions.
[0030] Return to Figure 3The merged local video word 308a and the merged local audio word 308b output by the merging and fusion module 110a, and the fused local video word 308c and the fused local audio word 308d output by the global cross-fusion module 110b are input to the average calculation function 110c, and the average calculation function 110c averages the merged local video word 308a and the fused local video word 308c to output the averaged local video word 308g, and averages the merged local audio word 308b and the fused local audio word 308d to output the averaged local audio word 308h. These averaged local word-grams 308g, 308h can be iteratively input into subsequent fusion layers (indicated by "layer i+1") of the cross-modal encoder 110 until an output final multimodal embedding 308 is output, which includes averaged local video word-grams 308g, averaged local audio word-grams 308h, fused global video word-grams 308e, fused global audio word-grams 308f, and feature dimensions 308i. Thus, local features captured by local word-grams 304b, 304c may include words in speech or objects in video frames, and global features captured by global word-grams 306b, 306c may include laughter or people gathering on a street, all of which can be captured by combining merge fusion and global cross fusion in a cross-modal encoder 110.
[0031] The subtitle decoder 112 receives an input of a multimodal embedding 308 to autoregressively generate and output a video subtitle word 120 based on the multimodal embedding 308. Subtitle generation may be initiated by inputting one or more different beginning of sentences (BOS) words 310 into the subtitle decoder 112. The BOS words 310 may include a first BOS word 310a that initiates a current prediction of a video subtitle word 120a and a second BOS word 310b that initiates a prediction of a next video subtitle word 120b. The subtitle decoder 112 may be implemented as a transformer. The number of transformer layers in the subtitle decoder 112 is not particularly limited and may be three or more.
[0032] See also Figure 6 , which details a method for training the cross-modal encoder 110 and the subtitle decoder 112 using training data Figure 1The data distillation and model distillation module 102 and the operation of the model training program 104 are similar to the subtitle generation module 116. Since the features of the model training program 104 are similar to the subtitle generation module 116, including a video extractor 400 configured to extract video frames 402, a video encoder 106 configured to output video embeddings 404, an audio encoder 108 configured to output audio embeddings 406, a cross-modal encoder 110 configured to output multimodal embeddings 408, and a subtitle decoder 112 configured to output video subtitle tokens 120 based on the multimodal embeddings 408 and BOS tokens 410, a detailed description thereof is omitted here. Similar parts in this example are numbered similarly to the subtitle generation module 116 and share their functions, and for the sake of brevity, will not be described again except for the following.
[0033] The video subtitle token 120 and the true subtitle 412 are input into a cross entropy loss function 414, which calculates a pure audio decoder loss 416, a pure video decoder loss 418, and a multimodal decoder loss 420 based on the video subtitle token 120 and the true subtitle 412. The pure audio decoder loss 416 is calculated by inputting the video embedding 404 set to all zeros and the audio embedding 406 of the video frame 402 into the cross modal encoder 110 to generate a pure audio unimodal embedding 408, inputting the pure audio unimodal embedding 408 into the subtitle decoder 112 to generate the video subtitle token 120, and then using the cross entropy loss function 414 to calculate the pure audio decoder loss 416 compared with the true subtitle 412. The pure audio decoder loss 416 (L a ) can be defined by the following formula:
[0034]
[0035] Here, 0 represents the video embedding 404 which is set to all zeros, γ a Indicates pure audio embedding 406, f θ represents a cross-modal encoder 110, g θ′ represents the subtitle decoder 112 , y represents the true value subtitle 412 , and L represents the cross entropy loss function 414 .
[0036] The video-only decoder loss 418 is calculated by inputting the audio embedding 406 set to all zeros and the video embedding 404 of the video frame 402 into the cross-modal encoder 110 to generate the video-only unimodal embedding 408, inputting the video-only unimodal embedding 408 into the subtitle decoder 112 to generate the video subtitle token 120, and then using the cross-entropy loss function 414 to calculate the video-only decoder loss 418 compared to the ground-truth subtitle 412. The decoder loss 418 (L v) can be defined by the following formula:
[0037]
[0038] Here, 0 represents the audio embedding 406 set to all zeros, γ v Indicates pure video embed 404, f θ represents a cross-modal encoder 110, g θ′ represents the subtitle decoder 112 , y represents the true value subtitle 412 , and L represents the cross entropy loss function 414 .
[0039] The multimodal decoder loss 420 is calculated by inputting the audio embedding 406 and the video embedding 404 of the video frame 402 into the cross-modal encoder 110 to generate the multimodal embedding 408, inputting the multimodal embedding 408 into the subtitle decoder 112 to generate the video subtitle token 120, and then using the cross entropy loss function 414 to calculate the multimodal decoder loss 420 compared with the ground-truth subtitle 412. The multimodal decoder loss 420 (L) can be defined by the following formula:
[0040]
[0041] Here, γ a Represents audio embedding 406, γ v Indicates pure video embed 404, f θ represents a cross-modal encoder 110, g θ′ represents the subtitle decoder 112 , y represents the true value subtitle 412 , and L represents the cross entropy loss function 414 .
[0042] The weight updater 422 may calculate an audio difference index to measure the difference between the pure audio decoder loss 416 and the multimodal decoder loss 420. Similarly, the weight updater 422 also calculates a video difference index to measure the difference between the pure video decoder loss 418 and the multimodal decoder loss 420. The calculated audio difference index and video difference index are used to update the training weights of the pure audio decoder loss 416 and the pure video decoder loss 418 in iterations. The audio difference index G a and video difference index G v It can be respectively expressed by formula G a =(L a –L) 2 and G v =(L v –L) 2 To define, where L a Indicates that the pure audio decoder loses 416, L v represents a pure video decoder loss 418, while L represents a multimodal decoder loss 420.
[0043] The weight updater 422 uses the calculated cross entropy losses 416, 418, 420 and the difference index to update the trainable parameters of the cross-modal encoder 110 and the subtitle decoder 112 through gradient back propagation. When the video difference index is high, a higher weight is assigned to the video modality of the cross-modal encoder 110 and the subtitle decoder 112, so that more attention is given to the video modality and the over-specialization of the audio modality is reduced. When the audio difference index is high, a higher weight is assigned to the audio modality of the cross-modal encoder 110 and the subtitle decoder 112, so that more attention is given to the audio modality and the over-specialization of the video modality is reduced.
[0044] As the optimization progresses in iterations and the video difference index and the audio difference index change over time, the weights assigned to the video modality and the audio modality are dynamically adjusted to increase the weight of the underfitting modality. The process of calculating the losses and parameters of the cross-modal encoder 110 and the subtitle decoder 112 is repeated for each stage so that the learning acquired in the previous stage can be inherited and trained. Therefore, the cross-modal encoder 110 and the subtitle decoder 112 are trained according to the weights learned in the previous training stage.
[0045] Pure audio decoder loss 416L a The weights and pure video decoder loss 418L v The weights can be based on whether the mode is modeled by L pretrain =L+w a L a +w v L v Good use to guide, where w a and w v The pure audio decoder loses 416 (L a ) and the pure video decoder loss 418 (L v The weight of the pure audio decoder loss is 416. a Can be updated by iterating as follows:
[0046]
[0047]
[0048] Here, β∈(0,1) is a smoothing hyperparameter, t is the number of iterations, is obtained using a softmax function on the gap between the multimodal decoder loss 420 of the current iteration and each pure audio decoder loss 416, and α>0 is a temperature hyperparameter. If the audio difference index G of the audio modality on the pre-training dataset aIf it is larger, it is model L pretrain =L+w a L a +w v L v Assign a higher weight w to the pure audio decoder loss 416 a As optimization progresses and the audio difference index G a The weight w of the audio-only decoder loss 416 may be adjusted dynamically as time changes. a To increase the weight of the underfitting mode.
[0049] Similarly, the pure video decoder loses a weight w of 418 v Can be updated by iterating as follows:
[0050]
[0051]
[0052] Here, β∈(0,1) is a smoothing hyperparameter, t is the number of iterations, is obtained using a softmax function on the gap between the multimodal decoder loss 420 of the current iteration and each pure video decoder loss 418, and α>0 is a temperature hyperparameter. If the video difference index G of the video modality on the pre-training dataset v If it is larger, it is model L pretrain =L+w a L a +w v L v Assign a higher weight w to the pure video decoder loss 418 v As optimization progresses and the video difference index G v The weight w of the pure video decoder loss 418 may be adjusted dynamically as time changes. v To increase the weight of the underfitting mode.
[0053] Although Figure 1-Figure 6 The video encoder 106, audio encoder 108, cross-modal encoder 110, and subtitle decoder 112 are depicted as separate modules, but different combinations of these modules may be implemented within a single video subtitle generation process.
[0054] See also Fig. 7A , which shows a schematic diagram of an application client 216 for a video rendering program, which can be used in a computer such as Figure 22 is implemented on a client computing device 214 shown in . The application client 216 receives input including an input video 118 through a graphical user interface 220. In response, the application client 216 generates and outputs video captions 120 corresponding to the input video 118 on the graphical user interface 220. In this example, the input video 118 is a video of a baby, and the video captions 120 correspond to the input video 118. In this example, the video captions 120 indicate that the woman is laughing and talking to the baby, and that the baby is crying.
[0055] Figure 7B A schematic diagram of an application client 216 for a video rendering program is shown that outputs video captions 120 corresponding to an input video 118 of a girl pointing to a picture in a book. In this example, the video captions 120 indicate that a little girl is reading a book while a man is talking to her.
[0056] Figure 8 A flow chart of a computerized method 500 for generating subtitles for video frames of an input video using a video subtitle generation process is illustrated. Figures 1 to 8 The following description of the computerized method 500 is provided with reference to the software and hardware components shown in FIG. It should be appreciated that the computerized method 500 may also be performed in other environments using other suitable hardware and software components.
[0057] At step 502, an input video is received from a user. At step 504, video frames are sampled from the input video. At step 506, a video embedding sequence is extracted from the video frames using a video encoder. The extracted video embedding may include feature dimensions, local video tokens, and global video tokens. The global video token may be a global representation of an image of the input video that captures high-level concepts, such as people gathering on a street.
[0058] At step 508, an audio embedding sequence is extracted from the associated audio spectrogram of the video frame using an audio encoder, which can be an audio transformer. The extracted audio embedding can include feature dimensions, local audio tokens, and global audio tokens. The global audio token can be a global representation of the audio of the input video that captures high-level concepts, such as laughter.
[0059] At step 510, cross-modal encoding is performed on the extracted embedded word-grams. Step 510 may include step 510a of performing a merge fusion of video word-grams and audio word-grams, and step 510b of performing a global cross fusion of video word-grams and audio word-grams. Step 510a includes step 510aa of splicing together local video word-grams and local audio word-grams, and step 510ab of inputting the spliced video word-grams and the spliced audio word-grams into a transformer block, which merges the spliced video word-grams and the spliced audio word-grams. The merged local video word-grams and the merged local audio word-grams are then output.
[0060] Step 510b includes: step 510ba, receiving local video words and global video words as a query of a video transformer, local video words as keys, and global audio words as values, and then outputting fused local video words and fused global video words; and step 510bb, receiving local audio words and global audio words as a query of an audio transformer, local audio words as keys, and global video words as values, and then outputting fused local audio words and fused global audio words.
[0061] Step 510 includes step 510c, averaging the merged local video word-gram and the fused local video word-gram to output the averaged local video word-gram, and averaging the merged local audio word-gram and the fused local audio word-gram to output the averaged local audio word-gram. At step 510d, the averaged local word-gram is iteratively input into the subsequent fusion layer of the cross-modal encoder until step 510e when the final multimodal embedding is output, which includes the averaged local video word-gram, the averaged local audio word-gram, the fused global video word-gram, the fused global audio word-gram and the feature dimension.
[0062] At step 512, the multimodal embedding is input into the subtitle decoder. At step 514, different BOS tokens are input into the subtitle decoder to initiate subtitle generation. The BOS may include a first BOS token that initiates prediction of the current video subtitle token and a second BOS token that initiates prediction of the next video subtitle token. At step 516, the generated video subtitle token is output autoregressively.
[0063] Fig. 9 A flowchart of a computerized method 600 for training a cross-modal encoder and a subtitle decoder using data and model distillation is illustrated. Figures 1 to 8 The following description of the computerized method 600 is provided with reference to the software and hardware components shown in FIG. It should be appreciated that the computerized method 600 may also be performed in other environments using other suitable hardware and software components.
[0064] At step 602, an input video is received from a user. At step 604, a video frame is sampled from the input video. At step 606, a video embedding sequence is extracted from the video frame using a video encoder. At step 608, an audio embedding sequence is extracted from an associated audio spectrogram of the video frame using an audio encoder, which may be an audio transformer.
[0065] At step 610, cross-modal encoding is performed on the extracted embedded tokens. At step 612, the multimodal embedding is input into the subtitle decoder. At step 614, different BOS tokens are input into the subtitle decoder to initiate subtitle generation. At step 616, the generated video subtitle tokens are output autoregressively.
[0066] At step 618, an audio-only decoder loss, a video-only decoder loss, and a multimodal decoder loss are calculated based on the generated video subtitle tokens and the true value subtitles. The audio-only decoder loss is calculated by inputting the video embedding set to all zeros and the audio embedding of the video frame into a cross-modal encoder to generate an audio-only unimodal embedding, inputting the audio-only unimodal embedding into a subtitle decoder to generate a video subtitle token, and then calculating the audio-only decoder loss compared to the true value subtitle using a cross-entropy loss function.
[0067] The video-only decoder loss is calculated by feeding the audio embeddings set to all zeros and the video embeddings of the video frames into the cross-modal encoder to generate video-only unimodal embeddings, feeding the video-only unimodal embeddings into the subtitle decoder to generate video subtitle tokens, and then using the cross-entropy loss function to calculate the video-only decoder loss compared to the ground-truth subtitles.
[0068] The multimodal decoder loss is calculated by feeding the audio embedding and video embedding of the video frame into the cross-modal encoder to generate multimodal embeddings, feeding the multimodal embeddings into the subtitle decoder to generate video subtitle tokens, and then using the cross-entropy loss function to calculate the multimodal decoder loss compared with the ground-truth subtitles.
[0069] At step 620, an audio difference index is calculated to measure the difference between the audio-only decoder loss and the multimodal decoder loss. Likewise, at step 622, a video difference index is calculated to measure the difference between the video-only decoder loss and the multimodal decoder loss.
[0070] At step 624, the calculated cross entropy loss and difference index are used to update the trainable parameters of the cross-modal encoder and subtitle decoder by gradient back propagation. When the video difference index is high, a higher weight is assigned to the video modality of the cross-modal encoder and subtitle decoder, so that more attention is given to the video modality and the over-specialization of the audio modality is reduced. When the audio difference index is high, a higher weight is assigned to the audio modality of the cross-modal encoder and subtitle decoder, so that more attention is given to the audio modality and the over-specialization of the video modality is reduced. Steps 618 to 624 are then repeated for each stage.
[0071] Fig.10 is a table from an ablation study showing the changes Figures 1 to 5 The effect of the cross-modal decoder's method of processing audio and video tokens on the performance of video captioning. Four metrics, BLEU-4 (B), METEOR (M), ROUGE-L (R), and CIDEr (C), are used to evaluate the video captioning performance of the captioning program on the YouCook2 video captioning dataset, which contains 2,000 cooking videos and 15,400 video clips.
[0072] When the cross-modal encoder only uses the merging fusion module to perform the merging fusion of local video tokens and local audio tokens, and omits the global cross-fusion module, the video caption generation performance measured by each of the four metrics is: 19.9 for BLEU-4, 23.6 for METEOR, 49.1 for ROUGE-L, and 210.7 for CIDEr.
[0073] When the cross-modal encoder only uses the global cross-fusion module to perform global cross-fusion of local tokens and global tokens, and omits the merge fusion module, the video captioning performance measured by each of the four metrics is: 18.6 for BLEU-4, 22.9 for METEOR, 48.0 for ROUGE-L, and 202.3 for CIDEr.
[0074] On the other hand, when the cross-modal encoder is implemented as Figure 3 When both the merged fusion module and the global cross fusion module described in the embodiment are used, the video subtitle generation performance measured by each of the four metrics is: BLEU-4 is 20.6, METEOR is 24.2, ROUGE-L is 49.6, and CIDEr is 217.0, which indicates that the implementation of the merged fusion module or the global cross fusion module is omitted in the cross modal encoder. Therefore, the inventors conclude that implementing the merged fusion module and the global cross fusion module in the cross modal encoder optimizes the video subtitle generation performance measured by the above metrics.
[0075] The above-described system and method use an effective scheme to use a cross-modal encoder so that the model used for video caption generation can learn to include important information conveyed by audio elements in the video. Therefore, excellent accessibility and overall viewing experience can be achieved by generating video description captions that describe audio elements that can convey important information or context, including acoustic effects and speaker identity.
[0076] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0077] Fig.11 A non-limiting embodiment of a computing system 700 that can perform one or more of the above methods and processes is schematically illustrated. The computing system 700 is shown in simplified form. The computing system 700 can embody the above-described and Figure 2 and Figure 3 1, 2, and 3. The computing system 700 may be in the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.
[0078] The computing system 700 includes a logic processor 702, a volatile memory 704, and a non-volatile storage device 706. The computing system 700 may optionally include a display subsystem 708, an input subsystem 710, a communication subsystem 712, and / or Fig.11 Other components not shown.
[0079] Logical processor 702 includes one or more physical devices configured to execute instructions. For example, logical processor 702 may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve a desired result.
[0080] The logical processor 702 may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor 702 may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. The various components of the logical processor 702 may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor 702 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it will be understood that these virtualized aspects are run on different physical logical processors of various different machines.
[0081] The non-volatile storage device 706 includes one or more physical devices configured to store instructions executable by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 706 may be transformed—for example, to store different data.
[0082] The non-volatile storage device 706 may include a removable and / or built-in physical device. The non-volatile storage device 706 may include an optical memory, a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.), or other mass storage device technology. The non-volatile storage device 706 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 706 is configured to save instructions even when the power to the non-volatile storage device 706 is cut off.
[0083] The volatile memory 704 may include physical devices including random access memory. The volatile memory 704 is typically utilized by the logical processor 702 to temporarily store information during the processing of software instructions. It should be understood that when the power to the volatile memory 704 is cut off, the volatile memory 704 typically does not continue to store instructions.
[0084] Aspects of the logic processor 702, volatile memory 704, and non-volatile storage device 706 may be integrated together into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).
[0085] The terms "module", "program", and "engine" may be used to describe an aspect of a computing system 700 that is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformative process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated by executing instructions stored by a non-volatile storage device 706 via a logical processor 702 using portions of volatile memory 704. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover a single or a group of executable files, data files, libraries, drivers, scripts, database records, and the like.
[0086] When included, the display subsystem 708 can be used to present a visual representation of the data stored by the non-volatile storage device 706. The visual representation can take the form of a graphical user interface (GUI). When the methods and processes described herein change the data stored by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 708 can also be transformed to visually represent the changes in the underlying data. The display subsystem 708 may include one or more display devices utilizing virtually any type of technology. Such a display device can be combined in a shared housing with the logical processor 702, the volatile memory 704, and / or the non-volatile storage device 706, or such a display device can be a peripheral display device.
[0087] When included, the input subsystem 710 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem 710 may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.
[0088] When included, the communication subsystem 712 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple with other devices. The communication subsystem 712 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or a wide area network (such as an HDMI connected by Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 700 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.
[0089] The following paragraphs provide additional description of the subject matter of the present disclosure. One aspect provides a video subtitle generation system, including a processor and a memory of a computing device, the processor being configured to use a portion of the memory to execute a program to receive an input video, extract a video frame from the input video, extract a video embedding and an audio embedding including a local video word unit and a local audio word unit, respectively, from the video frame, input the local video word unit and the local audio word unit into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding, and generate a video subtitle using the multimodal embedding of a subtitle decoder.
[0090] In this aspect, additionally or alternatively, generation of video subtitles may be initiated by inputting one or more start-of-sentence (BOS) tokens into a subtitle decoder.
[0091] In this aspect, additionally or alternatively, the BOS word-gram may include a first BOS word-gram configured to initiate prediction of a current video subtitle word-gram, and a second BOS word-gram configured to initiate prediction of a next video subtitle word-gram.
[0092] In this aspect, additionally or alternatively, the cross-modal encoder may include a merging and fusion module, which is configured to splice local video words and local audio words, input the spliced local video words and the spliced local audio words into the first transformer and the second transformer respectively, and output merged local video words and merged local audio words.
[0093] In this aspect, additionally or alternatively, the key and value of the first transformer can be obtained from the local video word element, and the query of the first transformer can be obtained from the local audio word element, and the key and value of the second transformer can be obtained from the local audio word element, and the query of the second transformer can be obtained from the local video word element.
[0094] In this aspect, additionally or alternatively, the cross-modal encoder includes a global cross-fusion module comprising a video transformer and an audio transformer, which global cross-fusion module can be configured to receive local video words, local audio words, global video words and global audio words as input, and output fused local video words, fused global video words, fused local audio words and fused global audio words.
[0095] In this aspect, additionally or alternatively, the video transformer may receive local video terms and global video terms as queries, local video terms as keys, and global audio terms as values, and output fused local video terms and fused global video terms, and the audio transformer may receive local audio terms and global audio terms as queries, local audio terms as keys, and global video terms as values, and output fused local audio terms and fused global audio terms.
[0096] In this aspect, additionally or alternatively, the cross-modal encoder may include: a global cross-fusion module including a video transformer and an audio transformer, the global cross-fusion module being configured to receive local video words, local audio words, global video words and global audio words as inputs, and output fused local video words, fused global video words, fused local audio words and fused global audio words; and a merging and fusion module being configured to splice local video words and local audio words, input the spliced local video words and the spliced local audio words into the first transformer and the second transformer respectively, and output merged local video words and merged local audio words, average the merged local video words and the fused local video words to output averaged local video words, average the merged local audio words and the fused local audio words to output averaged local audio words, and iteratively input the averaged local video words and the averaged local audio words into subsequent fusion layers of the cross-modal encoder.
[0097] In this aspect, additionally or alternatively, a cross entropy loss function can be used to calculate a pure audio decoder loss, a pure video decoder loss, and a multimodal decoder loss based on the generated video subtitles and the true subtitles, and the trainable parameters of the cross-modal encoder and subtitle decoder can be updated using the pure audio decoder loss, the pure video decoder loss, and the multimodal decoder loss.
[0098] In this aspect, additionally or alternatively, an audio difference index between a pure audio decoder loss and a multimodal decoder loss may be calculated, a video difference index between a pure video decoder loss and a multimodal decoder loss may be calculated, and trainable parameters of a cross-modal encoder and a subtitle decoder may be updated using the audio difference index and the video difference index.
[0099] On the other hand, a video subtitle generation method is provided, comprising: receiving an input video; extracting a video frame from the input video; extracting a video embedding and an audio embedding including a local video word unit and a local audio word unit, respectively, from the video frame; inputting the local video word unit and the local audio word unit into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding, and generating a video subtitle based on the multimodal embedding using a subtitle decoder.
[0100] In this aspect, the method may additionally or alternatively further comprise initiating generation of video subtitles by inputting one or more start-of-sentence (BOS) tokens into a subtitle decoder.
[0101] In this aspect, additionally or alternatively, the method may further include initiating prediction of a current video subtitle word by inputting a first BOS word of the one or more BOS words into a subtitle decoder, and initiating prediction of a next video subtitle word by inputting a second BOS word of the one or more BOS words into the subtitle decoder.
[0102] In this aspect, additionally or alternatively, the method may also include performing merging and fusion in the following manner: splicing local video words and local audio words, inputting the spliced local video words and the spliced local audio words into the first transformer and the second transformer respectively, and outputting merged local video words and merged local audio words.
[0103] In this aspect, additionally or alternatively, the method may also include performing global cross-fusion by receiving local video words, local audio words, global video words and global audio words as input, and outputting fused local video words, fused global video words, fused local audio words and fused global audio words.
[0104] In this aspect, additionally or alternatively, at a video converter, local video words and global video words are received as queries, local video words are received as keys, and global audio words are received as values, and the video converter outputs fused local video words and fused global video words, and at an audio converter, local audio words and global audio words are received as queries, local audio words are received as keys, and global video words are received as values, and the audio converter outputs fused local audio words and fused global audio words.
[0105] In this aspect, additionally or alternatively, the method may also include performing merging and fusion in the following manner: splicing local video words and local audio words, inputting the spliced local video words and the spliced local audio words into the first transformer and the second transformer respectively, and outputting merged local video words and merged local audio words; performing global cross-fusion in the following manner: receiving local video words, local audio words, global video words and global audio words as input, and outputting fused local video words, fused global video words, fused local audio words and fused global audio words; averaging the merged local video words and the fused local video words to output averaged local video words, averaging the merged local audio words and the fused local audio words to output averaged local audio words, and iteratively inputting the averaged local video words and the averaged local audio words to subsequent fusion layers of the cross-modal encoder.
[0106] In this aspect, additionally or alternatively, the method may further include calculating a pure audio decoder loss, a pure video decoder loss, and a multimodal decoder loss based on the generated video subtitles and the true subtitles using a cross entropy loss function, and updating trainable parameters of the cross-modal encoder and subtitle decoder using the pure audio decoder loss, the pure video decoder loss, and the multimodal decoder loss.
[0107] In this aspect, additionally or alternatively, the method may further include calculating an audio difference index between a pure audio decoder loss and a multimodal decoder loss, calculating a video difference index between a pure video decoder loss and a multimodal decoder loss, and using the audio difference index and the video difference index to update trainable parameters of a cross-modal encoder and a subtitle decoder.
[0108] In another aspect, a video caption generation system is provided, including a processor and a memory of a computing device, the processor being configured to use a portion of the memory to execute a program to receive an input video, extract a video frame from the input video, extract a video embedding including a local video word-gram, extract an audio embedding including a local audio word-gram corresponding to the local video word-gram, input the local video word-gram and the local audio word-gram into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding, and generate a video caption based on the multimodal embedding using a caption decoder.
[0109] It should be understood that the configuration and / or method described herein is exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described can be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process can be changed.
[0110] It should be understood that "and / or" used herein refers to a logical OR operation, so A and / or B has the following truth table.
[0111] A B A and / or B real real real real Fake real Fake real real Fake Fake Fake
[0112] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents.
Claims
1. A video subtitle generation system, comprising: A processor and memory of a computing device, the processor being configured to use portions of the memory to execute a program to: Receive input video; Extracting video frames from the input video; Extracting a video embedding and an audio embedding including local video tokens and local audio tokens, respectively, from the video frame; Inputting the local video word-gram and the local audio word-gram into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding; as well as Captions for the video are generated based on the multimodal embedding using a caption decoder.
2. The video subtitle generation system according to claim 1, wherein the generation of the video subtitles is initiated by inputting one or more beginning-of-sentence (BOS) tokens into the subtitle decoder.
3. The video subtitle generation system according to claim 2, wherein the BOS word-grams include: A first BOS word-element configured to initiate prediction of a current video subtitle word-element; as well as A second BOS word-gram is configured to initiate a prediction of a next video subtitle word-gram.
4. The video subtitle generation system according to claim 1, wherein the cross-modal encoder comprises a merging and fusion module, and the merging and fusion module is configured to: splicing the local video word-element and the local audio word-element; Inputting the spliced partial video word unit and the spliced partial audio word unit into a first converter and a second converter respectively; and Output the merged partial video word-gram and the merged partial audio word-gram.
5. The video subtitle generation system according to claim 4, wherein: deriving a key and a value of the first transformer from the local video word-gram, and deriving a query of the first transformer from the local audio word-gram; and The key and value of the second transformer are obtained from the local audio word-gram, and the query of the second transformer is obtained from the local video word-gram.
6. The video subtitle generation system according to claim 1, wherein the cross-modal encoder comprises: A global cross-fusion module including a video converter and an audio converter, wherein the global cross-fusion module is configured as follows: receiving the local video word-gram, the local audio word-gram, the global video word-gram, and the global audio word-gram as input; and Output the fused local video word-gram, the fused global video word-gram, the fused local audio word-gram and the fused global audio word-gram.
7. The video subtitle generation system according to claim 6, wherein: The video transformer receives the local video word-gram and the global video word-gram as a query, the local video word-gram as a key, and the global audio word-gram as a value, and outputs the fused local video word-gram and the fused global video word-gram; as well as The audio transformer receives the local audio words and the global audio words as queries, the local audio words as keys, and the global audio words as values, and outputs the fused local audio words and the fused global audio words.
8. The video subtitle generation system according to claim 1, wherein the cross-modal encoder comprises: A global cross-fusion module including a video converter and an audio converter, wherein the global cross-fusion module is configured as follows: receiving as input the local video word-gram, the local audio word-gram, the global video word-gram and the global audio word-gram, and Outputting fused local video word-grams, fused global video word-grams, fused local audio word-grams, and fused global audio word-grams; as well as A merging and fusing module, wherein the merging and fusing module is configured as follows: splicing the local video word-element and the local audio word-element, inputting the spliced partial video word unit and the spliced partial audio word unit into a first converter and a second converter respectively, and Output the merged partial video word unit and the merged partial audio word unit, where: Averaging the merged local video word-element and the fused local video word-element to output an averaged local video word-element; Averaging the merged partial audio word-meta and the fused partial audio word-meta to output an averaged partial audio word-meta; and The averaged local video word-gram and the averaged local audio word-gram are iteratively input into a subsequent fusion layer of the cross-modal encoder.
9. The video subtitle generation system according to claim 1, wherein: A cross entropy loss function is used to calculate a pure audio decoder loss, a pure video decoder loss, and a multimodal decoder loss based on the generated video subtitles and the true subtitles; as well as Trainable parameters of the cross-modal encoder and the subtitle decoder are updated using the audio-only decoder loss, the video-only decoder loss, and the multimodal decoder loss.
10. The video subtitle generation system according to claim 9, wherein: An audio difference index between the audio-only decoder loss and the multimodal decoder loss is calculated; A video difference index between the pure video decoder loss and the multimodal decoder loss is calculated; as well as Trainable parameters of the cross-modal encoder and the subtitle decoder are updated using the audio difference index and the video difference index.
11. A method for generating video subtitles, comprising: Receive input video; Extracting video frames from the input video; Extracting a video embedding and an audio embedding including local video tokens and local audio tokens, respectively, from the video frame; Inputting the local video word-gram and the local audio word-gram into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding; as well as Captions for the video are generated based on the multimodal embedding using a caption decoder.
12. The video subtitle generation method according to claim 11, further comprising: The generation of the video subtitles is initiated by inputting one or more beginning-of-sentence (BOS) tokens into the subtitle decoder.
13. The video subtitle generation method according to claim 12, further comprising: Initiating prediction of a current video subtitle word by inputting a first BOS word of the one or more BOS words into the subtitle decoder; as well as Prediction of a next video subtitle word-element is initiated by inputting a second BOS word-element of the one or more BOS words-element into the subtitle decoder.
14. The video subtitle generation method according to claim 11 further comprises performing merging and fusion in the following manner: splicing the local video word-element and the local audio word-element; Inputting the spliced partial video word unit and the spliced partial audio word unit into a first converter and a second converter respectively; and Output the merged partial video word-gram and the merged partial audio word-gram.
15. The video subtitle generation method according to claim 11 further comprises performing global cross-fusion in the following manner: receiving the local video word-gram, the local audio word-gram, the global video word-gram, and the global audio word-gram as input; and Output the fused local video word-gram, the fused global video word-gram, the fused local audio word-gram and the fused global audio word-gram.
16. The video subtitle generation method according to claim 15, wherein: At a video transformer, the local video word-gram and the global video word-gram are received as queries, the local video word-gram is used as a key, and the global audio word-gram is used as a value, and the video transformer outputs a fused local video word-gram and a fused global video word-gram; as well as At the audio transformer, the local audio words and the global audio words are received as queries, the local audio words are used as keys, and the global video words are used as values, and fused local audio words and fused global audio words are output by the audio transformer.
17. The video subtitle generation method according to claim 11, further comprising: Perform the merge fusion by: splicing the local video word-element and the local audio word-element, inputting the spliced local video word unit and the spliced local audio word unit into the first converter and the second converter respectively, and Outputting a merged partial video word unit and a merged partial audio word unit; Perform a global cross-fusion by: receiving the local video word-gram, the local audio word-gram, the global video word-gram and the global audio word-gram as input, and Outputting fused local video word-grams, fused global video word-grams, fused local audio word-grams, and fused global audio word-grams; Averaging the merged local video word-element and the fused local video word-element to output an averaged local video word-element; Averaging the combined local audio word unit and the fused local audio word unit to output an averaged local audio word unit; as well as The averaged local video word-gram and the averaged local audio word-gram are iteratively input into a subsequent fusion layer of the cross-modal encoder.
18. The video subtitle generation method according to claim 11, further comprising: Using a cross entropy loss function, the audio-only decoder loss, the video-only decoder loss, and the multimodal decoder loss are calculated based on the generated video subtitles and the true subtitles; as well as Trainable parameters of a cross-modal encoder and a subtitle decoder are updated using the audio-only decoder loss, the video-only decoder loss, and the multimodal decoder loss.
19. The video subtitle generation method according to claim 18, further comprising: calculating an audio difference index between the audio-only decoder loss and the multimodal decoder loss; calculating a video difference index between the pure video decoder loss and the multimodal decoder loss; as well as Trainable parameters of the cross-modal encoder and the subtitle decoder are updated using the audio difference index and the video difference index.
20. A video subtitle generation system, comprising: A processor and memory of a computing device, the processor being configured to use portions of the memory to execute a program to: Receive input video; Extracting video frames from the input video; extracting video embeddings including local video tokens; extracting an audio embedding including a local audio word-metaphor corresponding to the local video word-metaphor; Inputting the local video word-gram and the local audio word-gram into at least one transformer layer of a cross-modal encoder to generate a multimodal embedding; as well as Captions for the video are generated based on the multimodal embedding using a caption decoder.
