A large model-based educational video caption generation method

Abstract

This invention discloses a method for generating subtitles for educational videos based on a large model, comprising the following steps: inputting original video data, segmenting frames and extracting visual features; performing multimodal feature fusion and decoding through a cross-attention mechanism and a Swing Transformer; and jointly optimizing model parameters during training using the aforementioned weighted combined loss function to generate video subtitles. This method for generating subtitles for educational videos, by fusing multimodal features, enables the model to more comprehensively understand the video content, significantly improving the matching degree between the generated subtitles and the original video's audio and scene, achieving high-accuracy subtitle output. Furthermore, by introducing language model perplexity as a readability loss, it significantly improves the fluency and naturalness of the generated subtitles, making the subtitle content easier to understand and read.

Description

Technical Field

[0001] This invention relates to the field of video caption generation technology, and in particular to a method for generating educational video captions based on a large model. Background Technology

[0002] In recent years, with the rapid development of the internet and multimedia technologies, video has become an important medium for education and knowledge dissemination. Educational videos, as the core content of online courses and MOOC platforms, place higher demands on efficient and accurate subtitle generation. Subtitles not only provide barrier-free access for learners with hearing impairments and different language backgrounds, but also play a fundamental role in knowledge retrieval, course content organization, and smart note-taking.

[0003] Currently, the mainstream methods for generating video subtitles can be divided into two main categories: One type is manual annotation, which is highly accurate but inefficient and costly, making it difficult to meet the needs of large-scale educational video production. Another type of technology is based on Automatic Speech Recognition (ASR), which can automatically transcribe lecture audio to some extent. However, this method can only process audio information and has difficulty recognizing and integrating non-audio information in the video, such as whiteboard writing, PPT text, and teacher gestures. This results in incomplete subtitles, affecting the learning experience and knowledge acquisition. In addition, traditional methods have lower recognition accuracy in complex scenarios such as accents, multiple speakers, and noise interference, and lack a deep understanding of educational terminology and contextual semantics.

[0004] With the development of deep learning and large-scale pre-trained multimodal models, the ability to fuse multi-source information such as audio and video has been significantly improved. For example, novel network structures such as the Swin Transformer can extract and fuse spatial and temporal information between video frames more precisely, while multi-attention mechanisms support dynamic joint modeling of multimodal features such as audio and vision. However, existing methods still have significant shortcomings in efficient video feature representation in large models, deep multimodal collaboration, and subtitle decoding in complex contexts. New technological breakthroughs are urgently needed to meet the practical requirements of high accuracy, high robustness, and high scalability for automatic subtitle generation in educational videos. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for generating subtitles for educational videos based on a large model, which solves the following: 1. Automatic extraction and representation of multimodal features: This research aims to address how to automatically extract and model rich visual content (such as PPTs and blackboard notes) and their context in educational videos using large-scale pre-trained models, thereby improving the comprehensiveness and accuracy of feature representations and overcoming the limitations of traditional convolutional networks and single-modal methods in spatial, temporal, and semantic dimensions.

[0006] 2. Deep fusion and understanding of audio and video multimodal features: We propose a feature fusion method based on large models and multi-attention mechanisms to achieve dynamic joint modeling of features from multiple modalities such as audio and vision, thereby improving the ability to accurately capture and semantically understand core knowledge points in complex scenarios (such as classroom interaction, multi-person dialogue, and noise interference).

[0007] 3. High-quality subtitle decoding: By leveraging deep feature fusion and subtitle decoding networks, global and local information are organically integrated to improve the accuracy, fluency, and expressiveness of subtitle generation, meeting the needs of different educational scenarios.

[0008] 4. End-to-end automation and high generalization capability: Through end-to-end training and automated modeling, an automatic subtitle generation system is achieved, effectively enhancing its adaptability and generalization ability to diverse educational video content and multilingual environments.

[0009] This invention provides a method for generating subtitles for educational videos based on a large model, comprising the following steps: inputting raw video data, segmenting frames and extracting visual features; performing multimodal feature fusion and decoding through a cross-attention mechanism and a Swing Transformer; and generating video subtitles by jointly optimizing model parameters through the aforementioned weighted combined loss function during training.

[0010] Preferably, in the video preprocessing stage, the original video is first decoded, splitting it into an image frame sequence and an audio stream. Subsequently, the multimodal large model Ming-Omni is used to extract and encode features from the video frames, audio information, and subtitle content, respectively, and map their respective features to a unified multimodal feature space.

[0011] Preferably, in the video feature fusion stage, a cross-attention mechanism is used to deeply fuse multimodal features.

[0012] Preferably, a cross-attention mechanism is used to deeply fuse multimodal features; the corresponding subtitle features are used as queries, while video frames and audio features are used as values ​​and keys, and the cross-attention module realizes the information interaction and fusion between subtitles and video multimodal features.

[0013] Preferably, in the decoding stage of video subtitle generation, the Swing Transformer is first introduced to perform deep modeling of the complex dynamic dependencies of the video; then, a spatiotemporal attention mechanism is further adopted to model feature dependencies in both spatial and temporal dimensions, and to perform more refined modeling and decoding of the fused multimodal global features.

[0014] Preferably, during the training phase of the video subtitle generation model, a joint optimization loss function is used to train the model end-to-end. The loss function integrates three objectives: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss.

[0015] Preferably, the model is trained using a joint optimization loss function, which includes three loss functions: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss, as shown in the formula: in, Let cross-entropy be the sequence loss function. For multimodal semantic consistency loss function, For readability loss function, and This represents the loss coefficient. Preferably, it is the sequence cross-entropy loss. This is used to measure the difference between the sequence generated by the model and the real sequence, to ensure the correctness of the generated subtitles; its formula is: j = 1,2,......,L in Let L be the predicted probability of the j-th word, and L be the length of the subtitle sequence.

[0016] Preferably, the multimodal semantic consistency loss function By minimizing the cosine distance between the generated subtitles and the video content in the semantic space, the features of the two are aligned and mapped to a unified semantic space to enhance the semantic relevance between the generated subtitles and the video content. in, For subtitle encoders, For visual encoders; For voice encoders; As a visual feature, These are audio features.

[0017] Preferably, the readability loss function We employ a language model-based perplexity metric to improve the fluency and readability of subtitles by minimizing the perplexity of the generated subtitles. and Normalization is performed using the Softmax function to ensure that the weights are non-negative and sum to 1, thus achieving adaptive and normalized loss weight allocation. , The parameters are learnable and trained along with the model network. The optimal weight allocation is automatically obtained through gradient descent, and the parameters are adaptively adjusted during training, requiring no manual setting or updating. Compared with related technologies, the educational video subtitle generation method based on a large model provided by this invention has the following beneficial effects: 1. Improved accuracy of subtitle generation By integrating multimodal features (video, audio, and text), the model can more comprehensively understand video content, significantly improve the matching degree between the generated subtitles and the original video audio and scene, and achieve high-accuracy subtitle output.

[0018] 2. Optimized subtitle fluency and readability Introducing language model perplexity as a readability loss significantly improves the fluency and naturalness of the generated subtitles, making the subtitle content easier to understand and read.

[0019] 3. Enhanced semantic consistency Multimodal semantic consistency loss effectively aligns subtitles with video multimodal information, ensuring that the generated subtitles are highly relevant to the video content at the semantic level, and avoiding semantic deviation or content disconnect.

[0020] 4. Improved spatiotemporal correlation modeling capabilities By combining the Swing Transformer and the spatiotemporal attention mechanism, the temporal relationships and motion changes between video frames can be better captured during the generation process, enabling the synchronization of subtitles with dynamic elements such as images and actions.

[0021] 5. Multi-loss adaptive optimization enhances robustness. The loss weight adaptive learning effectively balances the accuracy, relevance, and readability of subtitles, enabling the system to exhibit strong adaptability and robustness under various video types and complex content.

[0022] The present invention provides a method for generating subtitles for educational videos based on a large model. It mainly solves the key technologies for achieving deep fusion of multimodal information, efficient feature extraction, and automatic generation of high-quality subtitles in educational video scenarios, and has innovative and widely applicable value. Attached Figure Description

[0023] Figure 1 A schematic diagram of the subtitle generation model training program provided by this invention; Figure 2 A schematic diagram of the video subtitle reasoning program provided by the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] Please refer to the following: Figures 1-2 ,in, Figure 1 A schematic diagram of the subtitle generation model training program provided by this invention; Figure 2 A schematic diagram of the video subtitle reasoning program provided by the present invention.

[0026] In the specific implementation process, such as Figures 1-2 As shown, an educational video subtitle generation method based on a large model comprehensively utilizes advanced technical architectures such as the multimodal model Ming-Omni, a multi-attention mechanism, and the Swin Transformer to achieve efficient and intelligent video subtitle generation. Specifically, the Ming-Omni model is used to extract multimodal features such as video frames, audio, and text; the cross-attention mechanism achieves deep fusion of multimodal features; and the spatiotemporal attention mechanism and the Swin Transformer module jointly model the complex relationship between video frames, audio signals, and subtitle text, thereby effectively improving the accuracy and fluency of subtitle generation.

[0027] In the video preprocessing stage, the original video is first decoded, splitting it into a sequence of image frames and an audio stream. Then, the Ming-Omni multimodal large model is used to extract and encode features from the video frames, audio information, and subtitle content, mapping each feature to a unified multimodal feature space. This process yields a high-dimensional multimodal feature representation of the video, providing a solid data foundation for subsequent video subtitle generation, multimodal feature fusion, and complex correlation modeling.

[0028] In the video feature fusion stage, a cross-attention mechanism is employed to deeply fuse multimodal features. The corresponding subtitle features are used as queries, while video frames and audio features are used as values ​​and keys. The cross-attention module facilitates information interaction and fusion between subtitles and video multimodal features. This mechanism effectively captures the correlation between subtitle semantics and video content, enhancing the richness of multimodal feature expression and the contextual relevance of subtitle generation, thus providing more semantically consistent fused features for subsequent subtitle generation.

[0029] In the decoding stage of video subtitle generation, the Swin Transformer is first introduced to deeply model the complex dynamic dependencies of the video. Leveraging its hierarchical structure and moving window mechanism, the Swin Transformer efficiently captures long-distance dependencies and fine-grained dynamic changes between video frames. Furthermore, a spatiotemporal attention mechanism is employed, utilizing internal attention to simultaneously model feature dependencies in both spatial and temporal dimensions. This allows for more refined modeling and decoding of the fused multimodal global features, comprehensively uncovering the intrinsic connections between video, audio, and subtitles. This joint modeling effectively enhances the context-awareness and spatiotemporal consistency of subtitle generation, resulting in more accurate, natural, and fluent video subtitles.

[0030] During the training phase of the video subtitle generation model, a joint optimization loss function is used for end-to-end training. This loss function integrates three objectives: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss. This joint loss function not only ensures the correctness of subtitle generation but also considers the semantic consistency between subtitles and video content, as well as the fluency and expressive quality of the subtitles themselves, thereby significantly improving the practical value and user experience of the final subtitle generation.

[0031] Assume the generated subtitle sequence is Y = (y1, y2, ..., y...). n The video's multimodal features are represented by X, and the real subtitles by Y. * The probability distribution for generating subtitles is P(Y|X).

[0032] This invention utilizes a joint optimization loss function to train the model, which includes three loss functions: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss, as shown in the formula: in, Let cross-entropy be the sequence loss function. For multimodal semantic consistency loss function, For readability loss function, and The loss coefficient is the sequence cross-entropy loss. This is used to measure the difference between the sequence generated by the model and the real sequence, to ensure the correctness of the generated subtitles; its formula is: in Let L be the predicted probability of the j-th word, and L be the length of the subtitle sequence.

[0033] Multimodal semantic consistency loss function By minimizing the cosine distance between the generated subtitles and the video content in the semantic space, the features of both are aligned and mapped to a unified semantic space, thereby enhancing the semantic relevance between the generated subtitles and the video content. in, For subtitle encoders, For visual encoders; For voice encoders; As a visual feature, Audio features. Readability loss function. We employ a language model-based perplexity metric to improve the fluency and readability of subtitles by minimizing the perplexity of the generated subtitles. and Normalization is performed using the Softmax function to ensure that the weights are non-negative and sum to 1, thus achieving adaptive and normalized loss weight allocation. , As learnable parameters, they are trained along with the model network, automatically obtaining the optimal weight allocation through gradient descent, and adaptively adjusting during training without requiring manual setting or updates. This invention provides a method for generating educational video subtitles based on a large model, utilizing feature extraction from a multimodal large model: It proposes using a multimodal large model (such as Ming-Omni) to simultaneously perform unified feature extraction and encoding on video frames, audio information, and existing subtitles. Compared to traditional schemes that only utilize a single modality (such as video or audio only), this invention achieves fused input of multimodal data, improving the richness and representational power of input features.

[0034] A cross-attention mechanism enables deep multimodal fusion: This innovative approach uses subtitle features as the query and video frame and audio features as the key and value, respectively, introducing a cross-attention mechanism for feature fusion. This method better captures the complex relationships between video content and subtitle semantics, resulting in more context-relevant and semantically consistent subtitles.

[0035] Joint Modeling with Swin Transformer and Spatiotemporal Attention Mechanism: This invention employs Swin Transformer and spatiotemporal attention mechanism at the decoding end to jointly model the complex spatiotemporal relationships of video frame sequences and the correlation between audio signals and subtitle text. This can more effectively capture long-term dependencies and subtle dynamic changes, improving the spatiotemporal consistency of subtitles and the natural fluency of the generated text.

[0036] Joint multi-loss function end-to-end optimization: This invention proposes a joint optimization strategy of "sequence cross-entropy loss + multimodal semantic consistency loss + readability loss" to comprehensively consider subtitle correctness, video-subtitle semantic relevance, and the fluency and readability of the subtitles themselves. Furthermore, learnable parameters are introduced, and adaptive loss weights are achieved through Softmax normalization, automatically and dynamically adjusting without manual intervention.

[0037] This invention provides a method for generating subtitles for educational videos based on a large model: Feature extraction based on large visual models: This invention proposes a Ming-Omni visual large model that enables unified feature extraction and encoding of video frames, audio information, and subtitles simultaneously. Compared to traditional schemes that only utilize a single modality, this invention achieves the fusion of multimodal data input, improving the richness and representational power of input features.

[0038] Cross-attention mechanism enables deep multimodal fusion: This approach innovatively uses subtitle features as the query and video frame and audio features as the value and key, respectively, introducing a cross-attention mechanism for feature fusion. This method can better capture the complex relationship between video content and subtitle semantics, making the generated subtitles more context-relevant and semantically consistent.

[0039] Joint modeling using Swin Transformer and spatiotemporal attention mechanism: This invention employs the Swing Transformer and a spatiotemporal attention mechanism at the decoding end to jointly model the complex dynamic dependencies of video frame sequences and the correlation between audio signals and subtitle text. This enables more effective capture of long-term dependencies and subtle dynamic changes, improving the spatiotemporal consistency of subtitles and the natural fluency of their generation.

[0040] Joint multi-loss function end-to-end optimization: This invention proposes a joint optimization strategy employing "sequence cross-entropy loss + multimodal semantic consistency loss + readability loss," comprehensively considering subtitle accuracy, video-subtitle semantic relevance, and the fluency and readability of the subtitles themselves. Furthermore, it introduces learnable parameters and achieves adaptive loss weights through Softmax normalization, automatically and dynamically adjusting them without manual intervention.

[0041] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating subtitles for educational videos based on a large model, characterized in that, Includes the following steps: Input the raw video data, segment the frames and extract visual features; Multimodal feature fusion and decoding are performed using a cross-attention mechanism and a Swing Transformer. During training, the model parameters are jointly optimized using the aforementioned weighted combined loss function to generate video subtitles.

2. The method for generating subtitles for educational videos based on a large model according to claim 1, characterized in that: In the video preprocessing stage, the original video is first decoded, splitting it into a sequence of image frames and an audio stream. Subsequently, the multimodal large model Ming-Omni is used to extract and encode features from the video frames, audio information, and subtitle content, and map their respective features to a unified multimodal feature space.

3. The method for generating subtitles for educational videos based on a large model according to claim 2, characterized in that: In the video feature fusion stage, a cross-attention mechanism is used to deeply fuse multimodal features.

4. The method for generating subtitles for educational videos based on a large model according to claim 3, characterized in that: A cross-attention mechanism is used to deeply fuse multimodal features. The corresponding subtitle features are used as queries, while video frames and audio features are used as values ​​and keys. The cross-attention module enables information interaction and fusion between subtitles and video multimodal features.

5. The method for generating subtitles for educational videos based on a large model according to claim 4, characterized in that: In the decoding stage of video subtitle generation, the Swing Transformer is first introduced to deeply model the complex dynamic dependencies of the video; then, a spatiotemporal attention mechanism is adopted to model feature dependencies in both spatial and temporal dimensions, and to perform more refined modeling and decoding of the fused multimodal global features.

6. The method for generating subtitles for educational videos based on a large model according to claim 5, characterized in that: During the training phase of the video caption generation model, a joint optimization loss function is used to train the model end-to-end. The loss function integrates three objectives: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss.

7. The method for generating subtitles for educational videos based on a large model according to claim 6, characterized in that: The model is trained using a joint optimization loss function, which includes three loss functions: sequence cross-entropy loss, multimodal semantic consistency loss, and readability loss, as shown in the formula: in, Let cross-entropy be the sequence loss function. For multimodal semantic consistency loss function, For readability loss function, and This is the loss coefficient.

8. The method for generating subtitles for educational videos based on a large model according to claim 7, characterized in that: Sequence cross-entropy loss It is used to measure the difference between the sequence generated by the model and the real sequence, in order to ensure the correctness of the generated subtitles; The formula is: j = 1,2,......,L in Let L be the predicted probability of the j-th word, and L be the length of the subtitle sequence.

9. The method for generating subtitles for educational videos based on a large model according to claim 8, characterized in that: Multimodal semantic consistency loss function By minimizing the cosine distance between the generated subtitles and the video content in the semantic space, the features of the two are aligned and mapped to a unified semantic space to enhance the semantic relevance between the generated subtitles and the video content. in, For subtitle encoders, For visual encoders; For voice encoders; As a visual feature, These are audio features.

10. The method for generating subtitles for educational videos based on a large model according to claim 9, characterized in that: Readability loss function We employ a language model-based perplexity metric to improve the fluency and readability of subtitles by minimizing the perplexity of the generated subtitles. and Normalization is performed using the Softmax function to ensure that the weights are non-negative and sum to 1, thus achieving adaptive and normalized loss weight allocation. , These are learnable parameters that are trained along with the model network. They automatically obtain the optimal weight allocation through gradient descent and adaptively adjust during training, requiring no manual setting or updating.

Images