A multimodal video summary generation method based on temporal alignment and semantic enhancement
By introducing a dual-scale attention module and a semantic enhancement loss function in the multimodal video digest generation method, the problem of insufficient timing information and semantic alignment in the prior art is solved, and a higher quality video digest generation is achieved.
Patent Information
- Application Number
- CN202411730016.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-28
- Estimated Expiration
- 2044-11-29
AI Technical Summary
The existing multimodal video digest generation method has shortcomings in utilizing timing information and semantic alignment, resulting in insufficient correlation, completeness and accuracy of video digests.
A multimodal video digest generation method based on timing alignment and semantic enhancement is proposed, and a model including feature extraction module, dual-scale attention module and video digest generation module is adopted. The dual-scale attention module aligns and fuses visual features and text features on local and global scales through conditional self-attention networks and collaborative interactions across modal attention networks to generate conditional video representations and global text/video representations. The model uses a backpropagation algorithm for training, considering classification loss, dynamic time regularization loss, and semantic enhanced hinge loss.
By fully utilizing the timing information and semantic information of multimodal video features, the generated video digest has higher correlation, completeness and accuracy, which significantly improves the generation effect of multimodal video digest.