Video description method and medium based on autonomous optimization alignment of vision and language semantics
By extracting and aligning visual and language features in video title generation, and using modules of factorization and non-factorization for optimization training, the problem of difficult visual and language semantics to be aligned independently is solved, and a more accurate and semantic video title generation is achieved.
CN114880521BActive Publication Date: 2025-04-04JINGGANGSHAN UNIVERSITY
2 Cites 0 Cited by
Patent Information
- Application Number
- CN202210616243.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2042-05-31
AI Technical Summary
Technical Problem
In the prior art, visual and linguistic semantics are difficult to align and complement each other independently, resulting in the accuracy and semanticity of video title generation.
Method used
Multimodal alignment features are obtained by extracting visual and linguistic features and aligning and fusion at each time step. Then, based on these features, a video title generation model is used, including modules based on factorization and non-factorization, for optimization training, to achieve autonomous optimization alignment of vision and language.
Benefits of technology
Improve the accuracy and semanticity of video title generation, and the generated title or description statement is more accurate and semantic.
✦ Generated by Eureka AI based on patent content.
Abstract
The present invention relates to a video description method and medium based on autonomous optimization alignment of visual and language semantics, the method comprising the following steps: extracting visual features and language features, aligning and fusing the visual features and language features at each time step to obtain multimodal alignment features; based on the multimodal alignment features, obtaining vocabulary output at each time step through a video title generation model to obtain a video description result; wherein the video title generation model comprises a VL-AOA based on a factor decomposition method u Modules and non-factored VL‑AOA f module, based on the VL‑AOA u Modules and VL‑AOA f The output of the module and the fusion output of the two modules are subjected to sequence weighted fusion to obtain the vocabulary output at each time step. Compared with the prior art, the present invention has the advantages of high accuracy.
Need to check novelty before this filing date? Find Prior Art