A method and apparatus for generating fine-grained semantic descriptions from action video data

By constructing a fine-grained semantic action description generation model and utilizing multimodal feature fusion and mask reconstruction techniques, the problem of insufficient semantic understanding of action videos in existing technologies is solved, generating high-quality fine-grained semantic descriptions and improving the stability and accuracy of text generation in complex scenarios.

CN120747964BActive Publication Date: 2025-12-05ZHEJIANG UNIV
2 Cites 0 Cited by

Patent Information

Application Number
CN202511225221.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-08-29
Publication Date
2025-12-05
Estimated Expiration
2045-08-29

AI Technical Summary

Technical Problem

Existing technologies struggle to generate semantic descriptions of human actions. Natural language generation methods also struggle to produce text descriptions. Furthermore, existing technologies often fail to generate high-quality text descriptions. They also exhibit superficial understanding of human action semantics, lacking detail, and are particularly prone to instability when processing complex action videos, resulting in ambiguous actions and disordered sentence structure.

Method used

By constructing a fine-grained semantic action description generation model, utilizing isolated word sign language recognition datasets and continuous sign language recognition datasets, and combining pre-trained human pose feature encoders and dynamic feature encoders, action video features and text features are extracted. Multimodal feature fusion and mask reconstruction mechanisms are used to fine-tune the large language model to generate fine-grained semantic descriptions.

Benefits of technology

It achieves stability and accuracy in generating results when processing highly dynamic and complex scenarios such as sign language videos and interactive actions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The application provides a method and device for generating fine-grained semantic description from action video data, based on an isolated word sign language recognition data set and a continuous sign language recognition data set containing word target notes, a training data set is established, fine-grained semantic description modeling action video data and action description text data are obtained, through a training architecture composed of an action video feature encoding module, a multi-modal feature fusion module, and a text feature encoding module, combined with user prompt words and system prompt words and introducing a mask reconstruction mechanism, a fine-grained semantic action description style pre-training generation model is obtained, the action video data is fine-tuned, and a loss function is established, a fine-grained semantic action description generation model is obtained, which is used to generate high-quality fine-grained semantic action description data, and the problem of lack of current fine-grained semantic action description data is alleviated. Ensure the stability and accuracy of the generated results when processing high-dynamic complex scenes such as sign language videos and interactive actions.
Need to check novelty before this filing date? Find Prior Art