Multimodal method for real-time high-fidelity video transmission based on semantic stream

Through a multimodal method based on semantic streams, the potential representation of video is extracted and multimodal fusion is carried out, which solves the limitations of the existing technology in long-term time-dependent modeling, perceptual optimization, generative model stability and semantic utilization, and realizes efficient real-time high-fidelity video transmission.

CN119728994AActive Publication Date: 2025-03-28GUANGZHOU INSTITUTE OF TECHNOLOY XIDIAN UNIVERSITY +1
7 Cites 3 Cited by

Patent Information

Application Number
CN202510213570.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-02-26
Publication Date
2025-03-28
Estimated Expiration
2045-02-26

AI Technical Summary

Technical Problem

The existing video compression technology has significant limitations in long-term time-dependent modeling, perceptual optimization, generative model stability and semantic utilization, and it is difficult to meet the needs of real-time high-fidelity video transmission.

Method used

Using a multimodal method based on semantic streams, the potential representation of video is extracted through a space-time compressor, and the semantic translator maps it to the semantic space, the Transformer fusion model performs multimodal fusion, the codebook model performs quantization processing, and a reconstructed video sequence is generated through the video control network.

Benefits of technology

It significantly improves compression efficiency, realizes priority transmission of key semantic information, maintains efficient compression and perceived correlation under bandwidth-constrained conditions, and ensures semantic consistency and temporal coherence of video content.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The embodiment of the invention relates to the technical field of video transmission, and particularly discloses a multi-mode method for real-time high-fidelity video transmission based on a semantic stream. According to the embodiment of the invention, the method comprises the steps: receiving a multi-frame video sequence, extracting spatial and temporal correlation through a space-time compressor, and outputting a potential representation; mapping the potential representation to a semantic space through a semantic translator, and gradually converting visual features and text features; performing multi-modal fusion on the potential representation and the text features through a preset Transform fusion model, and outputting a fusion representation; quantizing the fused representation into a quantized representation through a preset codebook model; and processing the quantized representation and the text features through a video control network to generate a reconstructed video sequence. According to the method, the compression efficiency can be remarkably improved, and priority transmission of key semantic information is realized, so that high-efficiency compression and perceptual correlation are kept under the condition that the bandwidth is limited, and semantic consistency and time coherence of video contents can be ensured.
Need to check novelty before this filing date? Find Prior Art