Video compression communication method based on spatio-temporal semantic decoupling and generative reconstruction

Abstract

The application relates to a video compression communication method based on video space-time semantic decoupling and generative reconstruction. Frame preprocessing is performed on a to-be-transmitted video to generate a video frame sequence; semantic decoupling is performed on the video frame sequence to extract time features and spatial anchor points; the time features are input into a video description model to generate a text description, and the text description is subjected to semantic arrangement and lossless compression to output time semantic loads; key frames are extracted according to a key frame selection strategy and subjected to image semantic encoding to output spatial semantic loads; the time semantic loads are sent to a receiving end via a reliable communication path; the spatial semantic loads are sent to the receiving end via a semantic communication path; the receiving end performs semantic decoding to recover time semantic information and spatial anchor point information, combines receiving end preference information to construct a generation condition; and the generation condition is input into a generative video model to perform video reconstruction, and a target video with consistent semantics with the to-be-transmitted video is output. The method can significantly reduce the video communication data volume and transmission delay and enhance privacy protection.

Description

Technical Field

[0001] This application relates to the field of video compression communication technology, and in particular to a video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction. It is especially suitable for real-time video transmission in low-bandwidth network environments and can be deployed in mobile terminals, edge devices and wireless communication systems. It belongs to the cross-application category of semantic communication and generative artificial intelligence. Background Technology

[0002] Existing video communication systems are built upon the framework of classical communication theory. Their basic paradigm involves the transmitter encoding and compressing the original video, transmitting the bitstream through a channel, and the receiver decoding and reconstructing the video image. The core objective of this approach is to ensure a high degree of consistency between the received video signal and the transmitter in the pixel domain. When channel bandwidth is ample and the signal-to-noise ratio is good, such systems can achieve relatively good visual reconstruction results. However, their essence remains the transmission of the bit representation of video data itself, rather than the transmission of the information content carried by the video.

[0003] As video resolution evolves towards 4K / 8K and frame rates increase to 60fps and even 120fps, the bandwidth requirements of traditional video transmission methods are growing exponentially. In wireless communication scenarios, weak mobile network environments, and low-bandwidth networks in remote areas, if traditional compression transmission methods continue to be used, resolution, frame rate, or quantization accuracy will usually be significantly reduced. The direct consequences are blurred images, motion blur, significant blockiness, and frequent buffering and stuttering, resulting in a severely degraded user viewing experience.

[0004] The rise of semantic communication technology has provided a new solution to the aforementioned problems. The basic concept of semantic communication is to shift the communication goal from "accurately recovering every bit" to "accurately conveying the meaning of information." The sending end only transmits semantic information relevant to understanding the target, and the receiving end recovers the content based on this semantic information. Currently, semantic communication has achieved some success in tasks such as text transmission, image classification, and visual question answering. However, existing semantic communication methods are mostly task-oriented designs, with the receiving end outputting classification labels, detection results, or question-and-answer answers, but unable to output video content that can be directly viewed by humans.

[0005] Video data possesses unique bimodal attributes. On one hand, video contains spatial structural information, manifested in the appearance of the subject, scene arrangement, target location, and visual composition; on the other hand, video contains temporal evolution information, manifested in the execution process of actions, the sequence of events, the trajectory of state changes, and the laws of motion. Both are indispensable: discarding spatial information results in a loss of visual recognizability, while discarding temporal information reduces it to a static image sequence. Transmitting only highly abstract task semantics cannot meet the human eye's viewing needs in video communication; insisting on transmitting complete pixel data cannot overcome the bandwidth bottleneck in weak network environments.

[0006] In recent years, the rapid development of large language models, visual language models, and diffusion-based generative models has provided new technological possibilities for video communication. However, how to fully utilize the capabilities of multimodal generative models in video communication, maintain the viewability and semantic integrity of video content under extremely low bitrate conditions, and simultaneously consider transmission efficiency, privacy protection, and the personalized needs of the receiving end, has become a pressing technical problem to be solved in the field of video communication technology. Summary of the Invention

[0007] Therefore, it is necessary to provide a video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction to address the aforementioned technical problems. This method aims to solve the technical issues of insufficient transmission efficiency in existing video communication systems under low bandwidth conditions and the inability of semantic communication methods to generate watchable videos. Its core design concept is as follows.

[0008] First, this invention decouples video content from time to time at the semantic level, dividing it into two independent categories of information: temporal features and spatial anchors. Temporal features carry dynamic evolutionary information within the video, including action changes, event sequences, state transitions, and motion processes. This type of information is expressed in natural language, possessing high compression potential and semantic lossless characteristics. Spatial anchors carry static structural information within the video, including the appearance of the main subject, scene layout, target location, and visual composition. This type of information is expressed through keyframe selection and semantic encoding, significantly reducing data volume while maintaining semantic integrity.

[0009] Secondly, addressing the fundamental differences in transmission characteristics between temporal semantics and spatial anchors, this invention designs a dual-path transmission architecture. Temporal semantics utilizes natural language text, which has a small data volume but stringent requirements for semantic accuracy. Any textual error could lead to the receiver generating incorrect actions or events. Therefore, it is transmitted via a reliable communication path configured with lossless compression, channel coding, and error control coding. Spatial anchors utilize latent variables or codeword indexes encoded from image semantics. While the data volume is relatively large, it has a certain tolerance for pixel-level accuracy. The receiver only needs to obtain the approximate semantics of the scene structure and the main subject's appearance to generate visually reasonable video. Therefore, it is transmitted via a semantic communication path configured with image semantic encoding, quantization, and codeword indexing transmission functions, further reducing bandwidth usage through compression rate adjustment.

[0010] Finally, the receiving end does not reconstruct the original video pixel by pixel. Instead, it uses temporal semantic information, spatial anchor information, and receiver preference information as joint conditions to input the generative video model for semantically guided video reconstruction. The generated video maintains consistency with the original video at high-level semantic levels such as actions, events, subjects, and scenes. At the same time, it can achieve personalized output such as style transfer, privacy filtering, resolution adaptation, and frame rate adjustment based on receiver preference information.

[0011] The above technical solution constitutes a new paradigm for video communication based on "semantic understanding—semantic transmission—semantic generation". The invention will be further described below with reference to specific steps.

[0012] The video data to be transmitted is acquired, and frame segmentation and format preprocessing are performed on the video data to be transmitted to generate a video frame sequence.

[0013] The video frame sequence undergoes video understanding and semantic decoupling processing to extract temporal features and spatial anchors. The temporal features characterize action changes, event sequence, state transitions, and temporal dynamics within the video; the spatial anchors characterize the appearance of main subjects, scene layout, target locations, and spatial structure. This decoupling process is accomplished through separate branches for temporal feature extraction and spatial anchor extraction. The former generates subtitles, prompts, and temporal semantic descriptions, while the latter selects keyframes and represents image semantics and spatial structure.

[0014] The temporal features are input into a video description model to generate corresponding subtitles or text descriptions. The video description model is selected from a visual language model or a video question-answering model. The subtitles or text descriptions contain information about the subject's behavior, action changes, event sequence, and state changes in the video. Semantic processing and lossless text compression are performed on the subtitles or text descriptions to output a temporal semantic payload.

[0015] One or more keyframes are extracted from the video frame sequence according to a keyframe selection strategy, and image semantic encoding is performed on the keyframes to output a spatial semantic payload. The determination criteria for the keyframe selection strategy are selected from any one or more of the following: sender intent, receiver intent, task requirements, image saliency, target region, or degree of semantic change. The image semantic encoding adopts any one of the following: autoencoder, vector quantization variational autoencoder, image semantic feature extraction model, or codebook quantization model; the spatial semantic payload is expressed in the form of continuous latent variables, discrete codeword index, or compressed image semantic features.

[0016] The temporal semantic payload is transmitted to the receiving end via a reliable communication path, which is configured with text compression, channel coding, and modulation transmission functions. The text compression is lossless compression, and the channel coding includes error control coding.

[0017] The spatial semantic payload is transmitted to the receiving end via a semantic communication path, which is configured with image semantic encoding, quantization, and codeword index transmission functions. The output of the image semantic encoding is converted into the codeword index after quantization and then transmitted. The semantic communication path is also configured with a compression rate adjustment mode, which is applied to the quantized codeword index.

[0018] At the receiving end, semantic decoding is performed on the temporal semantic payload and the spatial semantic payload respectively to recover the temporal semantic information and the spatial anchor information.

[0019] The generation conditions are constructed based on the temporal semantic information, the spatial anchor point information, and the receiver preference information. These generation conditions include textual conditions, spatial conditions, and personalized control conditions. The receiver preference information is selected from any one or more of the following: visual style, identity consistency, privacy protection level, display resolution, clarity enhancement strategy, frame rate requirements, or interaction method.

[0020] The generation conditions are input into a generative video model to perform video reconstruction, and the output is a target video that is semantically consistent with the video to be transmitted. The generative video model is selected from any one of the following: text-to-video model, image-to-video model, diffusion-based video generation model, conditional video generation model, or multimodal generation model; the generative video model constrains the dynamic changes of the video based on the temporal semantic information and constrains the visual structure of the video based on the spatial anchor point information.

[0021] Secondly, this application also provides a video compression communication device based on video spatiotemporal semantic decoupling and generative reconstruction, characterized in that the device comprises: The video capture module is used to acquire video data to be transmitted. The video understanding module is used to perform video understanding and semantic decoupling processing on the video data to be transmitted, and extract temporal features and spatial anchors; the temporal features are used to characterize the action changes, event sequence, state transitions and temporal dynamics in the video; the spatial anchors are used to characterize the appearance of the main subject, scene layout, target position and spatial structure in the video; The semantic communication module is used to send the temporal semantic payload corresponding to the temporal feature to the receiving end via a reliable communication path, and to send the spatial semantic payload corresponding to the spatial anchor point to the receiving end via a semantic communication path. The video generation module is used to perform semantic decoding on the temporal semantic payload and the spatial semantic payload at the receiving end, recover the temporal semantic information and spatial anchor information, construct generation conditions based on the temporal semantic information, the spatial anchor information and the receiving end preference information, input the generation conditions into the generative video model to perform video reconstruction, and output a target video that is semantically consistent with the video to be transmitted.

[0022] The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction described above has the following advantages compared with existing technologies: First, by decoupling video content into two compact representations—temporal semantics and spatial anchors—direct transmission of raw pixel data is avoided, significantly reducing the amount of data required for video communication.

[0023] Second, through the differentiated design of the dual-path transmission architecture, the temporal semantic path ensures semantic correctness, while the spatial semantic path achieves high compression rate transmission. The two work together to optimize the overall communication efficiency.

[0024] Third, by introducing a generative video model at the receiving end, a generative mapping from semantic conditions to watchable video is realized, enabling the output content of the receiving end to break through the limitation of traditional semantic communication that only outputs task results and has human-eye watchability.

[0025] Fourth, since the original video pixels do not appear during transmission, privacy information such as faces, backgrounds, and locations is not easily intercepted or restored, thus enhancing the privacy and security of communication content.

[0026] Fifth, the introduction of receiver preference information makes the video reconstruction process controllable, and the same semantic payload can generate personalized video content with different styles and resolutions at different receivers. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating the overall system architecture of a video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction in one embodiment. Figure 2 This is a flowchart illustrating the video understanding module in one embodiment; Figure 3 This is a flowchart illustrating the semantic communication module in one embodiment; Figure 4 This is a flowchart illustrating the video generation module in one embodiment; Figure 5 This is a flowchart of the steps of a video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction in one embodiment; Figure 6 This is a schematic diagram comparing the data volume and latency of different transmission schemes in one embodiment; Figure 7 This is a graph showing the speedup results compared to the original video transmission scheme under different codebook sizes and channel conditions in one embodiment; Figure 8 This is a schematic diagram of mobile terminal deployment and application in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] In the embodiments of this application, such as Figure 1As shown, the overall system architecture of this invention comprises three components: a transmitter, a network channel, and a receiver.

[0030] The transmitting end first uses a video acquisition module to acquire the video to be transmitted. The acquired video can be from a real-time camera or a locally stored video file. The acquired raw video then enters a video understanding module, which decomposes the video content into two information streams: a temporal feature branch outputting textual information describing the video's dynamic processes, and a spatial anchor branch outputting image semantic information representing the video's static structure. The temporal feature then enters a reliable communication path, where text compression, channel coding, and modulation operations are performed sequentially; the spatial anchor then enters a semantic communication path, where image semantic coding, quantization, and low-bitrate encapsulation operations are performed sequentially. Both information streams are then transmitted to the receiving end via a physical channel.

[0031] After receiving two signals from the physical channel, the receiving end performs corresponding decoding operations to recover the temporal semantic information and spatial anchor information. Subsequently, these two types of semantic information are fused with the receiving end's local preference information to form a unified generation condition. The video generation module receives this condition, performs video generation inference, and finally outputs a displayable target video.

[0032] Figure 1 The document also illustrates three types of constraints: sender constraints, including task intent, semantic granularity, and privacy level, are used to control the depth and direction of video understanding and semantic decoupling; network-side constraints, including bandwidth, signal-to-noise ratio, codebook size, compression rate, and modulation and coding strategy, are used to control the parameter configuration of dual-path transmission; and receiver constraints, including style preference, identity consistency, privacy protection level, and display requirements, are used to control the personalized direction of video generation.

[0033] In the embodiments of this application, such as Figure 2 As shown, the video understanding module of this invention adopts a dual-branch parallel architecture.

[0034] The input video clips or frame sequences are simultaneously fed into both the temporal feature extraction branch and the spatial anchor extraction branch. In the temporal feature extraction branch, the Captioner component performs temporal semantic understanding on the video frame sequence, generating initial text describing the video's dynamic processes. Subsequently, the LLM semantic processing component performs semantic refinement, redundancy removal, and format normalization on this initial text, outputting refined temporal features. This branch is regulated by constraints such as task intent, semantic granularity, and privacy level.

[0035] In the spatial anchor point extraction branch, the Selector component filters keyframes from the video frame sequence according to a preset selection strategy. After selecting keyframes, the VLM semantic extraction component performs visual semantic analysis on the keyframes, extracts semantic representations at the image level, and outputs spatial anchor points. This branch is also subject to constraints.

[0036] In the embodiments of this application, such as Figure 3 As shown, the semantic communication module of the present invention employs differentiated transmission processing for the two types of semantic payloads.

[0037] The time semantic payload enters the reliable communication path. This path first performs text compression, using lossless compression methods such as Huffman coding, arithmetic coding, or LZ series algorithms to compress the text-based time semantic payload into a compact bit sequence. Next, channel coding is performed, using coding schemes with error control functions such as LDPC codes, Polar codes, or Turbo codes. Finally, modulation transmission is performed, using modulation methods such as QPSK or 16QAM to map the digital signal into an analog waveform suitable for transmission through the physical channel.

[0038] The spatial semantic payload enters the semantic communication path. This path first performs image semantic encoding, which can be achieved by using an autoencoder to map keyframes to continuous latent variables, or by using a vector quantization variational autoencoder to map keyframes to discrete codeword indices. The encoded semantic payload undergoes quantization to further reduce representation precision, and is then encapsulated and transmitted at a low bitrate. This path is also configured with a compression rate adjustment mechanism, which can dynamically adjust the codebook size or latent variable dimension during the encoding stage.

[0039] Figure 3 The constraints include bandwidth, signal-to-noise ratio, codebook size, compression ratio, and modulation and coding strategy. Figure 1 The network-side constraints correspond to these. Based on the current channel state and resource availability, the network side uses the aforementioned parameters to jointly regulate dual-path transmission.

[0040] In the embodiments of this application, such as Figure 4 As shown, the video generation module of the present invention performs a mapping from semantic conditions to the target video.

[0041] The temporal semantic information, spatial anchor information, and receiver preference information recovered by the receiver together constitute the input condition set. The condition construction and semantic alignment submodule first preprocesses the three types of conditions: encapsulating the temporal semantic information into text conditions conforming to the generative model's input format, encoding the spatial anchor information into visual conditions conforming to the generative model's input format, and converting the receiver preference information into control parameters that the generative model can parse. Subsequently, this submodule performs multimodal semantic alignment to ensure that the text conditions, visual conditions, and control parameters are in a consistent embedding representation within the semantic space of the generative model.

[0042] The aligned generation conditions are then fed into the generative video reconstruction submodule. This submodule performs video generation inference based on a diffusion model or an autoregressive model. The generation process is constrained by temporal semantic conditions to ensure the temporal correctness of actions and events, by spatial anchor point conditions to ensure visual consistency of subject appearance and scene structure, and by preference conditions to achieve personalized control over style, resolution, and privacy strategies. The generated results undergo spatiotemporal consistency enhancement and video post-processing to output a semantically consistent target video that meets the requirements of the receiving end.

[0043] Figure 4 The parameters such as the number of generation steps or duration frames, resolution or frame rate, and personalization or privacy policies correspond to the receiver's preference information, which can be set by the receiver user or automatically configured by the system according to the terminal's capabilities.

[0044] In the embodiments of this application, such as Figure 5 As shown, the overall method flow of this invention includes nine stages: video frame acquisition and preprocessing, semantic decoupling to extract temporal features and spatial anchors, temporal semantic payload generation, spatial semantic payload generation, reliable temporal semantic communication, low-bitrate spatial semantic transmission, receiver semantic decoding, generation condition construction, and generative video reconstruction. This flow logically corresponds one-to-one with the technical steps described in the aforementioned invention content section. The order of each stage is not absolutely fixed; some stages can be adjusted in order or executed in parallel according to actual deployment requirements.

[0045] The following specific embodiments illustrate the detailed implementation steps of the video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction of this application.

[0046] Example 1: Video Transmission Experiment Based on UCF101 Dataset This embodiment uses the publicly available video dataset UCF101 as the source of the video to be transmitted. The UCF101 dataset contains videos of human actions in multiple categories, covering a wide range of action and scene changes, making it suitable for verifying the ability to express temporal dynamics and spatial structure in video communication scenarios.

[0047] Human motion videos from the UCF101 dataset were selected as test samples. Decoding, frame sampling, size normalization, and format conversion were performed on the videos to obtain video frame sequences. In the experimental implementation, video frames were uniformly adjusted to 256×256 RGB images, and spatial anchor points could be selected according to the first frame or random frame strategy. This represents a video segment to be transmitted from the UCF101 dataset. Indicates the first Frame video images, This indicates the number of video frames contained in the video.

[0048] The NVIDIA Describe Anything Model (DAM) is used as the video description model to perform temporal semantic understanding on the video frame sequence, generating initial text describing the video content. Subsequently, semantic processing is performed on this initial text, including removing duplicate descriptions, retaining core actions and events, standardizing the grammatical format of prompt words, and compressing redundant vocabulary, ultimately yielding the temporal semantic payload. Taking a "basketball" video as an example, the descriptive text generated by the DAM model, after being organized, is approximately 50 bytes, containing key information such as the subject's identity, the actions performed, and the scene background.

[0049] The first frame is used as the spatial anchor point, that is... The first frame provides crucial visual information such as the initial scene, main subject appearance, background structure, and target location, sufficient to serve as the foundational visual conditions for the receiver's generation model. Image semantic encoding is performed on the first frame: In this embodiment, two encoding methods, AE and VQ-VAE, are compared. The AE encoder uses a ResNet18 encoding backbone with a depth of 4, mapping the 256×256 first frame to a continuous latent variable feature map with 512 channels, and uniformly quantizing it with 2, 3, 4, 6, or 8 bits before transmission. The VQ-VAE encoder uses the same ResNet18 deep 4 encoding backbone, with a main training configuration of 512 latent variable dimensions, 512 codeword dimensions, a codebook size K=256, and a commitment weight of 0.1. In multi-codebook experiments, K=256, 512, 1024, and 2048 are further compared, corresponding to 8, 9, 10, and 11 bits for each codeword index, respectively.

[0050] The temporal semantic payload T is transmitted via a reliable communication path, employs Huffman coding for lossless text compression, LDPC coding for channel coding including error control, and QPSK modulation for transmission. The spatial semantic payload S is transmitted via a semantic communication path; the AE scheme transmits the uniformly quantized 512-channel continuous latent variable features, while the VQ-VAE scheme transmits the discrete codeword index sequence obtained from keyframe coding.

[0051] After receiving the two signals, the receiving end performs corresponding decoding: the time semantic payload is restored to the text prompt. The spatial semantic payload is restored to the semantic representation of the first frame. .

[0052] Receiver settings preference conditions The resolution is set to 320×240, the frame rate to 25fps, the style to natural and realistic, and the privacy protection level to no face blurring.

[0053] Will , and A common input generative video model is used. In this embodiment, EasyAnimate is employed as the generative video model. Temporal semantic cues are also used. After being encoded by the text encoder, it is injected into each layer of the diffusion model through a cross-attention mechanism; spatial anchor points The latent variables are mapped to the initial noise distribution by the VAE decoder, serving as the starting point for the inverse denoising of the diffusion model. The generation process uses the DDIM sampler, with 50 inference steps, producing an output video of 3 seconds in length, 320×240 resolution, and 25fps frame rate. The generated result is denoted as... .

[0054] Using raw video transmission as the baseline method, the raw video frame sequence is transmitted directly, and compared with conventional semantic compression transmission schemes. Data volume statistics only include the payload and do not include physical layer overhead. Latency statistics include the sum of the sender's processing latency, transmission latency, and receiver's processing latency.

[0055] The experimental data are shown in Table 1.

[0056] Table 1 Comparison of Transmission Data Volume and Latency for Different Schemes

[0057] As shown in Table 1, under simulation settings of 256×256 keyframes, 20MHz bandwidth, and SNRs of -10dB, -5dB, and 10dB, the proposed scheme (AE) generates different data volumes and latency depending on the number of quantization bits. For the proposed scheme (VQ-VAE), when K=256, 512, 1024, and 2048, the index load of the image spatial anchor points is approximately 2048, 2304, 2560, and 2816 bits, respectively, and the total load after adding lossless compressed text is approximately 2.67×10^4 to 2.74×10^4 bits. These data demonstrate that the proposed invention can significantly reduce the transmission resources required for video communication while maintaining the semantic content of the video.

[0058] like Figure 6 As shown, the data volume and latency comparison of different transmission schemes are presented in a logarithmic histogram. The horizontal axis represents, in order, the original video transmission, conventional semantic compression, the present invention's scheme (AE), and the present invention's scheme (VQ-VAE). The left vertical axis represents the data volume transmitted, and the right vertical axis represents the transmission latency. The data volume and latency of the present invention's scheme are significantly lower than those of the baseline scheme and the conventional semantic compression scheme.

[0059] like Figure 7As shown in the figure, the speedup ratio of the VQ-VAE scheme varies under different codebook sizes (K) and channel conditions. The horizontal axis represents the codebook size (K), with experimental values ​​including 256, 512, 1024, and 2048. The vertical axis represents the data speedup ratio or transmission delay, with channel conditions corresponding to SNR = -10dB, -5dB, and 10dB. The link rate is calculated based on Shannon capacity with a 20MHz bandwidth. As the K value increases, the codeword indexing capability of the spatial anchors improves, but the number of bits required for a single index increases, leading to a rise in the amount of data transmitted. Conversely, as the SNR increases, the available link rate increases, and the transmission delay decreases. This trend indicates that the network side can dynamically adjust the codebook size based on real-time channel quality, flexibly balancing transmission efficiency and semantic fidelity.

[0060] Example 2: Deploying Applications on Mobile Terminals This embodiment illustrates the deployment method and application process of the present invention on a smartphone.

[0061] like Figure 8 As shown, the transmitting smartphone performs the following operations: It acquires a real-time video stream via its camera, performs frame segmentation and format preprocessing on the video stream; it runs a lightweight video understanding model on the endpoint to generate a temporal semantic description of the video; simultaneously, it runs a keyframe selector to extract keyframes at fixed time intervals or when a scene change is detected; and it performs VQ-VAE encoding on the keyframes to generate a discrete codeword index. The temporal semantic description is compressed using lossless compression methods such as zlib / DEFLATE, and the spatial codeword index is channel-coded before being transmitted at a low code rate via the cellular uplink.

[0062] After receiving the signal from the transmitter, the base station or edge node executes an adaptive channel strategy based on the current downlink bandwidth, signal-to-noise ratio, and receiver terminal capability report: if the downlink quality is good, a higher-order modulation method and a lower compression rate are used to forward the semantic information to the receiver as completely as possible; if the downlink quality deteriorates, the compression rate is increased and switched to a lower-order modulation to prioritize the reliable transmission of the temporal semantic payload and appropriately reduce the accuracy of the spatial anchor point.

[0063] The receiving smartphone performs the following steps: receiving and demodulating the downlink signal, recovering the temporal semantic cue words and spatial anchor codeword indexes respectively; mapping the codeword indexes back to image latent variables through a VQ-VAE decoder, and then reconstructing the first frame image through the decoder; inputting the text cue words and the reconstructed first frame image into a locally deployed lightweight generative video model, and combining it with user-preset preference parameters to generate and display the target video.

[0064] This deployment method is suitable for the following practical scenarios: real-time video calls in weak network environments, where the sending end uploads semantic information at an extremely low bitrate, and the receiving end generates a smooth video; data-saving transmission of short videos on mobile devices, where users only send semantic payloads when uploading videos, and the platform generates the video and distributes it to other users, reducing upload bandwidth consumption; online teaching and remote training, where teachers send semantic descriptions and keyframes of course videos, and students generate video quality adapted to their own network conditions and display devices; mobile monitoring and anomaly reporting, where monitoring terminals only transmit semantic descriptions and keyframes, significantly reducing power consumption and data costs, while avoiding the risk of privacy leaks of monitoring footage; and remote guidance in medical or industrial settings, where visualized remote guidance is achieved through semantic transmission and reconstruction in remote areas lacking high-definition video transmission capabilities.

[0065] Example 3: Dual-path adaptive transmission strategy This embodiment describes the adaptive adjustment mechanism on the network side.

[0066] In one optional implementation, the network side is equipped with a channel monitoring unit to collect parameters such as downlink bandwidth, signal-to-noise ratio, bit error rate, and transmission delay in real time. The receiving end reports its computing power level to the network side when a session is established. The network side dynamically adjusts the text compression method, channel coding method, spatial semantic codebook size, image resolution, compression ratio, or modulation and coding strategy based on any one or more parameters among bandwidth, signal-to-noise ratio, bit error rate, transmission delay, terminal computing power, or network utility target.

[0067] Specifically, for reliable communication paths of time semantic payloads, when the bit error rate exceeds the first threshold, the code rate redundancy of LDPC codes is increased to enhance error correction capability; when the bit error rate is below the second threshold and bandwidth is sufficient, a coding scheme with lower redundancy is switched to improve transmission efficiency.

[0068] For the semantic communication path of spatial semantic payloads, when the bandwidth is below the third threshold, the VQ-VAE codebook size K is reduced from the first value to the second value, or the number of quantization bits for the continuous latent variables of the AE is reduced to decrease the number of transmission bits required for each spatial anchor point. When the bandwidth is above the fourth threshold, the value of K is increased or the number of quantization bits is increased to improve the representation accuracy of the spatial anchor points. When the signal-to-noise ratio (SNR) is above the fifth threshold, the effective link rate can be increased or a higher-order modulation scheme can be used to improve spectral utilization. When the SNR is below the sixth threshold, the modulation order is reduced or coding redundancy is increased to ensure the demodulation of the signal. When the terminal computing power is low, the resolution requirement is reduced in the receiver generation conditions, and the inference steps of the diffusion model are reduced to reduce the computational load.

[0069] The aforementioned adjustment strategy aims to optimize network utility, with the utility function comprehensively considering multiple dimensions such as data speedup, latency, semantic fidelity, and terminal power consumption. Specific strategy parameters can be pre-configured by network operators based on the actual deployment scenario, or optimized online using intelligent algorithms such as reinforcement learning.

[0070] In this embodiment, it should be noted that "semantic consistency" refers to the generated video and the original video maintaining a high-level semantic match, rather than frame-by-frame consistency at the pixel level. In one optional implementation, semantic consistency is evaluated using CLIP text-image similarity scoring. Specifically, keyframes of the original video and corresponding frames of the generated video are input into a CLIP image encoder, and the cosine similarity of their feature vectors is calculated. The average similarity of all frame pairs is used as the score. When the score is not lower than a first preset threshold, semantic consistency is determined. In another optional implementation, when the video content involves human actions, an action recognition model can be used to classify and predict the original video and the generated video separately. If the prediction results are consistent or both belong to the same upper-level category, semantic consistency is determined. The above evaluation methods can be selected and used according to specific application scenarios, and this invention does not limit their application.

[0071] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0072] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0073] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0074] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0076] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction, characterized in that, The method includes: The acquired video data to be transmitted is subjected to frame segmentation and format preprocessing to generate a video frame sequence; The video frame sequence is subjected to video understanding and semantic decoupling processing to extract temporal features and spatial anchors. The temporal features are used to characterize action changes, event sequence, state transitions and temporal dynamics in the video. The spatial anchors are used to characterize the appearance of the main subject, scene layout, target position and spatial structure in the video. The temporal features are input into the video description model to generate corresponding subtitles or text descriptions. Semantic processing and text compression are then performed on the subtitles or text descriptions to output the temporal semantic payload. One or more key frames are extracted from the video frame sequence according to the key frame selection strategy, and image semantic encoding is performed on the key frames to output spatial semantic payload; The time semantic payload is transmitted to the receiving end via a reliable communication path, which is configured with text compression, channel coding and modulation transmission functions. The spatial semantic payload is sent to the receiving end via a semantic communication path, wherein the semantic communication path is configured with image semantic encoding, quantization and codeword index transmission functions; At the receiving end, semantic decoding is performed on the temporal semantic payload and the spatial semantic payload respectively to recover the temporal semantic information and the spatial anchor information; The generation conditions are constructed based on the temporal semantic information, the spatial anchor information, and the receiver preference information. The generation conditions include textual conditions, spatial conditions, and personalized control conditions. The generation conditions are input into a generative video model to perform video reconstruction, and the output is a target video that is semantically consistent with the video to be transmitted.

2. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The video understanding and semantic decoupling process includes a temporal feature extraction branch and a spatial anchor point extraction branch; the temporal feature extraction branch is used to generate subtitles, prompts, and temporal semantic descriptions; the spatial anchor point extraction branch is used to select keyframes, image semantics, and spatial structure representations.

3. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The video description model is selected from any one of Captioner, visual language model or video question answering model; the subtitle or text description contains information on the subject's behavior, action changes, event sequence and state changes in the video.

4. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The criteria for determining the keyframe selection strategy are selected from any one or more of the following: sender's intent, receiver's intent, task requirements, image salience, target area, or degree of semantic change.

5. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The image semantic encoding adopts any one of the following: autoencoder, vector quantization variational autoencoder, image semantic feature extraction model, or codebook quantization model; the spatial semantic payload is expressed in the form of continuous latent variables, discrete codeword index, or compressed image semantic features.

6. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The text compression is lossless compression, and the channel coding includes error control coding.

7. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The output of the image semantic encoding is converted into the codeword index after quantization and then transmitted; the semantic communication path is also configured with a compression rate adjustment mode, which is applied to the quantized codeword index.

8. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The receiver preference information is selected from any one or more of the following: visual style, identity consistency, privacy protection level, display resolution, clarity enhancement strategy, frame rate requirement, or interaction method.

9. The video compression communication method based on video spatiotemporal semantic decoupling and generative reconstruction according to claim 1, characterized in that, The generative video model is selected from any one of the following: text-to-video model, image-to-video model, diffusion-based video generation model, conditional video generation model, or multimodal generation model; the generative video model constrains the dynamic changes of the video based on the temporal semantic information and constrains the visual structure of the video based on the spatial anchor information.

10. A video compression communication device based on video spatiotemporal semantic decoupling and generative reconstruction, characterized in that, The device includes: The video capture module is used to acquire video data to be transmitted. The video understanding module is used to perform video understanding and semantic decoupling processing on the video data to be transmitted, and extract temporal features and spatial anchors; the temporal features are used to characterize the action changes, event sequence, state transitions and temporal dynamics in the video; the spatial anchors are used to characterize the appearance of the main subject, scene layout, target position and spatial structure in the video; The semantic communication module is used to send the temporal semantic payload corresponding to the temporal feature to the receiving end via a reliable communication path, and to send the spatial semantic payload corresponding to the spatial anchor point to the receiving end via a semantic communication path. The video generation module is used to perform semantic decoding on the temporal semantic payload and the spatial semantic payload at the receiving end, recover the temporal semantic information and spatial anchor information, construct generation conditions based on the temporal semantic information, the spatial anchor information and the receiving end preference information, input the generation conditions into the generative video model to perform video reconstruction, and output a target video that is semantically consistent with the video to be transmitted.

Images