A deep learning-based video joint source-channel coding method, system, device, medium and product

Abstract

The application discloses a kind of based on deep learning video joint source channel coding method, system, equipment, medium and product, it is related to wireless video transmission and deep learning technical field, this method includes: historical reconstruction frame is as time sequence reference, current video frame is input into coding network, generates latent variable of current frame, according to target rate configuration, the latent variable is adaptively filtered by dynamic rate control module, obtains the latent variable to be transmitted, obtains sending signal, receives signal after transmission by additive white gaussian noise channel, realize the joint mapping of source compression and channel transmission, the rough alignment feature is enhanced by using reference refinement network, obtain time sequence reference feature, through main decoder network is jointly modeled, with the weighted sum of rate constraint and reconstruction distortion as optimization goal carries out end-to-end joint training, overcome the sudden quality collapse, the repeated transmission of redundant information is reduced to a minimum, adapts dynamic bandwidth configuration.

Description

Technical Field

[0001] This application relates to the fields of wireless video transmission and deep learning technology, and in particular to a video joint source-channel coding method, system, device, medium and product based on deep learning. Background Technology

[0002] In wireless multimedia transmission scenarios, information typically appears in the form of video sequences. Traditional video compression usually employs a separate "prediction-residual coding" link, designing source compression and channel coding independently. However, under conditions of limited wireless channel capacity and temporal fluctuations, separate source compression and channel coding are more prone to abrupt failures, making it difficult to maintain smooth degradation quality characteristics when the signal-to-noise ratio decreases, i.e., exhibiting a "cliff effect." Furthermore, video frames have significant temporal correlations. Encoding and transmitting each frame independently leads to the repeated transmission of a large amount of temporally redundant information, reducing the overall channel utilization efficiency of the system. Simultaneously, available bandwidth resources in actual wireless transmission are often not fixed. Traditional fixed-structure coding schemes struggle to achieve flexible transmission control under different bandwidth configurations, resulting in insufficient representation of critical information under low bandwidth conditions or wasted channel resources under high bandwidth conditions. Therefore, there is an urgent need for a joint video source-channel coding method that can effectively utilize temporal correlations, adapt to dynamic bandwidth constraints, and maintain smooth degradation characteristics under channel disturbances. Summary of the Invention

[0003] The purpose of this application is to provide a video joint source-channel coding method, system, device, medium and product based on deep learning, so as to solve the problems of traditional split video coding, which is prone to sudden quality collapse, insufficient utilization of timing redundancy and difficulty in adapting to dynamic bandwidth configuration when wireless channel capacity fluctuates.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a deep learning-based video joint source-channel coding method, including: The process involves acquiring the current video frame and historical reconstructed frames, using the historical reconstructed frames as a temporal reference, inputting the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filtering the latent variables through a dynamic rate control module based on the target rate configuration to obtain the latent variables to be transmitted. The encoding network is an end-to-end joint modeling framework that includes a backbone convolutional compression path, a residual downsampling module, and ECA. The dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables. The power is normalized according to the latent variable to be transmitted to obtain the transmitted signal, and the received signal is obtained after being transmitted through the additive white Gaussian noise channel, thus realizing the joint mapping of source compression and channel transmission. Based on the historical reconstructed frames and the received signals, coarse alignment features are obtained through feature domain motion alignment, and the coarse alignment features are enhanced using a reference thinning network to obtain temporal reference features. Based on the received signal and the timing reference features, joint modeling is performed through the master decoder network to obtain the reconstructed frame of the current frame, and end-to-end joint training is performed with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

[0005] Secondly, this application provides a deep learning-based video joint source-channel coding system, comprising: The latent variable determination module is used to acquire the current video frame and historical reconstructed frames, use the historical reconstructed frames as a temporal reference, input the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filter the latent variables through a dynamic rate control module according to the target rate configuration to obtain the latent variables to be transmitted; the encoding network is an end-to-end joint modeling framework including a backbone convolutional compression path, a residual downsampling module, and ECA; the dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables; The joint mapping module is used to perform power normalization based on the latent variable to be transmitted to obtain the transmitted signal, and then obtain the received signal after transmission through the additive white Gaussian noise channel, thereby realizing the joint mapping of source compression and channel transmission. The timing parameter feature determination module is used to obtain coarse alignment features by feature domain motion alignment based on the historical reconstructed frames and the received signals, and to enhance the coarse alignment features using a reference thinning network to obtain timing reference features. The end-to-end joint training module is used to perform joint modeling through the master decoder network based on the received signal and the timing reference features to obtain the reconstructed frame of the current frame, and to perform end-to-end joint training with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

[0006] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described deep learning-based video joint source-channel coding method.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned deep learning-based video joint source-channel coding method.

[0008] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned deep learning-based video joint source-channel coding method.

[0009] According to the specific embodiments provided in this application, this application has the following technical effects: This application acquires the current video frame and historical reconstructed frames to generate latent variables. It then utilizes a dynamic rate control module to adaptively filter based on the target rate configuration, effectively solving the problem of traditional fixed coding structures being unable to adapt to dynamic bandwidth configurations. This achieves flexible transmission control under different channel bandwidth ratios. By normalizing the power of the latent variables and transmitting them directly through an additive white Gaussian noise channel, the dependence on traditional bitstream and entropy coding is avoided. This achieves joint mapping between source compression and channel transmission, enabling the system to automatically learn robust feature representations when facing channel noise disturbances. This overcomes the sudden quality collapse of split links when capacity is insufficient, and achieves video quality that varies with the signal-to-noise ratio. The system achieves smooth degradation; furthermore, it enhances the feature domain motion alignment and reference refinement network by using historical reconstructed frames and received signals to fully exploit and utilize the significant temporal correlation in the video sequence, minimizing the repeated transmission of redundant information, and compensating for the shortcomings of the backbone expression through temporal reference features under low bandwidth or low signal-to-noise ratio configurations; finally, it uses the weighted sum of reconstruction distortion and rate constraints as the end-to-end joint optimization objective, synchronously updating the encoding, decoding, and motion alignment branch parameters, and achieving a coordinated balance between spatial compression, temporal compensation, and channel resource constraints under a unified framework, significantly improving channel utilization efficiency, reconstruction stability, and perception quality in wireless video transmission scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic flowchart of a deep learning-based video joint source-channel coding method is provided in one embodiment of this application; Figure 2 This is a schematic diagram of the DCV-JSCC system framework provided in one embodiment of this application; Figure 3 This is a schematic diagram of the encoder and decoder network structure provided in one embodiment of this application; Figure 4 This is a schematic diagram of a residual block downsampling network structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of a residual block upsampling network structure provided in an embodiment of this application; Figure 6This is a schematic diagram of the network structure of an ECA module provided in an embodiment of this application; Figure 7 This is a schematic diagram of PSNR versus CBR curves under different encoding schemes in the HECV-B dataset provided in an embodiment of this application; Figure 8 This is a schematic diagram of the MS-SSIM variation curves with CBR under different encoding schemes in the HECV-B dataset provided in an embodiment of this application; Figure 9 A schematic diagram illustrating the PSNR versus CBR curves under different encoding schemes in the UVG dataset provided in an embodiment of this application; Figure 10 This is a schematic diagram of the MS-SSIM variation curves with CBR under different encoding schemes in the UVG dataset provided in an embodiment of this application; Figure 11 This is a schematic diagram of the visualization result of the first set of sample reconstructed images provided in an embodiment of this application; Figure 12 This is a schematic diagram of the visualization results of the second set of sample reconstruction images provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] This application, from the perspective of distributed joint source-channel coding, models related images as side information that is "obtainable at the decoding end and requires no additional transmission." Furthermore, in wireless multimedia transmission scenarios, information often appears in the form of video sequences. Compared to static images, video frames exhibit more significant temporal correlation: adjacent frames are typically highly consistent over large background areas, differing only in moving areas or localized changes. Therefore, if temporal correlation modeling and multiplexing mechanisms can be explicitly introduced into the end-to-end JSCC (Joint Source-Channel Coding) framework, it is expected to reduce redundant information transmission and improve overall transmission efficiency and reconstruction stability without increasing additional transmission overhead.

[0015] Traditional video compression typically employs a separate "prediction-residual coding" link, generating prediction frames through motion compensation and encoding the residuals to effectively compress temporal redundancy. However, under conditions of limited and time-varying wireless channel capacity, this separate source compression and channel coding are more prone to abrupt failures, making it difficult to maintain smooth quality degradation as the signal-to-noise ratio decreases. In contrast, end-to-end video JSCC directly learns the mapping relationship from video frames to channel symbols through neural networks, enabling it to adapt more naturally to time-varying channels. Combined with a bandwidth adaptive mechanism, it achieves more flexible transmission control under different rate configurations, thus exhibiting robust transmission characteristics that continuously degrade with changes in channel quality.

[0016] In summary, significant temporal correlations are prevalent between adjacent frames in video sequences. This allows historically reconstructed frames to serve as naturally available reference information for the decoding end, aiding in the reconstruction of the current frame and reducing redundant transmissions. In this sense, the core issue is how to effectively utilize relevant information to improve transmission efficiency and robustness within the end-to-end JSCC framework; the difference lies only in the transformation of this information from spatial to temporal reference. Since cross-frame correlations in video scenes exhibit significant motion structures and are dynamically changing due to occlusion, noise, and other factors, this application employs a structured approach of learning-based alignment and feature domain compensation to stably utilize temporal correlations. Furthermore, a dynamic rate control mechanism enables adaptive transmission under different bandwidth conditions, naturally extending the concept of "correlation-driven selective utilization" to wireless video transmission scenarios.

[0017] like Figure 1 As shown, this application provides a deep learning-based video joint source-channel coding method, including: S1, acquire the current video frame and historical reconstructed frames, use the historical reconstructed frames as a temporal reference, input the current video frame into the encoding network, generate latent variables for the current frame, and adaptively filter the latent variables through the dynamic rate control module according to the target rate configuration to obtain the latent variables to be transmitted; the encoding network is an end-to-end joint modeling framework including a backbone convolutional compression path, a residual downsampling module and ECA (Efficient Channel Attention); the dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables.

[0018] S2, power normalization is performed according to the latent variable to be transmitted to obtain the transmitted signal, and the received signal is obtained after transmission through an additive white Gaussian noise channel, realizing the joint mapping of source compression and channel transmission.

[0019] S3. Based on the historical reconstructed frame and the received signal, coarse alignment features are obtained through feature domain motion alignment, and the coarse alignment features are enhanced using a reference thinning network to obtain temporal reference features.

[0020] S4. Based on the received signal and the timing reference features, joint modeling is performed through the master decoder network to obtain the reconstructed frame of the current frame, and end-to-end joint training is performed with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

[0021] This application uses Group of Pictures (GOP) as the processing unit for consecutive frames to limit the scope of historical reference information. Each frame is processed using a unified structure in the backbone network, and its final transmission and reconstruction method is jointly determined by the availability of historical reconstructed frames and the target rate configuration. The encoding and decoding process of the current frame is modeled as an end-to-end joint mapping conditioned on historical reconstructed frames and the target rate.

[0022] In an exemplary embodiment, S1 specifically includes: S11, with the assistance of the historical reconstruction frame, the current video frame is normalized and input into a hierarchical convolutional structure containing a residual downsampling module. Spatial compression and channel expansion are performed step by step, and ECA is introduced after each convolution to perform adaptive weighting to generate the latent variable.

[0023] S12, Generate a binary or continuous mask according to the target rate configuration, and perform element-wise multiplication operation between the mask and the latent variable to obtain the latent variable to be transmitted.

[0024] In practical applications, let's assume a video sequence. Represented as: (1) in Indicates the first Frame video images. For ease of information theory analysis, this application treats each frame as a random variable sampled from a natural video distribution, with its sample space corresponding to the pixel space. This abstraction method helps to characterize the temporal dependencies in video sequences from a statistical perspective.

[0025] Unlike single-frame images, video data exhibits significant continuity over time. Due to the typically high sampling frequency, scene changes between adjacent frames are often smooth, background areas remain largely consistent across multiple time steps, and object motion usually follows a continuous trajectory. Therefore, statistically, adjacent frames are usually highly correlated. In other words, much of the information in the current frame has already appeared in the previous frame, with only a portion being newly added or changed information.

[0026] Within the framework of information theory, the uncertainty of the current frame can be represented by its entropy. Indicates. When the previous frame is introduced. When used as reference information, the remaining uncertainty of the current frame under the known reference condition is the conditional entropy. According to the fundamental properties of information theory, we have: (2) This inequality stems from the property that conditional entropy does not increase. Its essence is the non-negativity of mutual information; that is, as long as there is a statistical correlation between adjacent frames, using the previous frame as reference information can reduce the uncertainty of the current frame. In natural video scenes, since adjacent frames typically satisfy... Therefore, equation (2) is a strict inequality in most cases. This means that, theoretically, introducing time reference information can reduce the amount of effective information that the current frame needs to express.

[0027] Furthermore, within the joint source-channel coding framework, bandwidth resources are typically constrained by the dimension of latent variables or the actual number of transmitted symbols. Let the latent representation generated by the encoder be... The number of data points that actually participate in transmission after rate control is denoted as... ,in This indicates the target bandwidth configuration or bitrate level. Therefore, during video transmission, the system needs to consider not only "what to transmit" but also "how much to transmit within a given bandwidth." When bandwidth resources are limited, if all spatiotemporal information is still expressed in equal amounts, it can easily lead to insufficient representation of key information. However, if redundant information is compressed by combining temporal correlation and the transmission scale of latent variables is adjusted using a dynamic rate mechanism, it can help improve the efficiency of information expression under limited channel resources.

[0028] Furthermore, in wireless channel environments, the utilization of time correlation also plays a role in stabilizing system performance. If each frame adopts a fixed structure and is independently mapped to channel symbols, the reconstruction quality can easily degrade rapidly under low signal-to-noise ratio or low bandwidth configurations. However, by introducing time reference information, cross-frame redundancy can be used to compensate for the inadequacy of the backbone representation to a certain extent; further combined with a dynamic rate control mechanism, the transmission scale of latent variables can be adaptively adjusted according to the target bandwidth conditions, so that the system maintains relatively stable performance under different bandwidth configurations.

[0029] It should be noted that the above analysis does not rely on video frames satisfying a specific probability distribution, nor does it require the use of traditional explicit prediction-residual modeling methods. It is based solely on the fundamental fact that temporal correlation is prevalent in natural videos and that bandwidth constraints objectively exist in wireless transmission. Therefore, introducing both temporal correlation modeling and dynamic rate control into the joint source-channel coding framework is reasonable from both theoretical and engineering perspectives.

[0030] In summary, the statistical correlation of video sequences in the temporal dimension provides a theoretical basis for reducing redundant transmission, while bandwidth constraints further require the system to have adjustable information expression capabilities. Based on the spatial correlation modeling concept, this application extends the utilization of relevant information to the temporal dimension and combines it with a dynamic rate control mechanism to construct a joint source-channel coding modeling framework for wireless video transmission scenarios, thereby laying the foundation for subsequent conditional modeling, rate control, and decoding reconstruction structure design.

[0031] In practical applications, the utilization of temporal correlation in video joint source-channel coding scenarios typically relies on the introduction of historical reference information. Simultaneously, available bandwidth resources in actual wireless transmission are often not fixed but dynamically adjusted according to system configuration, service requirements, or channel environment changes. Therefore, to stably utilize inter-frame redundancy and adapt to different bandwidth conditions within the end-to-end JSCC framework, this application employs a conditional modeling mechanism based on temporal reference and dynamic rate to uniformly organize video sequences. Specifically, this application still uses GOPs as the processing unit for consecutive frames to limit the scope of historical reference information, but this definition is not equivalent to the strictly separated bitstream structure of I-frames and P-frames in traditional video coding. In this application's framework, each frame is processed using a unified encoding and decoding structure in the backbone network, and its final transmission and reconstruction method is determined by two conditions: whether historical reference frames are available and whether the target rate configuration has changed. In other words, the video joint source-channel coding process no longer depends solely on the current input frame itself but is modeled as an end-to-end mapping process simultaneously constrained by temporal reference conditions and bandwidth rate conditions.

[0032] To ensure consistency between timing prediction and compensation, the encoding end maintains a local reconstruction cache. The input is used as a time reference; the receiving end also constructs a corresponding reference based on the reconstructed result of the previous frame obtained from the decoding, thus sharing a consistent prediction benchmark at both ends and avoiding drift and error accumulation caused by inconsistency in the reference. Regarding rate conditions, this application introduces a rate control mechanism after the encoder outputs the latent variable. Let the target rate configuration be denoted as... It can represent a target CBR (Channel Bandwidth Ratio), a preset code rate level, or a rate control parameter corresponding to a certain bandwidth constraint. For the same input frame, different This will determine the scale of features actually involved in transmission among the latent variables, thus enabling the system to support different bandwidth configurations under a unified model structure. Therefore, the transmission representation of the current frame is not only related to the video content and its historical references, but also closely related to the target rate conditions.

[0033] Based on the above two conditions, the encoding and decoding process of video frames can be uniformly described as: end-to-end joint mapping conditioned on historical reconstructed frames and the target rate. When the historical reference is unavailable or unreliable, the system degenerates into single-frame JSCC modeling; when the historical reference is available, the system performs conditional modeling by introducing preceding reconstructed frames, thereby utilizing temporal correlation to reduce redundant representations. Furthermore, when the target rate... When changes occur, the system adaptively filters the latent variables output by the encoder through the rate control module, so that it can generate a transmission representation that matches the resource conditions under different bandwidth configurations.

[0034] In summary, the conditional modeling mechanism based on temporal reference and dynamic rate expands the joint source-channel coding problem of video from "mapping from a single frame to a single latent variable" to "joint mapping of the current frame under historical reference and rate constraints." This mechanism not only provides a modeling foundation for subsequent learning-based motion alignment, feature domain compensation, and temporal reference enhancement modules, but also lays a unified theoretical framework for the collaborative design of the rate control module and the backbone JSCC path.

[0035] In practical applications, within the joint source-channel coding framework for video, the encoder not only needs to extract the spatial semantic features of a single frame, but also needs to generate a transmittable representation under bandwidth-constrained and channel perturbation conditions, while fully utilizing temporal correlation. Based on the foregoing analysis, this application models the video frame coding process as an end-to-end mapping conditioned on historical reconstructed frames and the target rate. Let the coding function be expressed as: in Indicates the first The latent representation of the channel input corresponding to the frame. For an encoding network with parameter 𝜃, This represents the local reconstruction buffer of the previous frame maintained at the encoder. When the time reference is unavailable or unreliable, the encoding function can degenerate into a single-frame mapping form. When a time reference is available, the model utilizes inter-frame correlations through a conditional modeling mechanism, thereby reducing the repetitive representation of redundant information.

[0036] To accommodate different bandwidth configurations, this application further introduces a rate control module after the encoder output. Let the target rate configuration at the current moment be denoted as... Then the rate control module according to For complete latent variables Selective filtering is performed to obtain the latent variable representations that actually participate in the transmission: (4) in Indicates the target rate The corresponding rate control function, The data to be transmitted after rate adjustment. If a mask control method is used, the process can be further written as: (5) in, This represents a binary or continuous mask determined by the target rate, and ⊙ represents element-wise multiplication. The purpose of this mechanism is to achieve dynamic rate transmission under different CBR conditions by adjusting the number of features effectively participating in transmission among the latent variables, while maintaining a unified network structure.

[0037] In wireless channels, transmitted signals are subject to noise disturbances. Taking an AWGN (Additive White Gaussian Noise) channel as an example, the signal obtained by the receiver can be expressed as: (6) in The decoder reconstructs video frames based on the received signal and time reference information: (7) in This indicates the main decoder network. This is the result of the previous frame that has been reconstructed at the receiving end. This is due to the local reconstruction buffer at the encoding end. Reconstruction results from the receiving end Originating from the same encoding and decoding process, and under consistent system assumptions, the two maintain common origin and consistency, thereby ensuring that the prediction benchmark of the time reference is consistent and avoiding error drift.

[0038] During the training phase, collaborative design of source representation and channel transmission is achieved by jointly optimizing reconstruction distortion and bandwidth constraints. The unified optimization objective can be expressed as: (8) in Indicates the reconstruction distortion term. This represents the rate constraint term, used to characterize the channel resources consumed by the current latent variable under the target rate configuration. The coefficients represent the trade-offs. This joint optimization framework avoids the abrupt performance degradation that may occur in traditional separate source compression and channel coding links, enabling the system to achieve smoother and more stable video quality changes under different time-varying channel conditions.

[0039] Therefore, this application uniformly characterizes joint source-channel coding for video as follows: with the assistance of historical reconstructed frames, time-related modeling is performed on the current frame; under the constraint of the target rate, latent variables are dynamically filtered and transmitted; and finally, a coordinated balance between reconstruction distortion, timing reference utilization, and bandwidth control is achieved through end-to-end joint optimization. This system model provides a theoretical basis for the design of subsequent rate control modules, backbone coding and decoding structures, and timing reference enhancement mechanisms.

[0040] In practical applications, this application analyzes the temporal correlation of video sequences and its theoretical significance in joint source-channel coding from an information theory perspective, pointing out that utilizing historical references can reduce the uncertainty of the current frame and reduce redundant transmission. Based on this, this application combines... Figure 2 The overall structure shown in this application provides a systematic explanation of the DCV-JSCC (Deep Channel-aware Video Joint Source–Channel Coding) framework proposed in this application.

[0041] Overall, DCV-JSCC consists of an end-to-end JSCC encoding / decoding path in the backbone, a dynamic rate module, a temporal correlation enhancement branch (learned motion alignment and feature domain motion compensation), a channel resource constraint module, a wireless channel module, and a local reconstruction buffer mechanism at the encoder end. This application uses a Group of Pictures (GOPs) only as processing units for consecutive frames to limit the scope of historical references; this definition is not equivalent to the strictly separated bitstream structure of I / P frames in traditional video coding. Each frame is processed using a unified structure in the backbone network, while temporal correlation is utilized by constructing reliable cross-frame references in the feature domain through the temporal correlation enhancement branch, and then fused with backbone decoding features to complete reconstruction. This achieves joint utilization of video spatiotemporal redundancy and robust transmission under bandwidth-constrained and channel disturbance conditions.

[0042] like Figure 3 As shown, the encoder hierarchy and downsampling residual module design are as follows.

[0043] Let the input video frame be Before entering the encoding network, the input image is first normalized, mapping pixel values ​​to a uniform distribution range to reduce the impact of statistical differences between different video sequences on training stability. The encoder as a whole adopts a staged hierarchical convolutional structure. The initial layer of the network uses... Convolution extracts shallow texture and edge information and expands the number of channels to 64. Then, a four-level downsampling structure is used. Each level achieves spatial compression through convolution with a stride of 2, while gradually increasing the channel dimension, so that the feature representation gradually transitions from low-level local texture to high-level semantic abstraction.

[0044] Convolution operations are uniformly represented as: ,in Indicates the kernel size. Indicates the number of output channels. The stride represents the step size. In each downsampling stage, the core module is the residual downsampling block (Residualblocks2). This structure consists of three convolutional layers: the first layer is a convolution with a stride of 2, halving the spatial resolution; the second and third layers are convolutions with a stride of 1, used for feature reconstruction and enhancement; Leaky ReLU activation functions are used between each convolutional layer; and GDN (Generalized Divisive Normalization) is introduced at the end for non-linear normalization modeling.

[0045] like Figure 4 As shown, the residual block output format is: in This represents the main branch convolution mapping. This is a downsampling matching map. The mathematical expression for GDN is: (9) This nonlinear form effectively models the statistical dependencies between features, demonstrating superior performance compared to traditional ReLU in image compression and joint source-channel coding tasks. As the network downsamples at each stage, the feature size... Meanwhile, the channel dimension gradually increases. This design realizes a hierarchical representation process of "spatial compression-semantic expansion," enabling the encoder to characterize the high-level semantic structure of video frames at a lower bit rate.

[0046] After each convolutional stage, an ECA (Efficient Channel Attention) module is introduced to adaptively weight the channel features. The ECA module extracts channel statistics through global average pooling. Subsequently, one-dimensional convolution is used to achieve local cross-channel interaction, and the kernel size is adaptively determined by the number of channels: ,in This is the mapping function. This design avoids the fully connected dimensionality reduction operation in traditional SE modules, enhancing the ability to model inter-channel dependencies while maintaining lightweight design.

[0047] The encoder ultimately generates the latent representation: Its spatial dimensions are The number of channels is This latent variable represents the compact semantic information of the current video frame and serves as the input for subsequent dynamic rate control and channel mapping. The system will adaptively filter this complete latent variable according to the target bandwidth configuration to obtain the feature representation that actually participates in the transmission.

[0048] In practical applications, after multi-level convolutional compression and residual feature enhancement by the encoder, the network outputs latent feature representations at its deepest layer. This representation contains the compact encoded information of the current video frame in the semantic space, with a spatial size of [size missing]. The number of channels is In the joint source-channel coding framework, It is no longer used merely as a compressed representation, but as a foundational continuous feature expression for subsequent dynamic rate control and channel mapping.

[0049] To accommodate different bandwidth configurations, this application introduces a dynamic rate control module before the latent variable enters the channel. Let the latent variable after rate control be represented as... Then it can be represented as: (10) This module enables the system to adaptively filter complete latent variables based on different bandwidth configurations, thereby obtaining the feature representations actually involved in transmission. To meet the average power constraints in wireless transmission systems, the latent variables after rate control need to undergo power normalization before entering the channel. Let the vectorized latent variables be denoted as... Its power constraint is expressed as in This represents the average allowable transmit power of the system. To satisfy this constraint, this application adopts a normalized mapping method: (11) This operation ensures that the latent variables of different video frames have a consistent power scale in a statistical sense, thereby improving training stability and avoiding excessive fluctuations in the channel input amplitude.

[0050] In terms of channel modeling, this application uses an additive white Gaussian noise (AWGN) model to simulate the wireless transmission process. ,in noise variance The signal-to-noise ratio (SNR) is determined by the signal-to-noise ratio (SNR). During the training phase, channel noise is explicitly embedded in the forward propagation, allowing the encoder to automatically learn a more robust latent representation to noise disturbances during optimization. Because this framework uses continuous latent variables to directly map the channel, changes in channel conditions will smoothly affect the reconstruction quality, thus avoiding the abrupt quality collapse that may occur in discrete digital transmissions when capacity is insufficient. At the decoding end, the decoder, based on the received characteristics... Reconstructing video frames The system uses reconstruction distortion as the end-to-end optimization objective. (12) in MSE, MS-SSIM, or a combination thereof may be used. Let λ represent the rate constraint term and λ be the tradeoff coefficient. In summary, this application constructs a video latent variable mapping and power constraint mechanism under continuous channel transmission conditions. DCV-JSCC does not rely on bitstream and entropy coding, but rather, under the action of the dynamic rate control module, it achieves bandwidth adjustment under different CBR conditions by adaptively filtering latent variables, and jointly optimizes semantic representation and robustness under AWGN channel disturbances, thereby achieving stable video semantic transmission.

[0051] In practical applications, such as Figure 5 As shown, the decoder structure maintains symmetry with the encoder in the overall hierarchy, but improvements have been made in the specific module design for spatial recovery and detail reconstruction. The receiver obtains the channel output. Then, the received features are first restored to the high-dimensional feature space required by the main decoder network through convolutional mapping. These features originate from the representation of the encoder's latent variables after dynamic rate control and channel transmission; therefore, in the decoding stage, a deep network is needed to gradually restore their spatial structure and semantic details. Subsequently, the four-level upsampling residual module is used to restore the spatial resolution step by step.

[0052] Unlike the downsampled residual blocks used in the encoding stage, the decoding stage employs a residual upsampled structure (Residualblock upsamples2), which contains two branches, as follows: The main branch first generates feature maps through convolution and then uses a pixel shuffle operation to achieve spatial amplification. Pixel shuffle achieves a 2x resolution improvement through channel rearrangement, and its mapping form can be represented as follows: ,in This indicates the magnification factor. Compared to traditional deconvolution, Pixel Shuffle avoids the checkerboard effect problem and improves the smoothness of spatial reconstruction.

[0053] The secondary branch performs feature mapping through parallel convolution and pixel shuffle, adding it to the main branch output to form a residual connection, thereby enhancing the information retention capability of the upsampling stage. Internally, the module still uses the Leaky ReLU activation function; as upsampling progresses, the feature size successively increases from... Ultimately passed Convolution maps back to the three-channel RGB space to obtain the reconstructed frame: .

[0054] In practical applications, the features represented by different channels in deep convolutional networks often have different importance. Especially in joint source-channel coding tasks for video, some channels may carry more critical structural or semantic information, while others are relatively redundant. If all channels are weighted equally, it may reduce the efficiency of latent variable representation, thus affecting transmission performance under constrained channel conditions. After introducing dynamic rate control mechanisms, this difference in channel importance further affects the efficiency of latent variable selection and bandwidth utilization.

[0055] like Figure 6 As shown, to enhance the channel selection capability of the network, this application introduces an ECA (Efficient Channel Attention) mechanism after each level of feature mapping in the encoder and decoder. This module achieves cross-channel dependency modeling through a lightweight structure, improving feature representation capability without significantly increasing the number of parameters. Let the input feature map be... First, perform global average pooling to obtain the channel statistics vector: (13) This vector reflects the response intensity of each channel in the global space. Local cross-channel interaction is then achieved through one-dimensional convolution. ,in It is the Sigmoid activation function. Indicates the kernel size as One-dimensional convolution operation.

[0056] Unlike traditional SE modules that perform dimensionality reduction and expansion through fully connected layers, the ECA module performs local interactions directly on the original channel dimension, avoiding the problem of excessive parameters. (Kernel size) It is not fixed, but rather adaptively determined based on the number of channels: This mapping function ensures that when the number of channels is large, the convolutional kernel can cover a wider channel neighborhood; when the number of channels is small, it avoids over-smoothing. Through this adaptive mechanism, the network can achieve reasonable channel dependency modeling at different layers.

[0057] Finally, the original feature map is adjusted using a channel-by-channel weighted method: This process helps enhance the expression of important semantic features during the encoding stage and helps recover key structural information during the decoding stage. Since the ECA module contains only one-dimensional convolution and a small number of parameters, its computational complexity is much lower than that of SE or self-attention structures, making it suitable for embedding in joint source-channel coding systems.

[0058] From the perspective of channel robustness, the channel attention mechanism can also enhance the response to stable features in low signal-to-noise ratio environments, thereby improving the overall reconstruction quality.

[0059] In an exemplary embodiment, S3, obtaining coarse alignment features based on the historical reconstructed frame and the received signal through feature domain motion alignment, specifically includes: S31, a closed-loop prediction mechanism is adopted to locally reconstruct the noisy reception features of the previous frame using the master decoder network at the encoding end, generate the historical reconstruction frame and maintain the local reconstruction buffer.

[0060] S32, based on the current video frame and the historical reconstructed frame, multi-scale feature matching is performed using the motion estimation module to generate a motion vector field, which is then compressed and transmitted before being restored to the estimated motion field at the decoding end.

[0061] S33, based on the historical reconstructed frame, a historical feature map is obtained by mapping it to the feature space. The estimated motion field is used to spatially sample the historical feature map through bilinear interpolation to obtain the coarse alignment feature.

[0062] In an exemplary embodiment, step S3 involves using a reference thinning network to enhance the coarse alignment features to obtain temporal reference features, specifically including: S34, the coarse alignment features are input into the encoder-decoder structured reference refinement network, and the local distortion caused by spatial sampling is repaired by fusing the residual module and cross-scale features in the reference refinement network, thereby generating the temporal reference features.

[0063] In practical applications, adjacent frames in video sequence transmission scenarios typically exhibit strong temporal correlation. Performing joint source-channel coding independently on each frame would result in the repeated transmission of a large amount of temporal redundancy, thus reducing the overall channel utilization efficiency of the system. Therefore, in addition to the DCV-JSCC backbone encoding and decoding structure, this application introduces a temporal correlation enhancement branch. Through motion compensation and feature domain enhancement mechanisms, it achieves effective utilization of inter-frame correlation. After introducing a dynamic rate control mechanism, this branch can also provide effective time reference information for backbone decoding under low-bandwidth configurations, thereby compensating for the insufficient feature representation capability of the backbone.

[0064] To ensure consistency between motion estimation and time reference, the encoder employs a closed-loop prediction mechanism. After the previous frame completes channel transmission, the encoder uses the same master decoder network as the receiver to locally reconstruct the noisy reception features and buffer the reference frame. This local reconstruction at the encoder is denoted as... The receiving end is rebuilt as Since both ends use the same decoding path and the same channel observations, they are statistically consistent during the training and inference phases, and are therefore uniformly denoted as […]. .

[0065] Let the current input frame be The frame that was reconstructed in the previous moment is At the encoding end, a motion estimation network is first used to model the temporal relationship between the current frame and the reference frame, resulting in a dense motion vector field. ,in Represents pixel-level two-dimensional displacement information. A motion estimation module is constructed for a convolutional neural network. This network employs a multi-scale convolutional structure, achieving joint modeling of large-scale displacement and local detailed motion through layer-by-layer feature matching. To reduce the transmission overhead of motion information, the motion vector field is further compressed into a low-dimensional latent variable representation using a motion encoder. The compressed motion features, after being transmitted through the channel, are reconstructed into an estimated motion field at the decoding end using a motion decoder: .

[0066] In traditional video coding frameworks, motion compensation is typically performed directly in the pixel domain. However, pixel-domain warp operations are prone to introducing occlusion artifacts and spatial discontinuities. To improve the stability and expressiveness of temporal compensation, this application performs motion compensation operations in the feature domain. Specifically, at the decoding end, a feature extraction network first maps the reconstructed image from the previous frame to a high-dimensional feature space: ,in Consistent with the backbone encoder structure, multi-scale semantic features are extracted through multi-layer convolution and downsampling operations. Subsequently, the feature maps are spatially aligned using the estimated motion field. ,in This is a spatial sampling function based on bilinear interpolation. Compared to pixel-domain compensation, feature-domain alignment can achieve smooth mapping in the semantic space, enabling the network to automatically learn robust representations of occluded regions and deformation boundaries to a certain extent.

[0067] However, due to motion estimation errors and the presence of occluded regions, the features after direct warp may exhibit local discontinuities. To further improve the effectiveness of the temporal reference, this application designs a reference refinement network to enhance the coarse-aligned features: ,in Employing an encoder-decoder architecture, internally including a residual module and channel attention mechanism, it repairs distortion caused by warp operations and enhances temporal consistency through cross-scale feature fusion. The refined features... It represents the temporal information highly correlated with the current frame. During the backbone decoding stage, the reference features serve as auxiliary input, participating in the reconstruction process together with the backbone features obtained from channel decoding. It should be noted that the backbone received features here originate from the noisy representation of the encoder output latent variables after dynamic rate control and channel transmission. Let the backbone received features be... Then the reconstruction process of the current frame can be represented as: ,in This represents the master decoder network. Through joint modeling, the decoder can utilize time reference information features to compensate for insufficient backbone representation, thereby improving reconstruction quality, even under conditions of limited channel information or high noise.

[0068] In summary, DCV-JSCC achieves a collaborative design of spatial compression and temporal compensation. This mechanism explicitly utilizes inter-frame redundancy information, allowing the encoder to focus more on the innovative parts of the current frame, effectively reducing redundant transmission, and maintaining good timing stability and reconstruction performance under low signal-to-noise ratio and low bandwidth configurations.

[0069] In an exemplary embodiment, step S4, which involves jointly modeling the received signal and the timing reference features using the main decoder network to obtain the reconstructed frame of the current frame, specifically includes: S41, the received signal is recovered to the high-dimensional feature space required by the main decoder network through convolutional mapping.

[0070] S42, based on the features in the high-dimensional feature space, input the residual upsampling structure in the main decoder network, realize spatial amplification through the pixel shuffle operation in the main branch, and use the sub-branch to perform parallel feature mapping and add it to the main branch to form residual connections, gradually restore the spatial resolution, and obtain the reconstructed frame.

[0071] In an exemplary embodiment, S4, which uses the weighted sum of reconstruction distortion and rate constraint as the optimization objective, specifically includes end-to-end joint training, which includes: S43, During the training phase, the additive white Gaussian noise channel model is explicitly embedded in the forward propagation, enabling the coding network to learn feature representations that are robust to noise disturbances.

[0072] S44, based on the feature representation, calculate the reconstruction distortion term according to the current video frame and the reconstructed frame, and calculate the rate constraint term according to the latent variable to be transmitted. Multiply the reconstruction distortion term and the rate constraint term by a tradeoff coefficient and use the sum as the joint optimization objective, and synchronously update the parameters of the encoding network, the master decoder network and the feature domain motion alignment branch.

[0073] In practical applications, all experiments in this application were conducted on a server platform equipped with an NVIDIA RTX 4090 GPU, and the deep learning framework was implemented using PyTorch. To ensure the stability and reproducibility of the experimental results, a fixed random seed was used during training, including Python, NumPy, and the CUDA random number generator.

[0074] Regarding dataset selection, this study uses the Vimeo-90K dataset for training. This dataset contains approximately 89,800 video clips, each consisting of 7 consecutive frames, covering various motion intensities, scene changes, and texture complexities, effectively supporting video temporal correlation modeling. Let the video clips be represented as: in This represents RGB video frames. This application treats each consecutive 7-frame segment as a complete GOP input unit, enabling the network to learn inter-frame temporal redundancy under a unified structure.

[0075] In the testing phase, the HEVC-B and UVG datasets were used for performance evaluation. Both datasets are standard test sequences commonly used in the field of video compression, covering various video types such as static scenes, translational motion, and complex texture changes, and can comprehensively evaluate the model's generalization ability under different content complexity conditions.

[0076] During the data preprocessing stage, to control computational complexity and enhance the model's spatial generalization ability, all video frames were randomly cropped to 256 during the training phase. The resolution was set to 256, and pixel values ​​were normalized to the [0,1] range. During the testing phase, evaluation was conducted using either center cropping or maintaining the original resolution to ensure experimental fairness.

[0077] For model training, this application employs an end-to-end joint optimization strategy, simultaneously updating the parameters of the encoder, decoder, and feature domain temporal correlation enhancement module. The optimizer uses the Lion algorithm, with an initial learning rate set as follows: An exponential decay strategy is employed for dynamic adjustment. The training batch size is set to 48 to fully utilize the parallel computing power of the GPU, and the epoch is set to 300. For the dynamic rate control module, joint optimization is performed during the training phase using different target rate configurations within a unified model framework. This enables the network to learn latent variable screening and reconstruction mapping relationships under different bandwidth conditions, thereby supporting single-model multi-rate transmission.

[0078] In terms of channel modeling, this application employs an additive white Gaussian noise (AWGN) channel model to simulate the wireless transmission environment. The channel input-output relationship is expressed as follows: Among the noise Signal-to-noise ratio (SNR) definition: in This represents the average power of the transmitted signal. To evaluate the robustness of the model under different channel conditions, experiments were conducted under multiple SNR conditions.

[0079] To measure the efficiency of the model in channel resource utilization, the Channel Bandwidth Ratio (CBR) is introduced as a core metric. CBR is defined as the ratio of the dimension of the transmitted symbols to the dimension of the original image data. A lower CBR indicates less channel resources consumed per unit of source information; therefore, under the premise of ensuring reconstruction quality, a lower CBR represents higher channel utilization efficiency. This application uses a dynamic rate control module to adaptively filter the encoder output latent variables, thereby achieving performance testing under different CBR conditions. Specifically, under different target rate configurations, the system retains different numbers of effective latent variable channels for transmission, thus simulating the video transmission process under different bandwidth resource constraints.

[0080] In terms of performance evaluation, this application uses Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity Index (MS-SSIM) as objective evaluation criteria. PSNR is used to measure pixel-level reconstruction error, and its definition is: (14) in Indicates the maximum value of a pixel. This represents the mean squared error. A higher PSNR value indicates that the reconstructed image is closer to the original image.

[0081] Considering that PSNR only reflects pixel-level differences and is insufficient to fully characterize the quality of human visual perception, this application further adopts MS-SSIM as a perceptual quality evaluation index. To enhance the visualization effect of the comparison between different methods, MS-SSIM is converted into dB form: (15) This format can more intuitively reflect differences in structural similarity. To ensure fairness in the comparison, this application constructs a traditional separate video coding and channel coding combination scheme as a baseline, including combinations of H.264 and H.265 video encoders with different modulation schemes and LDPC channel coding. All comparison methods are tested under the same channel model and SNR conditions, and CBR is uniformly used as the horizontal axis index for performance comparison, thereby systematically analyzing the trade-off between channel efficiency and reconstruction quality of different methods.

[0082] Through the unified setting of the above-mentioned experimental environment configuration, training strategy and evaluation system, this application can systematically evaluate the proposed DCV-JSCC framework from multiple dimensions such as reconstruction quality, channel resource utilization and channel robustness, and provide a reliable basis for subsequent experimental result analysis.

[0083] like Figure 7 As CBR increases, the PSNR of each method shows a monotonically increasing trend, indicating that the reconstruction quality is continuously improved under the condition that channel resources are gradually sufficient.

[0084] In the low to medium CBR range (approximately 0.008–0.018), DCV-JSCC demonstrates a significant advantage over traditional robust transmission schemes employing BPSK+1 / 2LDPC. For example, at a CBR of approximately 0.018, DCV-JSCC achieves a PSNR of approximately 37 dB, an improvement of about 1.2 dB compared to the H.264+BPSK scheme. This result indicates that, under the same channel resource conditions, the joint source-channel modeling approach can more effectively utilize the latent representation space, thereby improving pixel-level reconstruction accuracy.

[0085] Compared to the high spectral efficiency scheme of 16QAM+1 / 4LDPC using H.265, the PSNR of DCV-JSCC is lower in the low CBR region than that of the higher-order modulation scheme. This is because higher-order modulation can provide a higher effective code rate under a fixed SNR, thus providing a larger representation space for the traditional compression encoder. However, as the CBR increases, the performance of DCV-JSCC gradually approaches that of the higher-order traditional scheme. In the high CBR region (above approximately 0.025), the gap between the two narrows significantly, indicating that the expressive power of the proposed model can be fully released when channel resources are sufficient.

[0086] from Figure 8 As shown in the MS-SSIM (dB) metric, DCV-JSCC outperforms the BPSK scheme under most CBR conditions, and the gap with higher-order modulation schemes is significantly smaller than the PSNR gap. For example, in the CBR range of approximately 0.025–0.03, the MS-SSIM (dB) of DCV-JSCC is not much different from that of the H.265+16QAM scheme (approximately 0.2dB). This indicates that the cross-frame reference feature refinement mechanism plays an important role in terms of structure preservation and perceptual quality, giving the model a competitive advantage in visual consistency.

[0087] like Figures 9-10 As shown, the overall trend on the UVG dataset is basically the same as that of HEVC-B, but the performance improvement is more significant. Because the UVG dataset contains higher resolution and more complex textures, it places higher demands on the semantic representation capabilities of the encoder.

[0088] In terms of PSNR, DCV-JSCC significantly outperforms the traditional H.264+16QAM scheme in the low CBR range (0.003–0.005), with an improvement of approximately 1–2 dB. As CBR increases, its performance gradually approaches that of the H.265+BPSK scheme. Under higher CBR conditions, DCV-JSCC's PSNR exceeds 40 dB, indicating that the model has strong reconstruction capabilities for complex video content.

[0089] In terms of MS-SSIM (dB), DCV-JSCC shows more stable performance. With increasing CBR, MS-SSIM (dB) steadily improves, significantly narrowing the gap with higher-order modulation schemes in the mid-to-high bandwidth range, and reaching near-perfect levels in some areas. This phenomenon indicates that, at the perceived quality level, the joint source-channel model can effectively preserve structural information through feature domain time modeling, exhibiting better continuity performance, especially in complex motion regions.

[0090] Note: All experimental results are presented under AWGN channel conditions with SNR=15dB, verifying the relationship between the performance of the DCV-JSCC method and the traditional method in terms of PSNR and MS-SSIM and CBR. The MS-SSIM value is converted to dB using a formula.

[0091] This application provides a deep learning-based video joint source-channel coding system, including: The latent variable determination module is used to acquire the current video frame and historical reconstructed frames, use the historical reconstructed frames as a temporal reference, input the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filter the latent variables through the dynamic rate control module according to the target rate configuration to obtain the latent variables to be transmitted; the encoding network is an end-to-end joint modeling framework including a backbone convolutional compression path, a residual downsampling module and ECA; the dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables.

[0092] The joint mapping module is used to perform power normalization based on the latent variable to be transmitted to obtain the transmitted signal, and then obtain the received signal after transmission through an additive white Gaussian noise channel, thereby realizing the joint mapping of source compression and channel transmission.

[0093] The timing parameter feature determination module is used to obtain coarse alignment features by motion alignment in the feature domain based on the historical reconstructed frames and the received signals, and to enhance the coarse alignment features using a reference thinning network to obtain timing reference features.

[0094] The end-to-end joint training module is used to perform joint modeling through the master decoder network based on the received signal and the timing reference features to obtain the reconstructed frame of the current frame, and to perform end-to-end joint training with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

[0095] To further verify the advantages of the proposed DCV-JSCC method in terms of perceptual quality and to examine its reconstruction performance under different image content conditions, Figures 11-12 The results of local magnification comparison of typical video frames are presented. Figures 11-12 Figures (a)-(e) show the original image, a cropped region of the original image, and the cropped region reconstructed using three schemes: H.264+LDPC, H.265+LDPC, and DCV-JSCC. To ensure fairness in the comparison, the traditional schemes all use 1 / 2 code rate LDPC channel coding and BPSK modulation, and the corresponding PSNR and CBR values ​​are marked in the figures.

[0096] like Figure 11 As shown, the first set of samples represents a portion of the hull (including high-frequency details such as hull edges, rope / structural lines, and water surface textures). In this sample, H.264+LDPC achieves 35.66dB at CBR=0.0171, with the overall structure still discernible. However, some fine lines remain slightly blurred, the hull edges lack a clean transition, and the subtle undulations in the water surface texture show a degree of smoothing. H.265+LDPC reaches 36.13dB at a similar CBR=0.0168, showing a slight improvement in objective indicators, but some loss of high-frequency textures still exists in certain areas, with some fine lines exhibiting a "flattening" trend. In contrast, DCV-JSCC achieves 35.82dB at a lower CBR=0.0166, with PSNR close to traditional schemes, but with more stable local visual performance: the hull outline is clearer, the continuity of slender structures is better, the water surface texture is more natural, and the overall appearance is closer to the original frame. This indicates that, under similar channel resources, end-to-end joint modeling can more effectively preserve structural details that are more sensitive to the human eye.

[0097] like Figure 12As shown, the second set of samples represents a partial view of the eaves (including the eaves outline, tile / wood texture, and structural details such as edge lines). In this sample, the H.264+LDPC (33.22dB, CBR=0.0163) reconstruction result is generally smooth, but the eaves edge lines are somewhat blurred, and texture details (such as tile patterns and fine structural boundaries) are weakened. H.265+LDPC (34.56dB, CBR=0.0161) shows a numerical improvement, and the structural performance is improved, but some details are still soft and the edge sharpness is insufficient. DCV-JSCC achieves 35.16dB (CBR=0.0170) on this sample, which not only has a higher objective index but also a more prominent local visual effect: the eaves boundary is cleaner, the fine texture is clearer, the continuity of structural lines is better, and there are no obvious block artifacts or over-smoothing phenomena, presenting a more natural texture and structural restoration effect.

[0098] Combining the two sets of samples, it can be found that under similar CBR conditions, DCV-JSCC performs more stably in terms of structural continuity, edge sharpness, and texture level preservation. Even in individual samples where the PSNR is close to the optimal traditional scheme, its local visual appearance is still closer to the original frame, indicating that end-to-end joint source-channel modeling can achieve a more reasonable allocation of feature representation under limited channel resources, thereby improving the fidelity of key structural information.

[0099] This application addresses the problem of joint source-channel coding in video. Building upon the image joint coding framework, it extends correlation modeling from the spatial dimension to the temporal dimension, constructing a DCV-JSCC (DeepChannel-aware Video Joint Source–Channel Coding) framework for wireless video transmission scenarios. First, it analyzes the statistical nature of inter-frame temporal correlation from an information theory perspective, pointing out that adjacent frames in natural video sequences possess significant mutual information. Theoretically, it demonstrates that introducing temporal reference information can reduce redundancy and improve channel resource utilization efficiency. To ensure consistency between cross-frame prediction and temporal reference, the system introduces a closed-loop local reconstruction buffer mechanism at the encoding end, ensuring that the reference frame remains consistent at both the encoding and decoding ends, thereby avoiding temporal drift caused by inconsistency in the reference.

[0100] In terms of network architecture design, this application proposes an end-to-end joint modeling framework including a backbone convolutional compression path, a residual downsampling module, and ECA. Dynamic rate control and power normalization mechanisms are introduced in the latent variable mapping stage to simultaneously meet transmission requirements and average transmit power constraints under different bandwidth configurations. AWGN channel perturbations are explicitly embedded in the training process to achieve robust learning and smooth degradation characteristics under channel conditions. At the decoding end, a symmetrical residual upsampling structure and a pixel shuffle spatial recovery mechanism are designed, and cross-frame feature enhancement is achieved through feature domain motion alignment and a reference refinement network, thus realizing joint spatial-temporal modeling in a unified end-to-end optimization. Unlike traditional separate video transmission schemes of "prediction-residual-channel coding," the proposed method does not rely on bitstream and entropy coding. Instead, it adaptively filters latent variables through a dynamic rate control module to control bandwidth overhead under different CBR conditions and directly minimizes reconstruction distortion under channel noise and power constraints. This results in a smoother degradation trend in video quality with changes in signal-to-noise ratio, effectively mitigating the "cliff effect" of traditional schemes when capacity is insufficient.

[0101] Experimental results were systematically validated on the HEVC-B and UVG datasets. The results show that in the low to medium CBR range, DCV-JSCC demonstrates strong competitiveness in PSNR and MS-SSIM metrics compared to some traditional H.264 / H.265+LDPC separate schemes, especially exhibiting better structure preservation and perceptual stability in low-bandwidth and complex texture scenes. Visual comparisons further show that the proposed method presents clearer reconstruction results in character outlines, texture details, and high-frequency structural regions. Even in some scenes where the PSNR value is close to that of traditional schemes, its subjective visual quality is still superior, indicating that end-to-end joint modeling has advantages in feature representation and channel utilization efficiency. Combining theoretical analysis and experimental results, it can be concluded that by explicitly introducing temporal correlation modeling into the joint source-channel coding framework, and combining dynamic rate control, cross-frame reference fusion of the feature domain, and power constraint mechanisms, DCV-JSCC can achieve more flexible bandwidth adaptation and more stable video quality performance under limited channel resource conditions. This work lays the structural and methodological foundation for subsequent extended research in more complex channel environments (such as fading channels), longer time-series modeling spans, and higher-resolution video scenarios.

[0102] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments. The computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device stores data to be processed. The I / O interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with an external terminal via a network connection. When the computer program is executed by the processor, it implements the above-described methods.

[0103] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0104] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0106] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0107] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] This application uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video joint source-channel coding method based on deep learning, characterized in that, include: The process involves acquiring the current video frame and historical reconstructed frames, using the historical reconstructed frames as a temporal reference, inputting the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filtering the latent variables through a dynamic rate control module based on the target rate configuration to obtain the latent variables to be transmitted. The encoding network is an end-to-end joint modeling framework that includes a backbone convolutional compression path, a residual downsampling module, and ECA. The dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables. The power is normalized according to the latent variable to be transmitted to obtain the transmitted signal, and the received signal is obtained after being transmitted through the additive white Gaussian noise channel, thus realizing the joint mapping of source compression and channel transmission. Based on the historical reconstructed frames and the received signals, coarse alignment features are obtained through feature domain motion alignment, and the coarse alignment features are enhanced using a reference thinning network to obtain temporal reference features. Based on the received signal and the timing reference features, joint modeling is performed through the master decoder network to obtain the reconstructed frame of the current frame, and end-to-end joint training is performed with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

2. The video joint source-channel coding method based on deep learning according to claim 1, characterized in that, The process involves acquiring the current video frame and historical reconstructed frames, using the historical reconstructed frames as a timing reference, inputting the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filtering these latent variables using a dynamic rate control module based on the target rate configuration to obtain the latent variables to be transmitted. Specifically, this includes: With the assistance of the historical reconstruction frames, the current video frame is normalized and input into a hierarchical convolutional structure containing a residual downsampling module. Spatial compression and channel expansion are performed step by step, and ECA is introduced after each convolution to perform adaptive weighting and generate the latent variables. Generate a binary or continuous mask according to the target rate configuration, and perform element-wise multiplication of the mask with the latent variable to obtain the latent variable to be transmitted.

3. The video joint source-channel coding method based on deep learning according to claim 1, characterized in that, Based on the historical reconstructed frames and the received signals, coarse alignment features are obtained through motion alignment in the feature domain, specifically including: A closed-loop prediction mechanism is used at the encoding end to locally reconstruct the noisy received features of the previous frame using the master decoder network, generating the historical reconstructed frame and maintaining the local reconstruction buffer. Based on the current video frame and the historical reconstructed frame, multi-scale feature matching is performed using the motion estimation module to generate a motion vector field, which is then compressed and transmitted before being restored to the estimated motion field at the decoding end. The historical feature map is obtained by mapping the historical reconstructed frames to the feature space. The estimated motion field is used to spatially sample the historical feature map through bilinear interpolation to obtain the coarse alignment features.

4. The video joint source-channel coding method based on deep learning according to claim 3, characterized in that, The coarse alignment features are enhanced using a reference thinning network to obtain temporal reference features, specifically including: The coarse alignment features are input into a reference refinement network with an encoder-decoder structure. The residual module in the reference refinement network is fused with cross-scale features to repair the local distortion caused by spatial sampling, thereby generating the temporal reference features.

5. The video joint source-channel coding method based on deep learning according to claim 1, characterized in that, Based on the received signal and the timing reference features, a joint modeling process is performed through the main decoder network to obtain the reconstructed frame of the current frame, specifically including: The received signal is recovered to the high-dimensional feature space required by the main decoder network through convolutional mapping; Based on the features in the high-dimensional feature space, the residual upsampling structure in the main decoder network is input. Spatial magnification is achieved through the Pixel Shuffle operation in the main branch, and the parallel feature mapping in the sub-branch is added to the main branch to form residual connections. The spatial resolution is restored step by step to obtain the reconstructed frame.

6. The video joint source-channel coding method based on deep learning according to claim 1, characterized in that, End-to-end joint training is performed with the weighted sum of reconstruction distortion and rate constraint as the optimization objective, specifically including: During the training phase, the additive white Gaussian noise channel model is explicitly embedded in the forward propagation, enabling the coding network to learn feature representations that are robust to noise disturbances. Based on the feature representation, a reconstruction distortion term is calculated according to the current video frame and the reconstructed frame, and a rate constraint term is calculated according to the latent variable to be transmitted. The sum of the reconstruction distortion term and the rate constraint term multiplied by a tradeoff coefficient is used as the joint optimization objective, and the parameters of the encoding network, the master decoder network, and the feature domain motion alignment branch are updated synchronously.

7. A video joint source-channel coding system based on deep learning, characterized in that, The method for joint source-channel coding of video based on deep learning as described in any one of claims 1-6 includes: The latent variable determination module is used to acquire the current video frame and historical reconstructed frames, use the historical reconstructed frames as a temporal reference, input the current video frame into the encoding network to generate latent variables for the current frame, and adaptively filter the latent variables through a dynamic rate control module according to the target rate configuration to obtain the latent variables to be transmitted; the encoding network is an end-to-end joint modeling framework including a backbone convolutional compression path, a residual downsampling module, and ECA; the dynamic rate control module is used to generate a mask based on the target rate and selectively filter the latent variables; The joint mapping module is used to perform power normalization based on the latent variable to be transmitted to obtain the transmitted signal, and then obtain the received signal after transmission through the additive white Gaussian noise channel, thereby realizing the joint mapping of source compression and channel transmission. The timing parameter feature determination module is used to obtain coarse alignment features by feature domain motion alignment based on the historical reconstructed frames and the received signals, and to enhance the coarse alignment features using a reference thinning network to obtain timing reference features. The end-to-end joint training module is used to perform joint modeling through the master decoder network based on the received signal and the timing reference features to obtain the reconstructed frame of the current frame, and to perform end-to-end joint training with the weighted sum of reconstruction distortion and rate constraint as the optimization objective.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the deep learning-based video joint source-channel coding method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the deep learning-based video joint source-channel coding method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the deep learning-based video joint source-channel coding method as described in any one of claims 1-6.

Images