Cross-channel distributed video coding method and system based on multi-dimensional attention

Abstract

This disclosure provides a cross-channel distributed video encoding and decoding method and system based on multidimensional attention, which can be applied to the field of video encoding and decoding technology. The method includes: acquiring a group of video images to be transmitted, the group of video images including a first key reference frame, a second key reference frame, and at least one intermediate video frame; encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data, respectively; encoding each intermediate video frame into third encoded data based on a frame encoder; extracting multi-scale features from each third encoded data based on a multidimensional attention mechanism; processing the multi-scale features into fourth encoded data based on a multi-channel feature extraction mechanism; processing at least one fourth encoded data into a first bitstream; and converting the first encoded data and the second encoded data into a second bitstream, so as to transmit the video image group to the decoder via the first bitstream and the second bitstream.

Description

Technical Field

[0001] This disclosure relates to the field of video encoding and decoding technology, and more specifically to a cross-channel distributed video encoding and decoding method and system based on multidimensional attention. Background Technology

[0002] Against the backdrop of rapid growth in digital video content, the demand for efficient and high-quality video transmission capabilities is increasing.

[0003] For example, traditional video compression techniques, such as High Efficiency Video Coding (HEVC) and H.264, have achieved some success in improving compression efficiency. However, when faced with highly complex video content, these techniques still face several challenges, including insufficient video image compression quality, low compression ratio, and artifacts in fast-moving images.

[0004] Video compression schemes based on deep learning technology learn to compress and reconstruct video content through trained deep neural networks. Specifically, a computationally intensive motion compensation prediction loop is applied at the encoder end to explicitly reduce inter-frame temporal redundancy. However, the main computational complexity of the above method comes from the motion-related operations (estimation, compensation, and compression) in the encoding process, while the reconstruction and design at the decoder end are relatively simple. This makes it difficult to apply this method to the encoder end where computational resources are limited.

[0005] Distributed Video Coding (DVC) addresses the deployment challenges of overburdened encoders and resource-constrained platforms, making it particularly important in scenarios with limited encoder resources, such as video surveillance, remote surgery, and multi-view image capture. DVC leverages the temporal correlation between video frames to generate Side Information (SI) at the decoder to aid compression. While DVC has made progress in reducing encoder complexity, its limitations in SI quality make it difficult to accurately capture temporal information between frames. Consequently, its compression efficiency and video quality are generally inferior to predictive coding techniques, failing to meet the ever-increasing demands for high-quality video encoding and decoding.

[0006] In summary, despite the significant progress made in existing video compression technologies over the past few decades, there are still significant technical challenges in achieving high-efficiency, high-quality, and low-latency video communication. Summary of the Invention

[0007] In view of the above problems, this disclosure provides a cross-channel distributed video encoding and decoding method and system based on multidimensional attention.

[0008] According to a first aspect of this disclosure, a cross-channel distributed video coding method based on multidimensional attention is provided, comprising: acquiring a group of video images to be transmitted, wherein the group of video images includes multiple video frames, the multiple video frames including a first key reference frame, a second key reference frame, and at least one intermediate video frame; encoding the first key reference frame and the second key reference frame into first coded data and second coded data, respectively; encoding each intermediate video frame into third coded data based on a frame encoder; extracting multi-scale features from each third coded data based on a multidimensional attention mechanism; processing the multi-scale features into fourth coded data based on a multi-channel feature extraction mechanism; processing at least one fourth coded data into a first bitstream; and converting the first coded data and the second coded data into a second bitstream, so as to transmit the group of video images to a decoder via the first bitstream and the second bitstream.

[0009] According to embodiments of this disclosure, extracting multi-scale features from each third encoded data based on a multi-dimensional attention mechanism includes: inputting the third encoded data into an adaptive pooling layer and outputting multi-scale features, wherein the adaptive pooling layer is constructed based on a multi-dimensional attention mechanism and is used to adjust the size of the pooling window according to the feature density of the third encoded data, and extracting multi-scale features from the third encoded data based on the adjusted pooling window.

[0010] According to embodiments of this disclosure, based on a multi-channel feature extraction mechanism, multi-scale features are processed into fourth encoded data, including: dividing each multi-scale feature into multiple first sub-features based on the number of channels; inputting each first sub-feature into a corresponding convolutional layer to convolve the first sub-features using convolutional kernels corresponding to the convolutional layers to obtain second sub-features, wherein the number of convolutional layers is the same as the number of first sub-features, and the convolutional kernels of the multiple convolutional layers are of different sizes; inputting each second sub-feature into an activation layer to output weight features; fusing the multiple second sub-features into a comprehensive feature based on the weight features corresponding to each second sub-feature; and encoding the comprehensive feature using channel conditional coding technology to obtain fourth encoded data.

[0011] According to embodiments of this disclosure, channel conditional coding technology is used to encode a composite feature to obtain fourth coded data, including: dividing the composite feature into multiple channel features, wherein the multiple channel features have a division order; encoding the current channel feature based on the division order between the multiple channel features and at least one channel feature preceding the current channel feature to obtain a coded feature corresponding to each channel feature; and combining the multiple coded features into fourth coded data.

[0012] According to embodiments of this disclosure, encoding a first key reference frame and a second key reference frame into first encoded data and second encoded data respectively includes: encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively using a super-prior technique based on a dual-space prior encoder; or, encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively using a variational autoencoder.

[0013] A second aspect of this disclosure provides a cross-channel distributed video decoding method based on multidimensional attention, comprising: acquiring a first bitstream and a second bitstream from an encoder; decoding at least one fourth coded data from the first bitstream and decoding first coded data and second coded data from the second bitstream, wherein the at least one fourth coded data uniquely corresponds to at least one intermediate video frame in a video image group, the first coded data corresponds to a first key reference frame in the video image group, and the second coded data corresponds to a second key reference frame in the video image group; determining at least one final reconstructed frame based on the first coded data and the second coded data, wherein the number of final reconstructed frames is the same as the number of fourth coded data; and reconstructing intermediate video frames based on the final reconstructed frames and the fourth coded data.

[0014] According to embodiments of this disclosure, determining at least one final reconstructed frame based on first encoded data and second encoded data includes: using frame interpolation technology, interpolating reconstructed video frames corresponding to the intermediate video frames based on the first encoded data and second encoded data; inputting each reconstructed video frame into a post-processing network, and outputting the final reconstructed frame.

[0015] According to an embodiment of this disclosure, reconstructing an intermediate video frame based on the final reconstructed frame and the fourth encoded data includes: inputting the final reconstructed frame and the fourth encoded data into a frame decoder and outputting an intermediate video frame, wherein the frame decoder has the same structure as the frame encoder at the encoder end.

[0016] A third aspect of this disclosure provides a cross-channel distributed video encoding and decoding system based on multidimensional attention, comprising: an encoder end and a decoder end, wherein the encoder end is used to implement a cross-channel distributed video encoding method based on multidimensional attention, and the decoder end is used to implement a cross-channel distributed video decoding method based on multidimensional attention.

[0017] A fourth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, implement the steps of the multi-channel distributed video encoding or decoding method based on multidimensional attention described above.

[0018] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described cross-channel distributed video encoding or decoding method based on multidimensional attention.

[0019] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the multi-channel distributed video encoding or decoding method based on multidimensional attention described above.

[0020] The embodiments of this disclosure employ a multi-dimensional attention mechanism and a multi-channel feature extraction mechanism during the encoding process of intermediate video frames. This allows for finer-grained attention to different scales and spatial regions within the intermediate video frames, optimizing the encoding of important visual information while reducing the encoding of non-critical information. This enhances the accuracy of intra-frame prediction and improves the feature representation capability before video encoding. Therefore, this method allows the encoding process to focus more on the important parts of visual information, improving compression efficiency while ensuring the possibility of high-quality video reconstruction, effectively solving the limitations of encoder encoding quality and device performance. Attached Figure Description

[0021] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 The illustration depicts an application scenario of a cross-channel distributed video encoding / decoding method based on multidimensional attention according to embodiments of the present disclosure.

[0023] Figure 2 A flowchart illustrating a cross-channel distributed video coding method based on multidimensional attention according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A flowchart illustrating an encoding method at the encoder end according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 4 A flowchart illustrating a cross-channel distributed video decoding method based on multidimensional attention according to an embodiment of the present disclosure is shown.

[0026] Figure 5 A flowchart illustrating a decoding method at the decoder end according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] While related technologies can reduce the operational burden on the encoder through distributed video coding, the transmitted video quality still suffers from defects. This disclosure proposes a multi-dimensional attention-based cross-channel distributed video coding or decoding method to further improve the quality and efficiency of video compression, especially in real-time video communication, thereby reducing latency and enhancing compression efficiency in distributed video coding and decoding applications.

[0032] This disclosure provides a cross-channel distributed video coding method based on multidimensional attention, comprising: acquiring a group of video images to be transmitted, wherein the group of video images includes multiple video frames, the multiple video frames including a first key reference frame, a second key reference frame, and at least one intermediate video frame; encoding the first key reference frame and the second key reference frame into first coded data and second coded data, respectively; encoding each intermediate video frame into third coded data based on a frame encoder; extracting multi-scale features from each third coded data based on a multidimensional attention mechanism; processing the multi-scale features into fourth coded data based on a multi-channel feature extraction mechanism; processing at least one fourth coded data into a first bitstream; and converting the first coded data and the second coded data into a second bitstream, so as to transmit the group of video images to the decoder via the first bitstream and the second bitstream. This method aims to address the shortcomings of existing technologies in maintaining compression efficiency and video quality.

[0033] Figure 1 The illustration depicts an application scenario of a cross-channel distributed video encoding / decoding method based on multidimensional attention according to embodiments of the present disclosure.

[0034] like Figure 1 As shown, in a real-time video transmission scenario, the encoder can encode the video image group 101 to be transmitted into a bitstream 102 and transmit the bitstream 102 to the decoder. The decoder can receive the bitstream 202 in real time and decode it to obtain the video image group 201. In an ideal scenario, the bitstream 202 received by the decoder is the same as the bitstream 102 sent by the encoder. However, in practical applications, due to signal loss during transmission, there are slight differences between bitstream 202 and bitstream 102.

[0035] It should be noted that the multi-channel distributed video coding method based on multi-dimensional attention provided in this disclosure can generally be executed by the encoder end, and correspondingly, the encoding device for implementing the multi-channel distributed video coding method based on multi-dimensional attention is also located at the encoder end. Similarly, the multi-channel distributed video decoding method based on multi-dimensional attention provided in this disclosure can generally be executed by the decoder end, and correspondingly, the decoding device for implementing the multi-channel distributed video decoding method based on multi-dimensional attention is also located at the decoder end.

[0036] In one specific embodiment, the encoder and decoder together form a cross-channel distributed video encoding and decoding system based on multidimensional attention.

[0037] Figure 2 A flowchart illustrating a cross-channel distributed video coding method based on multidimensional attention according to an embodiment of the present disclosure is shown.

[0038] like Figure 2 As shown, the method 200 includes operations S210 to S260.

[0039] In operation S210, a video image group to be transmitted is acquired, wherein the video image group includes multiple video frames, and the multiple video frames include a first key reference frame, a second key reference frame and at least one intermediate video frame.

[0040] In operation S220, the first key reference frame and the second key reference frame are encoded into first encoded data and second encoded data, respectively.

[0041] In operation with S230, each intermediate video frame is encoded into third-order encoded data based on the frame encoder.

[0042] In operation S240, multi-scale features are extracted from each third encoded data based on a multi-dimensional attention mechanism.

[0043] When operating the S250, based on the multi-channel feature extraction mechanism, multi-scale features are processed into fourth-level encoded data.

[0044] In operation S260, at least one fourth encoded data is processed into a first bit stream, and the first encoded data and the second encoded data are converted into a second bit stream so as to transmit a group of video images to the decoder via the first bit stream and the second bit stream.

[0045] According to embodiments of this disclosure, the Group of Pictures (GOP) to be transmitted at the encoder end can be a video composed of multiple video frames arranged in a time sequence. Each video frame in the GOP can be understood as an independent image within the video.

[0046] Each Group of Pictures (GOP) begins and ends with a key reference frame, also known as a keyframe. In this embodiment, the first key reference frame refers to a video frame that begins the GOP, and the second key reference frame refers to a video frame that ends the GOP. Video frames between the first and second key reference frames are called intermediate video frames. For GOPs of different lengths, the number of intermediate video frames in the GOP group can be different.

[0047] In the embodiments of this disclosure, the first key reference frame and the second key reference frame are encoded through one path to obtain first encoded data and second encoded data, respectively; each intermediate video frame is encoded and processed through another path to obtain fourth encoded data to be compressed. By performing different processing on the key reference frame and the intermediate video frames through two paths, compression efficiency can be improved while ensuring transmission quality.

[0048] According to embodiments of this disclosure, when encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively at the encoder end, a variety of encoding algorithms can be adopted; correspondingly, at the decoder end, a decoding algorithm corresponding to the encoding algorithm is adopted for decoding, such as dual-space prior encoding and dual-space prior decoding.

[0049] According to embodiments of this disclosure, in operation S230, each intermediate video frame can first be encoded using a frame encoder to obtain a third encoder. The frame encoder here can be constructed using an existing image encoder. For example, the frame encoder can be a Transformer encoder, and correspondingly, the frame decoder at the decoder end can be a Transformer decoder.

[0050] It should be noted that encoding the intermediate video frames into third-order encoded data is only a basic data processing operation. In order to further ensure video transmission quality and compression efficiency, after obtaining the third-order encoded data, it is necessary to further extract features and compress the third-order encoded data to obtain fourth-order encoded data with smaller data volume and more accurate representation of the important features in the intermediate video frames.

[0051] Specifically, for the third encoded data after encoding each intermediate video frame, features of the third encoded data can be extracted at multiple scales based on a multi-dimensional attention mechanism, and the features at multiple scales can be fused and optimized into multi-scale features. Then, based on a multi-channel feature extraction mechanism, the multi-scale features are processed on multiple channels, so as to minimize the amount of data that needs to be compressed later while ensuring the quality of video transmission, thus obtaining the fourth encoded data.

[0052] According to embodiments of this disclosure, after encoding the first key reference frame and the second key reference frame into first coded data and second coded data, they can also be encoded into a bitstream using entropy coding for transmission. For example, an efficient single-model Variable-bit-rate codec (EVC) technique can be used to entropy-encode the first and second coded data, converting them into a bitstream. Similarly, the fourth coded feature is encoded into a bitstream using entropy coding. For example, an ANS encoder using asymmetrical numeral systems (ANS) can be used to encode the fourth coded feature into a bitstream.

[0053] The embodiments of this disclosure employ a multi-dimensional attention mechanism and a multi-channel feature extraction mechanism during the encoding process of intermediate video frames. This allows for finer-grained attention to different scales and spatial regions within the intermediate video frames, optimizing the encoding of important visual information while reducing the encoding of non-critical information. This enhances the accuracy of intra-frame prediction and improves the feature representation capability before video encoding. Therefore, this method allows the encoding process to focus more on the important parts of visual information, improving compression efficiency while ensuring the possibility of high-quality video reconstruction, effectively solving the limitations of encoder encoding quality and device performance.

[0054] According to embodiments of this disclosure, extracting multi-scale features from each third encoded data based on a multi-dimensional attention mechanism includes: inputting the third encoded data into an adaptive pooling layer and outputting multi-scale features, wherein the adaptive pooling layer is constructed based on a multi-dimensional attention mechanism and is used to adjust the size of the pooling window according to the feature density of the third encoded data, and extracting multi-scale features from the third encoded data based on the adjusted pooling window.

[0055] According to embodiments of this disclosure, the adaptive pooling layer can dynamically adjust the size of the pooling window and perform pooling operations for different inputs. For example, for feature-dense regions, the size of the pooling window can be automatically adjusted, allowing the feature extraction operation to focus on details in small regions or expand to larger regions to capture broader contextual information. In this way, features at different scales can be effectively captured and utilized.

[0056] According to embodiments of this disclosure, a feature-dense region can refer to an area where pixel values ​​vary significantly. For example, in image A, the upper half is the sky, and the lower half is a dense flow of traffic. Therefore, the pixel values ​​related to the sky in the upper half of image A vary less, while in the lower half of the traffic flow, the pixel values ​​at the edges of each vehicle vary more. Compared to the upper half, the lower half is a feature-dense region.

[0057] In the embodiments of this disclosure, by adopting adaptive pooling operations, the size of the pooling window can be dynamically adjusted to adapt to scale changes in different regions of the video frame, ensuring the flexibility and accuracy of feature extraction.

[0058] According to embodiments of this disclosure, based on a multi-channel feature extraction mechanism, multi-scale features are processed into fourth encoded data, including: dividing each multi-scale feature into multiple first sub-features based on the number of channels; inputting each first sub-feature into a corresponding convolutional layer to convolve the first sub-features using convolutional kernels corresponding to the convolutional layers to obtain second sub-features, wherein the number of convolutional layers is the same as the number of first sub-features, and the convolutional kernels of the multiple convolutional layers are of different sizes; inputting each second sub-feature into an activation layer to output weight features; fusing the multiple second sub-features into a comprehensive feature based on the weight features corresponding to each second sub-feature; and encoding the comprehensive feature using channel conditional coding technology to obtain fourth encoded data.

[0059] In embodiments of this disclosure, multi-scale features can be divided into multiple first sub-features based on a predetermined number of channels. For example, multi-scale features can be divided into three equal first sub-features. Since the multi-scale features in the embodiments of this disclosure fuse image features from different spaces and scales, when further extracting features from the multi-scale features, convolutional kernels of different sizes can be used to perform targeted learning of the first sub-features at different scales, which reduces the number of parameters and enhances the learning ability for features from different channels.

[0060] Specifically, after each first sub-feature is input into the corresponding convolutional layer, convolution operations of different scales are performed within each convolutional layer using the convolutional kernel of that layer to obtain the second sub-feature based on the convolution of each convolutional layer. For example, three convolutional layers can be set up, using 1*1, 3*3, and 5*5 convolutional kernels respectively to perform convolution operations on the first sub-feature.

[0061] Multiple second sub-features after passing through different convolutional layers will exhibit scale differences due to variations in kernel size. To quantify this difference, the second sub-features can be input into an activation layer, outputting a third sub-feature. Each second sub-feature can be input into its corresponding activation layer, outputting a weighted feature.

[0062] In embodiments of this disclosure, the activation layer can be constructed based on the softmax function. By calculating softmax weights, different importance is assigned to features in different regions and at different scales, thereby further optimizing the representational power of the features. Thus, these weighted features reflect the importance of each feature in the final representation.

[0063] According to embodiments of this disclosure, after determining multiple weighted features, multiple second sub-features can be fused into a comprehensive feature based on these weighted features. Since the comprehensive feature is a comprehensive feature representation formed by grouping and convolving multi-scale features and then fusing them, it can more accurately capture important visual information within a video frame. Through this weighting mechanism, the model can emphasize features that are more critical to the final video compression and reconstruction, while suppressing more redundant information.

[0064] According to embodiments of this disclosure, after the feature extraction operation is completed, the comprehensive features can be directly encoded using channel conditional coding technology to obtain the fourth encoded data.

[0065] The embodiments of this disclosure apply grouped convolutional operations in the feature extraction stage after the adaptive pooling layer. Each convolutional layer contains a certain number of channels and performs convolution operations independently. This not only reduces the computational load but also allows the network to explore the feature correlations and differences between channels more deeply while maintaining low complexity. Subsequently, the grouped convolutional operation not only enhances the learning ability of different channel features but also further reduces the number of parameters, which helps to reduce encoding latency in real-time video transmission scenarios.

[0066] According to embodiments of this disclosure, channel conditional coding technology is used to encode a composite feature to obtain fourth coded data, including: dividing the composite feature into multiple channel features, wherein the multiple channel features have a division order; encoding the current channel feature based on the division order between the multiple channel features and at least one channel feature preceding the current channel feature to obtain a coded feature corresponding to each channel feature; and combining the multiple coded features into fourth coded data.

[0067] In the embodiments of this disclosure, the comprehensive feature can be divided into multiple channel features according to a preset segmentation channel. It should be noted that the number of channel features here can be the same as or different from the number of convolutional layers and channels in the aforementioned grouped convolution. For example, the comprehensive feature can be divided into 8 patches.

[0068] To further reduce data redundancy, after dividing the comprehensive feature into multiple channel features, information from all channel features preceding the current channel feature is used to reduce data redundancy during the compression of the current channel feature. These preceding channel features can also be referred to as the pre-current channel features of the current channel feature.

[0069] The channel conditional coding technique disclosed herein is an extraction of latent representations based on the entropy principle; that is, the channel conditional coding technique can be used as an entropy model. When encoding the current channel feature, the difference information between the current channel feature and each preceding channel feature can be calculated, and the statistical difference information can be obtained by combining each difference information. Then, the current channel feature can be encoded using this statistical difference information.

[0070] For example, the mean and variance of the current channel features and the previous channel features can be used as difference information. Statistical difference information can be obtained by using a parameterized probability model to obtain the probability distribution of the current channel features.

[0071] It should be noted that, since the channel conditionalization variation technique disclosed herein requires the use of preceding channel features, the first channel feature can be encoded independently.

[0072] The embodiments of this disclosure employ channel conditional coding technology to slice the comprehensive features and process each channel feature step by step (i.e., channel slices). The encoding of each channel feature depends on the statistical information of all preceding channel features. A parameterized probability model is used to estimate the probability distribution of the current channel feature, thereby achieving efficient entropy coding and reducing data redundancy in the video sequence.

[0073] At this point, the encoder has completed the encoding operation for each intermediate video frame, obtaining the corresponding fourth encoded data.

[0074] According to embodiments of this disclosure, encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively includes: encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively using a super-prior technique based on a dual-space prior encoder; or, encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively using a variational autoencoder.

[0075] In one specific embodiment, the dual-space prior encoder is used to encode and reconstruct the first key reference frame and the second key reference frame, and the dual-space prior encoder operates the same on the first key reference frame and the second key reference frame. The following will describe the first key reference frame as an example.

[0076] The dual-space prior encoder mainly consists of two steps: the first step is frame segmentation and preliminary coding; the second step is reconstruction and predictive coding.

[0077] After inputting the first key reference frame into the dual-spatial prior encoder, the first key reference frame can be segmented into two spatial information, P1 and P2. In the first step, hyperprior information is extracted from P1 using a hyperprior technique, and then P1 is initially encoded based on the hyperprior information. The hyperprior information is information extracted from higher-level data features to guide the encoding process of P1. For example, a hyperprior model can be used to generate side information related to P1, and this side information can be used as the hyperprior information.

[0078] In the second step, an approximate reconstruction of P1 is first built based on the information encoded in P1. Then, this reconstruction information is used as conditional information to predictively encode P2. The reconstructed P1 is used to more accurately estimate P2, thereby reducing data redundancy when encoding P2.

[0079] The EVC technology used in the embodiments of this disclosure can optimize distortion performance through deep convolutional networks and significantly reduce the amount of data.

[0080] Furthermore, after encoding the key reference frame using a dual-space prior encoder, data can be compressed using EVC technology, and the first and second encoded data can be converted into a first bit stream through entropy encoding.

[0081] In another specific embodiment, a variational auto-encoder (VAE) can also be used to optimize the encoding of key reference frames.

[0082] Alternatively, Generative Adversarial Networks (GANs) can be used to generate and reconstruct the frame content.

[0083] In the entropy coding process following the dual-space prior encoder, a Latent Residual Prediction (LRP) mechanism is introduced to compensate for errors in the quantization coding loss. Specifically, during the coding process, the residual of each channel slice is predicted based on a Gaussian conditional model, and then the latent representation of the quantization is adjusted, thereby reducing reconstruction errors and improving the quality of the reconstructed frame. When coding key reference frames, the channel slice operation method is similar to the operation using channel conditional coding techniques described above, and will not be repeated here.

[0084] According to embodiments of this disclosure, the channel conditional coding process for intermediate reference frames can be performed using either an autoregressive model or a discrete cosine transform (DCT).

[0085] Figure 3 A flowchart illustrating an encoding method at the encoder end according to an embodiment of the present disclosure is shown schematically.

[0086] like Figure 3 As shown, for the first key reference frame and the second key reference frame in the video image group, a dual-space prior encoder encodes the first key reference frame and the second key reference frame into first encoded data and second encoded data, respectively. Then, a bitstream coding system uses EVC technology to entropy encode the first encoded data and the second encoded data to obtain the second bitstream.

[0087] For each intermediate video frame, the frame encoder first encodes the intermediate video frame into third-code data, and then the third-code data is input into the multi-scale cross-channel feature integration module. The multi-dimensional attention mechanism (also known as the multi-scale attention mechanism) in the multi-scale cross-channel feature integration module pays fine-grained attention to different scales and spatial regions within the video frame, thereby obtaining multi-scale features.

[0088] Then, the multi-scale features are divided into multiple first sub-features, and a multi-channel feature extraction mechanism is used for group convolution, weight extraction, feature fusion, and channel conditional coding to obtain the fourth encoded data for each intermediate video frame. Finally, the fourth encoded data is converted into a first bitstream using a bitstream encoding system and an ANS encoder.

[0089] In the embodiments of this disclosure, the multi-scale cross-channel feature integration module significantly improves the efficiency of video compression and reconstruction quality, especially in video sequences with rich details or large scene variations. This is due to its ability to effectively reduce the encoding of non-critical information while enhancing feature representation power, thus achieving an excellent balance between compression efficiency and video quality.

[0090] Furthermore, the dual-space prior encoder disclosed herein can be integrated into a keyframe dual-space prior synthesis module, employing innovative segmentation techniques and parameterized probabilistic models during the encoding process. Based on channel conditionalization, this module progressively processes each channel slice in the video data, effectively utilizing the temporal correlation of the video sequence. Through serialization, the encoding process of each slice depends on information from previous slices; this dependency significantly reduces redundant encoding and optimizes overall encoding efficiency. In addition, this module utilizes efficient variable bit rate (EVC) coding and decoding techniques and a latent residual prediction (LRP) mechanism to further optimize rate-distortion performance, reduce errors during quantization, and improve the quality of reconstructed frames.

[0091] Figure 4 A flowchart illustrating a cross-channel distributed video decoding method based on multidimensional attention according to an embodiment of the present disclosure is shown.

[0092] like Figure 4 As shown, the method 400 includes operations S410 to S440.

[0093] In operation S410, the first bit stream and the second bit stream from the encoder are acquired.

[0094] In operation S420, at least one fourth encoded data is decoded from the first bitstream, and first encoded data and second encoded data are decoded from the second bitstream, wherein at least one fourth encoded data uniquely corresponds to at least one intermediate video frame in the video image group, the first encoded data corresponds to the first key reference frame in the video image group, and the second encoded data corresponds to the second key reference frame in the video image group.

[0095] In operation S430, at least one final reconstructed frame is determined based on the first encoded data and the second encoded data, wherein the number of final reconstructed frames is the same as the number of fourth encoded data.

[0096] In operation, the S440 reconstructs intermediate video frames based on the final reconstructed frame and the fourth encoded data.

[0097] According to embodiments of this disclosure, the first bitstream and the second bitstream acquired by operation S410 can be the first bitstream and the second bitstream generated by operations S210 to S260. Corresponding to the encoder end, after acquiring the first bitstream and the second bitstream, the fourth encoded data, the first encoded data, and the second encoded data can be obtained based on the ANS system.

[0098] For example, during the decoding process, the first bitstream is decoded in the ANS entropy decoder based on the probability parameters provided by the channel conditional coding technique to obtain the fourth encoded data; for the second bitstream, the second bitstream is entropy decoded using the EVC technique to obtain the first encoded data and the second encoded data, which helps to reconstruct high-quality video frames.

[0099] In the embodiments of this disclosure, since different encoding operations are performed on the key reference frame and the intermediate reference frame, the decoding end can not only directly decode the fourth encoded data corresponding to the intermediate reference frame, but also reconstruct the final reconstructed frame related to each intermediate video frame based on the first key reference frame and the second key reference frame. Thus, the intermediate video frame is obtained by fusing the final reconstructed frame and the fourth encoded data.

[0100] Because the embodiments of this disclosure use probability parameters provided by the channel conditionalization method to decode in the entropy decoder during the decoding process, the decoding process can extract important features of the encoding compression, thereby improving compression efficiency while ensuring high-quality video reconstruction and effectively solving the problems of encoder encoding quality and device performance limitations.

[0101] According to embodiments of this disclosure, determining at least one final reconstructed frame based on first encoded data and second encoded data includes: using frame interpolation technology, interpolating reconstructed video frames corresponding to the intermediate video frames based on the first encoded data and second encoded data; inputting each reconstructed video frame into a post-processing network, and outputting the final reconstructed frame.

[0102] According to embodiments of this disclosure, since the first and second encoded data are related to two key reference frames at the beginning and end of the video image group, a reconstructed video frame corresponding to each intermediate video frame can be interpolated using the Rapid Frame Interpolation (RIFE) algorithm, wherein the RIFE algorithm is implemented based on the Fast Fourier Transform.

[0103] After interpolating the reconstructed video frames corresponding to each intermediate video frame, the edge information of the reconstructed video frames can be obtained through a post-processing network (Side Refine Net) to enhance visual details and texture and optimize the video playback experience.

[0104] According to an embodiment of this disclosure, reconstructing an intermediate video frame based on the final reconstructed frame and the fourth encoded data includes: inputting the final reconstructed frame and the fourth encoded data into a frame decoder and outputting an intermediate video frame, wherein the frame decoder has the same structure as the frame encoder at the encoder end.

[0105] Figure 5 A flowchart illustrating a decoding method at the decoder end according to an embodiment of the present disclosure is shown schematically.

[0106] like Figure 5 As shown, after the decoder acquires the second bitstream, it uses a bitstream decoding system to perform bitstream conversion based on EVC technology. The bitstream then passes sequentially through entropy decoding and the dual-space prior decoder in the keyframe dual-space prior synthesis module to obtain the first and second encoded data. Subsequently, based on the frame interpolation module, the reconstructed video frames are obtained from the first and second encoded data. Finally, a post-processing network processes each reconstructed video frame into the final reconstructed frame.

[0107] After the decoder acquires the first bitstream, it passes through the bitstream decoding system and is decoded using the ANS decoder to obtain the fourth encoded data. For each intermediate video frame, the final reconstructed frame and the decoded fourth encoded data are input into the frame decoder to jointly decode the intermediate video frame.

[0108] The keyframe dual-space prior synthesis module not only effectively improves the efficiency of entropy coding and reduces the bit rate required for coding, but also lowers coding latency, meeting the needs of real-time video communication. Especially when processing dynamic scenes and high dynamic range (HDR) video, this module demonstrates superior performance. By effectively reducing redundant encoding of information and optimizing rate-distortion performance, this technology maintains good video quality at low bit rates, overcoming the traditional compression method's challenge of balancing high efficiency and high quality.

[0109] To better understand encoding and decoding methods, the following example will be used, where a video image group consists of N video frames, where x1 and x... N The first key reference frame and the second key reference frame are represented respectively, x i Represents intermediate video frames.

[0110] For the first key reference frame x1 and the second key reference frame x N :

[0111] At the encoder end, x1 and x N The input keyframe dual-space prior synthesis module processes the data. Within this module, x1 and x2 are first synthesized using a dual-space prior encoder. N Encode them respectively as the first encoded data Second encoded data Then, the first bit stream is obtained through entropy encoding and bit stream conversion.

[0112] At the decoder end, the first encoded data is obtained through bitstream conversion and entropy decoding in sequence. Second encoded data Subsequently, the first encoded data... Second encoded data Perform fast frame interpolation to obtain each intermediate video frame x i Reconstructed video frames Next, The input is fed into a side-refine network for processing to obtain edge information, which is the latent representation corresponding to the current intermediate video frame. That is, the final reconstructed frame .

[0113] For the intermediate video frame x i At the encoder end, the intermediate video frames x are first processed by a frame encoder (M-Encoder). i Perform first-level encoding to obtain third-level encoded data y i That is, the intermediate video frame x i The potential representation of the third encoded data y. i The input is processed by a multi-scale cross-channel feature integration module. First, it sequentially undergoes adaptive pooling, grouped convolution, softmax weight calculation, feature fusion, and channel conditional encoding to obtain the fourth encoded data. Secondly, the fourth encoded data Inputting into the ANS system yields the second bitstream.

[0114] At the decoder end, the fourth encoded data is obtained again through reconstruction. Then, the fourth encoded data... and the final reconstructed frame obtained by interpolating frames based on the first and second encoded data. The common input frame decoder (M-Dncoder) reconstructs the intermediate video frames. .

[0115] In summary, the encoding and decoding method disclosed herein not only has significant advantages in compression efficiency but also performs exceptionally well in terms of processing complexity and video quality preservation. It is suitable for a variety of application scenarios, especially real-time video communication scenarios with stringent requirements for latency and compression efficiency. Its core competitiveness lies in application scenarios where encoder performance is limited, such as distance education, real-time monitoring, and telemedicine.

[0116] The embodiments of this disclosure also provide a cross-channel distributed video encoding and decoding system based on multidimensional attention, including: an encoder end and a decoder end, wherein the encoder end is used to implement the above-mentioned cross-channel distributed video encoding method based on multidimensional attention, and the decoder end is used to implement the above-mentioned cross-channel distributed video decoding method based on multidimensional attention.

[0117] Embodiments of this disclosure also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, implement the steps of the multi-channel distributed video encoding or decoding method based on multidimensional attention described above.

[0118] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described cross-channel distributed video encoding or decoding method based on multidimensional attention.

[0119] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0121] When the computer program is executed by a processor, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0122] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices or magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed as signals over a network medium, and downloaded and installed via a communication component, and / or installed from a removable medium. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0123] In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0124] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0127] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this disclosure. It should be understood that the above descriptions are merely specific embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A cross-channel distributed video coding method based on multidimensional attention, comprising: Acquire a group of video images to be transmitted, wherein the group of video images includes multiple video frames, and the multiple video frames include a first key reference frame, a second key reference frame and at least one intermediate video frame; The first key reference frame and the second key reference frame are respectively encoded into first encoded data and second encoded data; Based on the frame encoder, each intermediate video frame is encoded into third encoded data; Multi-scale features are extracted from each of the third encoded data based on a multi-dimensional attention mechanism; Based on the number of channels of the multi-scale feature, the multi-scale feature is divided into multiple first sub-features; each first sub-feature is input into a corresponding convolutional layer so that the first sub-feature is convolved with the convolutional kernel corresponding to the convolutional layer to obtain a second sub-feature, wherein the number of convolutional layers is the same as the number of first sub-features, and the size of the convolutional kernels of the multiple convolutional layers is different; each second sub-feature is input into an activation layer to output a weight feature; based on the weight feature corresponding to each second sub-feature, the multiple second sub-features are fused into a comprehensive feature; The comprehensive feature is divided into multiple channel features, wherein the multiple channel features have a division order; Based on the segmentation order among the multiple channel features and at least one channel feature preceding the current channel feature, the current channel feature is encoded to obtain encoded features corresponding to each channel feature; the multiple encoded features are combined into fourth encoded data; wherein, when encoding the current channel feature, the difference information between the current channel feature and each preceding channel feature is calculated, and each difference information is combined to obtain statistical difference information, and then the current channel feature is encoded using the statistical difference information, wherein the difference information is the mean and variance of the current channel feature and the preceding channel feature, and the statistical difference information is the probability distribution of the current channel feature obtained through a parameterized probability model; At least one of the fourth encoded data is processed into a first bitstream, and the first encoded data and the second encoded data are converted into a second bitstream so as to transmit the video image group to the decoder via the first bitstream and the second bitstream.

2. The method according to claim 1, wherein, The multi-scale features extracted from each of the third encoded data based on the multi-dimensional attention mechanism include: The third encoded data is input into an adaptive pooling layer, which outputs the multi-scale features. The adaptive pooling layer is constructed based on a multi-dimensional attention mechanism and is used to adjust the size of the pooling window according to the feature density of the third encoded data, and extract the multi-scale features from the third encoded data based on the adjusted pooling window.

3. The method according to claim 1, wherein, The step of encoding the first key reference frame and the second key reference frame into first encoded data and second encoded data respectively includes: Based on a dual-space prior encoder, the first key reference frame and the second key reference frame are encoded into first coded data and second coded data respectively using a super-prior technique; or... Using a variational autoencoder, the first key reference frame and the second key reference frame are encoded into first encoded data and second encoded data, respectively.

4. A cross-channel distributed video encoding and decoding system based on multidimensional attention, comprising: The encoder and decoder are provided, wherein the encoder is used to implement the multi-channel distributed video coding method based on multi-dimensional attention as described in any one of claims 1 to 3, and the decoder is used to decode the first bit stream and the second bit stream transmitted by the encoder.

5. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 3.

Images