A semantic communication method and system for ultra-high-definition video

Abstract

The present application provides a semantic communication method and system for ultra-high-definition video, which relates to the field of semantic communication technology. A video P-frame semantic feature extractor is used to obtain semantic features based on the P-frame to be encoded and the channel state. Based on the semantic features, a quadtree-based spatial entropy model is used to obtain four-step semantic importance. Based on the semantic features and the four-step semantic importance, a four-way semantic feature codec is used to obtain received semantic features. Based on the received semantic features and the channel state, as well as multi-scale reference context features, a semantic information restorer and a denoising enhancement module are used to reconstruct the video P-frame. A module constructed using CNN is used to extract and restore the semantic features of video frames, and a codec module constructed using Swin‑Transformer is used to process the semantic features of video frames. This not only improves encoding and decoding efficiency, but also enhances the model's ability to understand video content, helping to better reconstruct high-quality video content and control the size of model parameters.

Description

Technical Field

[0001] The present application relates to the field of semantic communication technology, and in particular to a semantic communication method and system for ultra-high-definition video. Background Art

[0002] The amount of data required to transmit for ultra-high-definition video-related services is extremely large, which will bring huge communication overhead. It is necessary to implement a compact coding and transmission solution to reduce the use of channel bandwidth.

[0003] Traditional image / video communication systems with a separate design have the following problems when transmitting ultra-high-definition video: in the case of poor channel conditions or limited bandwidth, once there is a mismatch between the communication transmission bit rate and the channel capacity, it will lead to a cliff effect. That is, the communication performance will drop sharply when the channel capacity is lower than the communication transmission bit rate, and the receiving end is often unable to recover the image / video data; the system is only an independent optimization at the symbol level. In the case of poor channel environment or limited communication bandwidth, the accuracy at the symbol level cannot effectively guarantee the accuracy at the source semantic level.

[0004] Based on this, a new semantic communication method for ultra-high-definition video transmission is urgently needed. Summary of the Invention

[0005] This application provides a semantic communication method and system for ultra-high-definition video to solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a semantic communication method for ultra-high-definition video is provided, the method comprising:

[0007] Input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded;

[0008] Estimating a super-prior distribution of the semantic features to obtain a super-prior;

[0009] Inputting the smallest-scale feature among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior;

[0010] Evenly dividing the semantic features into four groups along the channel dimension to obtain four groups of feature elements;

[0011] Obtaining an initial fusion prior according to the super prior, the time domain prior, and the semantic features of the previous video P frame;

[0012] Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior;

[0013] Obtaining four-step semantic importance based on the channel state, the four groups of feature elements, and the four-step spatial domain prior estimation;

[0014] Based on the initial fusion prior and the four groups of feature elements, using a four-way semantic feature encoder to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model;

[0015] The semantic feature encoding results of each channel are mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel;

[0016] The symbol mapping module at the receiving end restores the symbols according to their dimensions;

[0017] For the i-th path among the four paths, the channel state, the received i-th path semantic feature encoding result, and the semantic feature decoding results of each path before the i-th path are input into the i-th path semantic feature decoder to obtain the i-th path semantic feature decoding result. The four-path semantic feature decoding results constitute the received semantic feature;

[0018] Inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features;

[0019] The noisy semantic feature, the channel state, and a first reference context feature in the multi-scale reference context feature of the previous video P frame are input into a denoising and enhancement module to reconstruct a video P frame.

[0020] In an optional embodiment of the present application, when a quadtree-based spatial entropy model is used for step-by-step prior estimation, each group of characteristic elements in each step of the step-by-step prior estimation is different;

[0021] The feature element in step 0 uses the initial fusion prior, which is generated by the super prior, the time domain prior, and the semantic features of the previous video P frame;

[0022] The feature elements in step 1 will use the priors estimated based on the feature elements in step 0 and the initial fusion priors;

[0023] The feature elements in step 2 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, and the initial fusion priors;

[0024] The feature elements in step 3 will use the priors estimated based on the feature elements in step 0, step 1, step 2, and the initial fusion priors;

[0025] The prior used for the characteristic element at each step is used to estimate the symbol length corresponding to the encoding of the characteristic element at that step.

[0026] In an optional embodiment of the present application, based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, including:

[0027] Using the initial fusion prior as the spatial domain estimation result of the zero-step spatial domain estimation;

[0028] In the process of spatial domain estimation from the first to the third step, the estimated distribution features obtained in the previous step and the initial fusion prior are used as references, and the estimation is completed using a network with shared weights to obtain the spatial domain prior for each step.

[0029] In an optional embodiment of the present application, the generation process of the multi-scale reference context features used in the video P frame semantic feature extraction and recovery process is divided into three steps:

[0030] Step 1: Input the reference video frame into the multi-scale reference feature generation module to extract the outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include the Xth multi-scale reference features;

[0031] Step 2: Inputting the multi-scale reference features and the reference video frame into a coarse granularity reference context generation module to obtain a multi-scale coarse granularity reference context feature, wherein the multi-scale coarse granularity reference context feature includes an X-th multi-scale coarse granularity reference feature;

[0032] Step 3: Input the multi-scale coarse-grained reference context features and the channel state into a refinement network to obtain the multi-scale reference context features, where the multi-scale reference context features include an X-th reference context feature, where the value of X ranges from 1 to 3.

[0033] In an optional embodiment of the present application, the method further includes:

[0034] Inputting the to-be-encoded video P frame and a reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame;

[0035] Inputting the motion features into a super prior estimation module to obtain a super prior;

[0036] generating a motion fusion prior based on the super prior and motion features of a reference video frame;

[0037] Inputting the motion fusion prior and the motion features into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate;

[0038] Inputting the motion feature prior distribution estimate, the channel state, and the motion feature into a motion importance estimation module to obtain four-step motion importance;

[0039] For the i-th path among the four paths, the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result;

[0040] The motion feature encoding results of each channel are mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel;

[0041] The symbol mapping module at the receiving end restores the symbols according to their dimensions;

[0042] For the i-th path among the four paths, the channel state, the received i-th path motion feature encoding result, and the i-th path motion feature decoding results are input into the four-path motion feature decoder to obtain the i-th path motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature;

[0043] The received motion feature, the channel state and the motion reference feature of the reference video frame are input into a motion information restorer to restore a motion vector.

[0044] In an optional embodiment of the present application, the motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module;

[0045] Inputting the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, including:

[0046] Input the to-be-encoded video P frame and the reference video frame into the SpyNet module to perform optical flow estimation to obtain a motion vector;

[0047] The motion vector, the channel state and the motion reference feature of the reference video frame are input into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

[0048] In an optional embodiment of the present application, the channel perception module includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module;

[0049] Inputting the motion vector, the channel state, and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the P frame of the video to be encoded, including:

[0050] Inputting the motion vector into the first downsampling module to obtain a first downsampling result;

[0051] Inputting the first downsampling result and the channel state into the channel attention module to obtain a channel attention result;

[0052] Based on the first downsampling result and the channel attention result, using the spatial attention module to obtain a spatial attention result;

[0053] Obtaining the channel perception result based on the first downsampling result and the spatial attention result;

[0054] The channel sensing result and the motion reference feature of the reference video frame are input into the second downsampling module to obtain the motion feature of the P frame of the video to be encoded.

[0055] In an optional embodiment of the present application, each motion feature decoder includes: a channel sensing module and a motion feature decoding module, and the motion feature decoding module includes: a first operator layer and a second operator layer;

[0056] Inputting the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel before the i-th motion feature into a four-channel motion feature decoder to obtain the i-th motion feature decoding result, including:

[0057] Inputting the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result;

[0058] According to the first processing result and the channel state, the channel perception module and the second operator layer are used to obtain the i-th motion feature decoding result.

[0059] In an optional embodiment of the present application, the motion information restorer includes: a channel sensing module, a first upsampling module, and a second upsampling module;

[0060] Inputting the received motion feature, the channel state, and the motion reference feature of the reference video frame into a motion information restorer to restore a motion vector, comprising:

[0061] Inputting the received motion features into the first upsampling module to obtain a first upsampling result;

[0062] Inputting the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of the P frame of the video to be encoded;

[0063] The motion vector is restored by utilizing the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

[0064] In a second aspect of an embodiment of the present application, a semantic communication system for ultra-high-definition video is provided, the system comprising:

[0065] The semantic feature extraction module is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded;

[0066] A semantic super-prior estimation module, configured to estimate the super-prior distribution of the semantic features to obtain a super-prior;

[0067] A time domain prior estimation module, configured to input the smallest-scale feature among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior;

[0068] A feature segmentation module is used to evenly segment the semantic features into four groups along the channel dimension to obtain four groups of feature elements;

[0069] An initial fusion module, configured to obtain an initial fusion prior based on the super prior, the time domain prior, and the semantic features of the previous video P frame;

[0070] A spatial prior estimation module, configured to perform step-by-step prior estimation based on the four groups of feature elements and the initial fusion prior using a quadtree-based spatial entropy model to obtain a four-step spatial prior;

[0071] A semantic importance estimation module, configured to obtain four-step semantic importance based on the channel state, the four groups of characteristic elements, and the four-step spatial domain prior estimation;

[0072] a semantic feature encoding module, configured to obtain a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model;

[0073] A first wireless transmission module is used to map the semantic feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel;

[0074] A first restoration module, used for each symbol mapping module of the receiving end, to restore the symbols according to the symbol dimensions;

[0075] A semantic feature decoding module is configured to input the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel, thereby obtaining the i-th semantic feature decoding result. The four semantic feature decoding results constitute the received semantic feature.

[0076] A semantic information recovery module, configured to input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information recoverer to recover the noisy semantic features;

[0077] The reconstruction module is used to input the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context feature of the previous video P frame into the denoising and enhancement module to reconstruct the video P frame.

[0078] The present application has the following advantages: an embodiment of the present application provides a semantic communication method and system for ultra-high-definition video, which inputs the multi-scale reference context features of the video P frame to be encoded, the multi-scale reference context features of the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; performs super-prior distribution estimation on the semantic features to obtain super-prior; inputs the smallest-scale feature in the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain time domain prior; divides the semantic features evenly into four groups along the channel dimension to obtain four groups of feature elements; obtains an initial fusion prior based on the super-prior, the time domain prior, and the semantic features of the previous video P frame; based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; four-step semantic importance is estimated based on the channel state, the four groups of feature elements, and the four-step spatial prior; based on the initial fusion prior and the four groups of feature elements, a four-way semantic A feature encoder obtains four-way semantic feature encoding results, and the four-way semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; each semantic feature encoding result is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping module of each channel; each symbol mapping module of the receiving end restores the symbol according to the dimension of the symbol; for the i-th channel of the four channels, the channel state, the received i-th semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th semantic feature decoder to obtain the i-th semantic feature decoding result, and the four-way semantic feature decoding results constitute the received semantic feature; the received semantic feature, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic feature; the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context features of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The module built using CNN is used to extract and restore the semantic features of video frames, and the codec module built using Swin-Transformer is used to process the semantic features of video frames. This not only improves the encoding and decoding efficiency, but also enhances the model's ability to understand video content, helps to better reconstruct high-quality video content, and controls the size of model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0080] Figure 1 This is a flowchart of the steps of a semantic communication method for ultra-high-definition video provided in an embodiment of the present application;

[0081] Figure 2 This is a flow chart of a semantic communication method for ultra-high-definition video provided in an embodiment of the present application;

[0082] Figure 3 This is an architectural diagram of a semantic communication system for ultra-high-definition video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0084] In order to solve the following problems caused by the traditional image / video communication system with a separate design when transmitting ultra-high-definition video: under poor channel conditions or limited bandwidth, once a mismatch occurs between the communication transmission bit rate and the channel capacity, it will lead to a cliff effect, that is, the communication performance will drop sharply when the channel capacity is lower than the communication transmission bit rate, and the receiving end is often unable to recover the image / video data; the system is only an independent optimization at the symbol level. Under poor channel conditions or limited communication bandwidth, the accuracy at the symbol level is difficult to effectively guarantee the accuracy at the source semantic level. This application proposes a semantic communication method for ultra-high-definition video to solve the above problems. In video stream transmission and storage, common video compression image frames include I frames, P frames and B frames, among which I frames contain more information and can be independently encoded and decoded. Any I frame can be randomly decoded to play the video stream, while P frames contain relatively less information and cannot be independently encoded and decoded. Since P frames use motion prediction to perform inter-frame prediction coding, only the prediction error and motion vector are transmitted, which can significantly reduce transmission redundancy. The image frames in the ultra-high-definition video transmitted in this application are P frames.

[0085] See Figure 1 , Figure 1 This is a flowchart of a method for semantic communication of ultra-high-definition video provided in an embodiment of the present application, the method comprising the following steps:

[0086] Step 101: Inputting the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into a video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded;

[0087] Step 102: Estimating the hyper-prior distribution of the semantic features to obtain a hyper-prior;

[0088] Step 103: inputting the smallest-scale feature among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior;

[0089] Step 104: evenly divide the semantic features into four groups along the channel dimension to obtain four groups of feature elements;

[0090] Step 105: obtaining an initial fusion prior according to the super prior, the time domain prior, and the semantic features of the previous video P frame;

[0091] Step 106: Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior.

[0092] Step 107: Obtain four-step semantic importance based on the channel state, the four groups of feature elements, and the four-step spatial domain prior estimation;

[0093] Step 108: Based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model;

[0094] Step 109: Mapping the semantic feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping module of each channel;

[0095] Step 110: The symbol mapping module of each channel at the receiving end restores the symbol according to the symbol dimension;

[0096] Step 111: For the i-th channel among the four channels, the channel state, the received i-th channel semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th channel semantic feature decoder to obtain the i-th channel semantic feature decoding result. The four-channel semantic feature decoding results constitute the received semantic feature;

[0097] Step 112: Input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features;

[0098] Step 113: input the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context feature of the previous video P frame into a denoising and enhancement module to reconstruct the video P frame.

[0099] In order to clearly illustrate the semantic communication method of ultra-high-definition video proposed in this application, the following Figure 2 To explain, Figure 2 This is a flow chart of a semantic communication method for ultra-high-definition video provided in an embodiment of the present application.

[0100] When implementing step 101, first obtain a frame of the ultra-high-definition video to be transmitted as the video P frame to be encoded, wherein ultra-high-definition video means that the resolution of each frame in the video reaches 4k or 8k or above. Combined with the multi-scale reference context features of the previous video P frame and the channel state, the video P frame semantic feature extractor is used to obtain the semantic features of the video P frame to be encoded. The multi-scale reference context features of the previous video P frame correspond to Figure 2 The originating residual reference in .

[0101] In an optional embodiment of the present application, the video P frame semantic feature extractor includes multiple Conv2d layers, ResBlock layers and a channel perception module (AF Module). and the first reference context feature in the multi-scale reference context feature of the previous video P frame Input the first Conv2d layer to obtain the first processing result, and combine the first processing result with the second reference context feature in the multi-scale reference context feature of the previous video P frame Input the first ResBlock layer to obtain the second processing result, input the channel state SNR into the channel perception module (AFModule) to obtain the channel perception result, and based on the second processing result, the channel perception result and the second reference context feature in the multi-scale reference context feature of the previous video P frame , using the remaining layers in the video P-frame semantic feature extractor to obtain semantic features .

[0102] When step 102 is specifically implemented, a super-prior distribution estimation is performed on the semantic features to obtain a super-prior. In order to compact the semantic features in this application, a super-prior is introduced to make a preliminary estimate of the potential distribution of the semantic features, and a super-prior distribution estimation is performed through a single neural network model. A super-prior distribution estimation is performed on the above-mentioned semantic features to obtain a super-prior. Specifically, the above-mentioned semantic features are input into a super-prior estimation module to obtain the above-mentioned super-prior. The super-prior estimation module is only deployed at the transmitting end, and the transmitting end does not transmit the super-prior.

[0103] Specifically, when implementing step 103, the smallest scale feature among the multi-scale reference context features of the previous video P frame is input into the time domain prior estimation module to obtain the time domain prior. The time domain prior estimation module includes two Conv2d layers and one Leaky Relu layer. Figure 2The temporal prior estimation module estimates the temporal prior distribution of semantic features to obtain temporal priors. Specifically, the feature with the smallest scale in the multi-scale reference context features of the previous video P frame is Input the super prior estimation module to obtain the temporal prior.

[0104] When implementing step 104, a quadtree entropy model is introduced in this application to perform subsequent spatial domain prior distribution estimation and semantic importance estimation. First, the extracted semantic features are evenly divided into four groups along the channel dimension to obtain four groups of feature elements. The four groups of feature elements are used for subsequent spatial domain prior distribution estimation and semantic importance estimation. The channel dimension of the semantic features is channel, which can be abbreviated as C.

[0105] Specifically, when step 105 is implemented, when the video P frame is transmitted, when the previous frame of the video P frame to be encoded is a P frame, an initial fusion prior is obtained based on the super prior, the time domain prior, and the semantic features of the previous video P frame; when the previous frame of the video P frame to be encoded is an I frame, the initial fusion prior of the video P frame to be encoded is generated by the super prior and the time domain prior.

[0106] When specifically implementing step 106, a quadtree-based spatial entropy model is used to estimate the spatial prior distribution and semantic importance. The quadtree-based spatial entropy model can fully identify the redundancy of spatial information in semantic features. First, based on four groups of feature elements and the above-mentioned initial fusion super-prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior. According to the step-by-step estimation strategy of the entropy model, a four-way semantic feature coding transmission link with mutual reference is realized; the semantic importance is estimated based on the output result of the entropy model to guide the feature encoder to fully reduce the channel bandwidth occupancy and realize unequal error protection transmission.

[0107] In an optional embodiment of the present application, when a quadtree-based spatial entropy model is used for step-by-step prior estimation, each group of feature elements in each step-by-step prior estimation is different; specifically, the prior used for the feature element of step 0 is the initial fusion prior, which is generated based on the super prior, the time domain prior, and the semantic features of the previous video P frame; the prior used for the feature element of step 1 is estimated based on the feature element of step 0 and the initial fusion prior; the prior used for the feature element of step 2 is estimated based on the feature element of step 0, the feature element of step 1, and the initial fusion prior; the prior used for the feature element of step 3 is estimated based on the feature element of step 0, the feature element of step 1, the feature element of step 2, and the initial fusion prior; since the prior used for the feature element of each step is different, the feature element of each step is also different. The prior used for the feature element of each step is used to estimate the symbol length corresponding to the feature element of that step after encoding.

[0108] In an optional embodiment of the present application, based on four groups of feature elements and the above-mentioned initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, specifically including: first, using the above-mentioned initial fusion prior as the spatial estimation result of the zero-step spatial estimation; then, in the process of the first to third steps of spatial estimation, the estimated distribution characteristics obtained in the previous steps and the initial fusion prior are used as references, and a shared weight network is used to complete the estimation to obtain the spatial prior for each step.

[0109] In an optional embodiment of the present application, the spatial domain prior distribution estimation is divided into four steps, which can be found in Figure 2 , along the channel dimension C, the dimension is The semantic features are evenly divided into four groups of dimensions: Each step will estimate the feature element of the corresponding index. C represents the channel dimension, H represents the height dimension, and W represents the width dimension. For example, in step 0 ( Figure 2 In Step 0) each group will Figure 2The position with index 0 in the [H, W] matrix is ​​estimated. Each group has a different index, but the combined indexes of the groups cover the dimension [H, W]. The indexes at each estimation step are also different, and the combination of feature elements used in each estimation step is the combination of all feature elements corresponding to the estimated index at each step. Therefore, each step of spatial prior distribution estimation covers every spatial position in the dimension [H, W], and the estimated feature elements can form a feature with dimensions [C / 4, H, W]. In subsequent steps 1, 2, and 3, the spatial prior distribution estimation uses the feature elements estimated in the previous steps as a reference. This allows the estimated position within the same group at each step to refer to its neighbors in the same group and the same position in different groups, thereby guiding the model to fully identify spatial information redundancy in the semantic features. The entropy model output parameters are also used to guide the semantic feature encoder to implement unequal error protection. The semantic importance estimation module uses the prior distribution parameters estimated by the entropy model and the semantic features to calculate the semantic importance of each feature element. The output of this module interacts with the semantic feature encoding process through the cross-attention layer in the semantic feature encoder.

[0110] Specifically, during step 107, when performing semantic importance estimation, the channel state, the four sets of feature elements, and the four-step spatial priors are input into the semantic importance estimation module to estimate the semantic importance of the four steps. The semantic importance estimation module used in each step includes multiple DepthConvBlock layers, multiple ResBlock layers, and a channel awareness module (AFModule).

[0111] Specifically, when step 108 is implemented, based on the above-mentioned initial fusion prior and the above-mentioned four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result. The above-mentioned four-way semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; the feature spatial domain component dimension estimated according to the spatial domain prior distribution of each step is The feature elements of the spatial domain prior distribution estimation are used to construct a four-way semantic feature encoder. The features composed of each step of the spatial domain prior distribution estimation are encoded separately to obtain the four-step semantic feature encoding results. The neural network model is constructed using the self-attention layer of Swin-Transformer and the cross-attention layer based on Swin-Transformer-V2 to remove the information redundancy of the semantic features identified in the spatial domain distribution estimation. The network structure of each semantic feature encoder is the same. Each semantic feature encoder includes: a channel perception module and a semantic feature encoding module. The above-mentioned semantic feature encoding module includes: a first cross-attention layer and a second cross-attention layer, as well as a first self-attention layer, a second self-attention layer, and a DepthConvBlock layer.

[0112] When implementing steps 109 and 110, after obtaining the i-th semantic feature encoding result, the corresponding symbol mapping module of each channel maps each semantic feature encoding result into symbols of different lengths for wireless transmission; then, the symbol mapping module of each channel at the receiving end restores the symbol according to the symbol dimension, and decodes the restored encoding result. The dimension of the output result of the semantic feature after the feature encoder is , for the dimension Each element of has a length of , according to the hyper-prior distribution estimated by the entropy model, the symbol length required for the element is calculated, and a linear layer matching the symbol length is selected for mapping. At the receiving end, the corresponding linear layer will be used to map it back to length , thereby recovering the dimension from the received symbols The coded transmission result is then transmitted to the semantic feature decoder. Based on the symbol mapping module, a semantic communication method for ultra-high-definition video is implemented, which dynamically adjusts the video codec output bit rate according to the channel status and semantic information.

[0113] When step 111 is specifically implemented, for the i-th path among the four paths, the above-mentioned channel state, the received i-th semantic feature encoding result, and the semantic feature decoding results of each path before the i-th path are input into the i-th semantic feature decoder to obtain the i-th semantic feature decoding result. The four semantic feature decoding results constitute the received semantic features; each semantic feature decoder includes: a channel perception module and a semantic feature decoding module, and the semantic feature decoding module includes: a third cross-attention layer, and also includes: a third self-attention layer, a fourth self-attention layer and a DepthConvBlock layer.

[0114] Specifically, when implementing step 112, after decoding the semantic features to obtain the received semantic features, semantic information reconstruction processing is performed on the received semantic features, which is implemented in the present application through a semantic information restorer and a denoising enhancement module. The received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic features.

[0115] In an optional embodiment of the present application, the semantic information restorer includes: multiple upsampling modules (ResBlock Upsample), two ResBlock layers and a channel perception module (AF Module). First, the received semantic features Input the first upsampling module to obtain the upsampling result; based on the upsampling result and the third reference context feature of the previous video P frame , using the first ResBlock layer, obtain the reference processing result; based on the channel state SNR and the reference processing result, use the channel sensing module to obtain the channel sensing processing result; based on the channel sensing processing result and the second reference context feature , use the remaining modules to obtain noisy semantic features .

[0116] Specifically, when step 113 is implemented, the noisy semantic features, the channel state, and the first reference context feature in the multi-scale reference context feature of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The multi-scale reference context feature of the previous video P frame used in steps 112 and 113 corresponds to Figure 2 The receiving end context in .

[0117] In an optional embodiment of the present application, the denoising enhancement module includes: two Conv2d layers, two Unet layers and a channel perception module (AF Module). Input the first Conv2d layer to obtain the first denoising result; based on the first denoising result and the channel perception state SNR, use the channel perception module and the residual module to obtain the video P frame .

[0118] In an optional embodiment of the present application, the generation process of the multi-scale reference context features used in the video P frame semantic feature extraction and recovery process is divided into three steps. Specifically, in the first step, the reference video frame The input multi-scale reference feature generation module extracts the output of different network layers to obtain the X-th multi-scale reference feature, where the value range of X is 1-3. The X-th multi-scale reference feature includes the first multi-scale reference feature , the second multi-scale reference feature and the third multi-scale reference feature ; In the second step, the above multi-scale reference features 、 and , and the above reference video frame Input the coarse granularity reference context generation module to obtain the Xth multi-scale coarse granularity reference context feature, wherein the multi-scale coarse granularity reference context feature includes the first multi-scale coarse granularity reference context feature , the second multi-scale coarse-grained reference context feature and the third multi-scale coarse-grained reference context feature .

[0119] The reference context features of the roughness intensity obtained above need to be refined by the neural network before they can be used. The refined network is based on the residual network, which guides the network to learn and supplement the fine-grained reference information, and introduces the channel perception module. In addition, the reference context features of different scales refer to each other during the generation process. In the third step, the multi-scale coarse-grained reference context features and the channel state are input into the refined network to obtain the Xth reference context feature, where the value range of X is 1-3. The multi-scale reference context features include the first reference context feature , the second reference context feature and the third reference context feature .

[0120] In an optional embodiment of the present application, the method further includes the following steps:

[0121] Step 201: Input the to-be-encoded video P frame and a reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, where the reference video frame is a previous video P frame of the to-be-encoded video P frame.

[0122] Step 202: Input the motion features into a super prior estimation module to obtain a super prior;

[0123] Step 203: Generate a motion fusion prior based on the super prior and the motion features of the reference video frame;

[0124] Step 204: inputting the motion fusion prior and the motion features into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate;

[0125] Step 205: Inputting the motion feature prior distribution estimate, the channel state, and the motion feature into a motion importance estimation module to obtain four-step motion importance;

[0126] Step 206: For the i-th path among the four paths, input the feature element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state, and the motion feature encoding results of each path before the i-th path into the four-path motion feature encoder to obtain the i-th path motion feature encoding result;

[0127] Step 207: Mapping the motion feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping module of each channel;

[0128] Step 208: The symbol mapping module of each channel at the receiving end restores the symbol according to the symbol dimension;

[0129] Step 209: For the i-th channel among the four channels, input the channel state, the received i-th channel motion feature encoding result, and the i-th channel motion feature decoding results into a four-channel motion feature decoder to obtain the i-th channel motion feature decoding result, and combine the four-channel motion decoding results into a received motion feature;

[0130] Step 210: Input the received motion feature, the channel state, and the motion reference feature of the reference video frame into a motion information restorer to restore a motion vector.

[0131] Specifically, when step 201 is implemented, the video P frame to be encoded and the reference video frame are input into a motion information extractor to obtain motion features of the video P frame to be encoded. The reference video frame is the previous video P frame of the video P frame to be encoded.

[0132] In an optional embodiment of the present application, the network structure of the motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module; the motion information extractor also includes: a ResBlock Downsample layer, a DepthConvBlock layer and a Conv2d layer. Based on the above network structure, the P frame of the video to be encoded and the reference video frame are input into the motion information extractor to obtain the motion features of the P frame of the video to be encoded, specifically including: first, the P frame of the video to be encoded is input into the motion information extractor. and reference video frames , input the above SpyNet module to perform optical flow estimation and obtain the motion vector , and then the motion vector , the above channel state (such as SNR) and the motion reference features of the reference video frame Input the above downsampling module and the above channel perception module for downsampling and channel perception to obtain the motion features of the above video P frame to be encoded .

[0133] In an optional embodiment of the present application, the above-mentioned channel perception module (AF Module) includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; the above-mentioned motion vector, the above-mentioned channel state and the motion reference feature of the reference video frame are input into the above-mentioned downsampling module and the above-mentioned channel perception module for downsampling and channel perception, and the motion feature of the above-mentioned video P frame to be encoded is obtained, specifically including: the above-mentioned motion vector Input the first downsampling module to obtain a first downsampling result; input the first downsampling result and the channel state (such as SNR) into the channel attention module to obtain a channel attention result; based on the first downsampling result and the channel attention result, use the spatial attention module to obtain a spatial attention result; based on the first downsampling result and the spatial attention result, obtain the channel perception result; and combine the channel perception result and the motion reference feature of the reference video frame. Input the second downsampling module to obtain the motion features of the P frame of the video to be encoded By integrating a channel perception module, the system can adaptively adjust the encoding and decoding strategy based on real-time channel conditions (e.g., SNR), and use the attention mechanism to enhance important features of the video data, thereby improving overall transmission efficiency and quality. The aforementioned motion reference features are the output of the intermediate network layer of the motion information recovery module.

[0134] When step 202 is specifically implemented, the motion features are input into the super-prior estimation module to obtain super-prior; the process of super-prior estimation of motion features is the same as the process in step 102, the difference being that the input object is motion features, not semantic features.

[0135] Specifically, when step 203 is implemented, a motion fusion prior is generated based on the above-mentioned super prior and the motion features of the reference video frame; and the motion fusion prior is used for subsequent motion prior estimation using a quadtree-based spatial entropy model.

[0136] Specifically, during step 204, the quadtree-based spatial entropy model can fully identify spatial information redundancy in motion features. Based on the aforementioned motion fusion prior and the aforementioned motion features, the quadtree-based spatial entropy model is input for step-by-step prior estimation, resulting in a four-step motion feature prior distribution estimate. The quadtree-based cross-entropy model is used to estimate the distribution prior in the same way as the semantic feature estimation described above, differing in that the inputs are motion features and the motion fusion prior.

[0137] Specifically, during step 205 , the motion importance estimation module inputs the motion feature prior distribution estimate, the channel state, and the motion features to estimate the four-step motion importance. Each step of the motion importance estimation module includes multiple DepthConvBlock layers, multiple ResBlock layers, and a channel awareness module (AF Module).

[0138] Specifically, during step 206, the motion features are divided into four groups of feature elements by channel. The index used in each estimation step is different. The feature element combination used in each estimation step is the combination of all feature elements corresponding to the estimation index at each step. The feature element combination corresponding to the estimation index at the i-th step is the feature element combination corresponding to the estimation index at the i-th step (the feature elements in this feature element combination are obtained by dividing the motion features by channel). Based on the motion importance and the feature element combination, a four-channel motion feature encoder is used to obtain the motion feature encoding results for each channel.

[0139] Each motion feature encoder includes: a channel perception module and a motion feature encoding module, and the motion feature encoding module includes: a first Conv2d layer, a second DepthConvBlock layer, a third ResBlock layer and a fourth ResBlock layer; based on the above network structure, for the i-th path in the four paths, the feature element combination corresponding to the i-th step estimation index, the i-th step motion importance, the above channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result, specifically including: , decoding results of motion features before the i-th path and the importance of the motion at step i , input the first Conv2d layer to obtain the first encoding result; based on the first encoding result and the channel state (such as SNR), the second DepthConvBlock layer and the channel perception module (AF Module) are used to obtain the second encoding result; based on the second encoding result, the third ResBlock layer and the fourth ResBlock layer are used to obtain the i-th motion feature encoding result .

[0140] During the specific implementation of steps 207 and 208, the corresponding symbol mapping module maps each motion feature encoding result into symbols of different lengths for wireless transmission. Each symbol mapping module at the receiving end restores the symbol based on its dimension and decodes the restored encoding result. This symbol mapping module is similar to the specific implementation of steps 109 and 110, except that the restored motion feature encoding result is transmitted.

[0141] When step 209 is specifically implemented, for the i-th path among the four paths, the above-mentioned channel state, the received i-th motion feature encoding result, and the motion feature decoding results of each path before the i-th path are input into the four-path motion feature decoder to obtain the i-th motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature; based on the received motion feature encoding result, the four-path motion feature encoder is used to obtain the motion feature encoding result of each path.

[0142] Each motion feature decoder includes: a channel perception module and a motion feature decoding module, the motion feature decoding module includes: a first operator layer and a second operator layer, the first operator layer is a Conv2d layer, and the second operator layer is a DepthConvBlock layer. Based on the above network structure, the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel are input into the four-channel motion feature decoder to obtain the i-th motion feature decoding result, which specifically includes: the received i-th motion feature encoding result is input into the four-channel motion feature decoder. and the decoding results of motion features of each path before the i-th path Input the first operator layer to obtain the first processing result; according to the first processing result and the channel state (such as SNR), use the channel perception module (AF Module) and the second operator layer to obtain the i-th motion feature decoding result The four-way motion feature decoding results are combined into the received motion feature for subsequent motion information recovery. After the motion vector is obtained, it is processed and used as the reference motion vector for the next video.

[0143] Specifically, when implementing step 210, the receiving end recovers the motion vector by upsampling and channel-aware allocation of the received motion features. Specifically, the received motion features, the channel state, and the motion reference features of the reference video frame are input into a motion information recovery device to recover the motion vector.

[0144] In an optional embodiment of the present application, the above-mentioned motion information restorer includes: a channel perception module, a first upsampling module, and a second upsampling module; the above-mentioned first upsampling module and the second upsampling module are both composed of a DepthConvBlock2 layer and a ResBlock Upsample layer. The above-mentioned motion information restorer also includes: a DepthConvBlock2 layer, a DepthConvBlock layer, and a ResBlock Upsample layer. Based on the above-mentioned network structure, the above-mentioned received motion features, the above-mentioned channel state, and the motion reference features of the reference video frame are input into the motion information restorer to restore the motion vector, specifically including: the above-mentioned received motion features are input into the motion information restorer to restore the motion vector. Input the first upsampling module to obtain the first upsampling result; input the first upsampling result and the channel state (such as SNR) into the channel perception module (AF Module) to obtain the second upsampling result and the motion reference feature of the P frame of the video to be encoded ; Motion reference features based on reference video frames The second upsampling module is used to recover the motion vector The motion vector recovered by the motion information restorer It is noisy and needs to be applied after offset prediction. The second upsampling result is used to extract motion information of the next video frame.

[0145] The embodiment of the present application provides a semantic communication method for ultra-high-definition video, which inputs the multi-scale reference context features of the video P frame to be encoded, the previous video P frame and the channel state into a video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; performs super-prior distribution estimation on the semantic features to obtain super-prior; inputs the smallest-scale feature in the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain time domain prior; divides the semantic features evenly into four groups along the channel dimension to obtain four groups of feature elements; obtains an initial fusion prior based on the super-prior, the time domain prior and the semantic features of the previous video P frame; based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; based on the channel state, the four groups of feature elements and the four-step spatial prior estimation, a four-step semantic importance is obtained; based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain To the four-way semantic feature encoding results, the four-way semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; each semantic feature encoding result is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping module of each channel; the symbol mapping module of each channel at the receiving end restores the symbol according to the dimension of the symbol; for the i-th channel among the four channels, the channel state, the received i-th semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th semantic feature decoder to obtain the i-th semantic feature decoding result, and the four-way semantic feature decoding results constitute the received semantic feature; the received semantic feature, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic feature; the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context features of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The module built using CNN is used to extract and restore the semantic features of video frames, and the codec module built using Swin-Transformer is used to process the semantic features of video frames. This not only improves the encoding and decoding efficiency, but also enhances the model's ability to understand video content, helps to better reconstruct high-quality video content, and controls the size of model parameters.

[0146] In a second aspect of the present application, a semantic communication system for ultra-high-definition video is provided. Figure 3 , Figure 3This is an architecture diagram of a semantic communication system for ultra-high-definition video provided in an embodiment of the present application, the system comprising:

[0147] The semantic feature extraction module 301 is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded;

[0148] A semantic super-prior estimation module 302 is used to estimate the super-prior distribution of the semantic features to obtain a super-prior;

[0149] A time domain prior estimation module 303 is configured to input the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior;

[0150] A feature segmentation module 304 is configured to evenly segment the semantic features into four groups along the channel dimension to obtain four groups of feature elements;

[0151] An initial fusion module 305 is configured to obtain an initial fusion prior based on the super prior, the time domain prior, and the semantic features of the previous video P frame;

[0152] A spatial prior estimation module 306 is configured to perform step-by-step prior estimation based on the four sets of feature elements and the initial fusion prior using a quadtree-based spatial entropy model to obtain a four-step spatial prior.

[0153] A semantic importance estimation module 307, configured to obtain four-step semantic importance based on the channel state, the four groups of characteristic elements, and the four-step spatial domain prior estimation;

[0154] a semantic feature encoding module 308 for obtaining a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model;

[0155] The first wireless transmission module 309 is used to map the semantic feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel;

[0156] The first restoration module 310 is used for the symbol mapping module of each channel of the receiving end to restore the symbol according to the dimension of the symbol;

[0157] The semantic feature decoding module 311 is used to input the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel, to obtain the i-th semantic feature decoding result. The four semantic feature decoding results constitute the received semantic feature;

[0158] A semantic information recovery module 312 is configured to input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information recovery module to recover the noisy semantic features;

[0159] The reconstruction module 313 is configured to input the noisy semantic feature, the channel state, and the first reference context feature of the multi-scale reference context feature of the previous video P frame into a denoising and enhancement module to reconstruct a video P frame.

[0160] Wherein, when the spatial domain entropy model based on quadtree is used to perform step-by-step priori estimation in the spatial domain priori estimation module, each group of characteristic elements in each step-by-step priori estimation is different; the spatial domain priori estimation module includes:

[0161] A priori estimation submodule of step 0, wherein the priori used for the feature element of step 0 is an initial fusion priori, and the initial fusion priori is generated by the super priori, the time domain priori and the semantic features of the previous video P frame;

[0162] The first step prior estimation submodule uses the feature elements of step 1 as the prior estimated based on the feature elements of step 0 and the initial fusion prior.

[0163] The prior estimation submodule of step 2 is used for the feature elements of step 2, and the prior used is estimated based on the feature elements of step 0, the feature elements of step 1, and the initial fusion prior.

[0164] The prior estimation submodule of step 3 uses the priors of the feature elements of step 3 as estimated based on the feature elements of step 0, step 1, step 2 and the initial fusion priors;

[0165] The prior estimation submodule is used to estimate the symbol length corresponding to the encoded characteristic element of each step using the prior information used in the characteristic element of each step.

[0166] Wherein, the spatial domain prior estimation module further includes:

[0167] An initial prior estimation submodule, configured to use the initial fusion prior as a spatial domain estimation result of a zero-step spatial domain estimation;

[0168] The first to third step estimation submodule is used to use the estimated distribution characteristics obtained in the previous steps and the initial fusion prior as a reference during the first to third step spatial domain estimation process, and use a shared weight network to complete the estimation to obtain the spatial domain prior for each step.

[0169] The generation process of the multi-scale reference context features used in the semantic information recovery module and the reconstruction module is implemented by the following submodules:

[0170] A first generation submodule is configured to input a reference video frame into a multi-scale reference feature generation module to extract outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include an Xth multi-scale reference feature;

[0171] A second generation submodule is configured to input the multi-scale reference features and the reference video frame into a coarse granularity reference context generation module to obtain a multi-scale coarse granularity reference context feature, wherein the multi-scale coarse granularity reference context feature includes an Xth multi-scale coarse granularity reference feature;

[0172] The third generation submodule is used to input the multi-scale coarse-grained reference context feature and the channel state into the refinement network to obtain the multi-scale reference context feature, where the multi-scale reference context feature includes the Xth reference context feature, and the value range of X is 1-3.

[0173] Wherein, the system further includes:

[0174] a motion feature extraction module, configured to input the to-be-encoded video P frame and a reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame;

[0175] A motion super-prior estimation module, configured to input the motion features into the super-prior estimation module to obtain a super-prior;

[0176] A motion fusion prior module, configured to generate a motion fusion prior based on the super prior and motion features of a reference video frame;

[0177] A motion feature prior distribution estimation module, configured to input the motion fusion prior and the motion feature into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimation;

[0178] A motion importance estimation module, configured to input the motion feature prior distribution estimate, the channel state, and the motion feature into the motion importance estimation module to obtain four-step motion importance;

[0179] A motion feature encoding module is configured to input, for an i-th path among the four paths, a feature element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state, and the motion feature encoding results of each path before the i-th path into a four-path motion feature encoder to obtain an i-th path motion feature encoding result;

[0180] The second wireless transmission module is used to map the motion feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel;

[0181] The second restoration module is used for the symbol mapping module of each channel of the receiving end to restore the symbol according to the dimension of the symbol;

[0182] A motion feature decoding module is configured to input the channel state, the received motion feature encoding result of the i-th channel, and the motion feature decoding results of each channel before the i-th channel into a four-channel motion feature decoder for the i-th channel, obtain the i-th motion feature decoding result, and combine the four-channel motion decoding results into a received motion feature;

[0183] The motion vector recovery module is used to input the received motion feature, the channel state and the motion reference feature of the reference video frame into a motion information recoverer to recover the motion vector.

[0184] The motion information extractor in the motion feature extraction module includes at least a SpyNet module, a channel perception module and a downsampling module; the motion feature extraction module includes:

[0185] An optical flow estimation submodule, configured to input the to-be-encoded video P frame and the reference video frame into the SpyNet module for optical flow estimation to obtain a motion vector;

[0186] The downsampling and channel sensing submodule is used to input the motion vector, the channel state and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

[0187] The channel perception module in the downsampling and channel perception submodule includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; the downsampling and channel perception submodule includes:

[0188] a first downsampling unit, configured to input the motion vector into the first downsampling module to obtain a first downsampling result;

[0189] a channel attention unit, configured to input the first downsampling result and the channel state into the channel attention module to obtain a channel attention result;

[0190] a spatial attention unit, configured to obtain a spatial attention result using the spatial attention module based on the first downsampling result and the channel attention result;

[0191] a channel sensing unit, configured to obtain the channel sensing result based on the first downsampling result and the spatial attention result;

[0192] The second downsampling unit is configured to input the channel sensing result and the motion reference feature of the reference video frame into the second downsampling module to obtain the motion feature of the P frame of the video to be encoded.

[0193] Wherein, each motion feature decoder in the motion feature decoding module includes: a channel sensing module and a motion feature decoding module, the motion feature decoding module includes: a first operator layer and a second operator layer; the motion feature decoding module includes:

[0194] A first processing submodule, configured to input the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result;

[0195] The second processing submodule is used to obtain the i-th motion feature decoding result by using the channel perception module and the second operator layer according to the first processing result and the channel state.

[0196] The motion information restorer in the motion vector recovery module includes: a channel sensing module, a first upsampling module, and a second upsampling module; the motion vector recovery module includes:

[0197] a first upsampling submodule, configured to input the received motion features into the first upsampling module to obtain a first upsampling result;

[0198] a channel sensing submodule, configured to input the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of the P frame of the video to be encoded;

[0199] The second upsampling submodule is configured to restore the motion vector using the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

[0200] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0201] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements that are inherent to such process, method, article, or terminal device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0203] The above is a detailed introduction to the semantic communication method and system for ultra-high-definition video provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A semantic communication method for ultra-high-definition video, characterized in that: The method comprises: Input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; Estimating a super-prior distribution of the semantic features to obtain a super-prior; Inputting the smallest-scale feature among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior; Evenly dividing the semantic features into four groups along the channel dimension to obtain four groups of feature elements; Obtaining an initial fusion prior according to the super prior, the time domain prior, and the semantic features of the previous video P frame; Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; Obtaining four-step semantic importance based on the channel state, the four groups of feature elements, and the four-step spatial domain prior estimation; Based on the initial fusion prior and the four groups of feature elements, using a four-way semantic feature encoder to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; The semantic feature encoding results of each channel are mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping module at the receiving end restores the symbols according to their dimensions; For the i-th path among the four paths, the channel state, the received i-th path semantic feature encoding result, and the semantic feature decoding results of each path before the i-th path are input into the i-th path semantic feature decoder to obtain the i-th path semantic feature decoding result. The four-path semantic feature decoding results constitute the received semantic feature; Inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features; The noisy semantic feature, the channel state, and a first reference context feature in the multi-scale reference context feature of the previous video P frame are input into a denoising and enhancement module to reconstruct a video P frame.

2. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: When the quadtree-based spatial entropy model is used for step-by-step prior estimation, each set of characteristic elements in each step of the step-by-step prior estimation is different; The feature element in step 0 uses the initial fusion prior, which is generated by the super prior, the time domain prior, and the semantic features of the previous video P frame; The feature elements in step 1 will use the priors estimated based on the feature elements in step 0 and the initial fusion priors; The feature elements in step 2 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, and the initial fusion priors; The feature elements in step 3 will use the priors estimated based on the feature elements in step 0, step 1, step 2, and the initial fusion priors; The prior used for the characteristic element at each step is used to estimate the symbol length corresponding to the encoding of the characteristic element at that step.

3. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, including: Using the initial fusion prior as the spatial domain estimation result of the zero-step spatial domain estimation; In the process of spatial domain estimation from the first to the third step, the estimated distribution features obtained in the previous step and the initial fusion prior are used as references, and the estimation is completed using a network with shared weights to obtain the spatial domain prior for each step.

4. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: The generation process of multi-scale reference context features used in the semantic feature extraction and recovery of video P frames is divided into three steps: Step 1: Input the reference video frame into the multi-scale reference feature generation module to extract the outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include the Xth multi-scale reference features; Step 2: Inputting the multi-scale reference features and the reference video frame into a coarse granularity reference context generation module to obtain a multi-scale coarse granularity reference context feature, wherein the multi-scale coarse granularity reference context feature includes an X-th multi-scale coarse granularity reference feature; Step 3: Input the multi-scale coarse-grained reference context features and the channel state into a refinement network to obtain the multi-scale reference context features, where the multi-scale reference context features include an X-th reference context feature, where the value of X ranges from 1 to 3.

5. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: The method further comprises: Inputting the to-be-encoded video P frame and a reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame; Inputting the motion features into a super prior estimation module to obtain a super prior; generating a motion fusion prior based on the super prior and motion features of a reference video frame; Inputting the motion fusion prior and the motion features into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate; Inputting the motion feature prior distribution estimate, the channel state, and the motion feature into a motion importance estimation module to obtain four-step motion importance; For the i-th path among the four paths, the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result; The motion feature encoding results of each channel are mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping module at the receiving end restores the symbols according to their dimensions; For the i-th path among the four paths, the channel state, the received i-th path motion feature encoding result, and the i-th path motion feature decoding results are input into the four-path motion feature decoder to obtain the i-th path motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature; The received motion feature, the channel state and the motion reference feature of the reference video frame are input into a motion information restorer to restore a motion vector.

6. The semantic communication method of ultra-high-definition video according to claim 5, characterized in that: The motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module; Inputting the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, including: Input the to-be-encoded video P frame and the reference video frame into the SpyNet module to perform optical flow estimation to obtain a motion vector; The motion vector, the channel state and the motion reference feature of the reference video frame are input into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

7. The semantic communication method of ultra-high-definition video according to claim 6, characterized in that: The channel perception module includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; Inputting the motion vector, the channel state, and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the P frame of the video to be encoded, including: Inputting the motion vector into the first downsampling module to obtain a first downsampling result; Inputting the first downsampling result and the channel state into the channel attention module to obtain a channel attention result; Based on the first downsampling result and the channel attention result, using the spatial attention module to obtain a spatial attention result; Obtaining the channel perception result based on the first downsampling result and the spatial attention result; The channel sensing result and the motion reference feature of the reference video frame are input into the second downsampling module to obtain the motion feature of the P frame of the video to be encoded.

8. The semantic communication method of ultra-high-definition video according to claim 5, characterized in that: Each motion feature decoder includes: a channel sensing module and a motion feature decoding module, and the motion feature decoding module includes: a first operator layer and a second operator layer; Inputting the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel before the i-th motion feature into a four-channel motion feature decoder to obtain the i-th motion feature decoding result, including: Inputting the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result; According to the first processing result and the channel state, the channel perception module and the second operator layer are used to obtain the i-th motion feature decoding result.

9. The semantic communication method of ultra-high-definition video according to claim 5, characterized in that: The motion information restorer includes: a channel sensing module, a first upsampling module, and a second upsampling module; Inputting the received motion feature, the channel state, and the motion reference feature of the reference video frame into a motion information restorer to restore a motion vector, comprising: Inputting the received motion features into the first upsampling module to obtain a first upsampling result; Inputting the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of the P frame of the video to be encoded; The motion vector is restored by utilizing the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

10. A semantic communication system for ultra-high-definition video, characterized in that: The system comprises: The semantic feature extraction module is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; A semantic super-prior estimation module, configured to estimate the super-prior distribution of the semantic features to obtain a super-prior; A time domain prior estimation module, configured to input the smallest-scale feature among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior; A feature segmentation module is used to evenly segment the semantic features into four groups along the channel dimension to obtain four groups of feature elements; An initial fusion module, configured to obtain an initial fusion prior based on the super prior, the time domain prior, and the semantic features of the previous video P frame; A spatial prior estimation module, configured to perform step-by-step prior estimation based on the four groups of feature elements and the initial fusion prior using a quadtree-based spatial entropy model to obtain a four-step spatial prior; A semantic importance estimation module, configured to obtain four-step semantic importance based on the channel state, the four groups of characteristic elements, and the four-step spatial domain prior estimation; a semantic feature encoding module, configured to obtain a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; A first wireless transmission module is used to map the semantic feature encoding results of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; A first restoration module, used for each symbol mapping module of the receiving end, to restore the symbols according to the symbol dimensions; A semantic feature decoding module is configured to input the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel, thereby obtaining the i-th semantic feature decoding result. The four semantic feature decoding results constitute the received semantic feature. A semantic information recovery module, configured to input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information recoverer to recover the noisy semantic features; The reconstruction module is used to input the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context feature of the previous video P frame into the denoising and enhancement module to reconstruct the video P frame.

Images