Semantic communication method and system for ultra-high-definition video

Abstract

The invention provides a semantic communication method and system for an ultra-high-definition video, and relates to the technical field of semantic communication, and the method comprises the steps: obtaining semantic features based on a to-be-coded video P frame and a channel state through employing a video P frame semantic feature extractor; based on the semantic features, obtaining four-step semantic importance by using a quadtree-based spatial domain entropy model; based on the semantic features and the four-step semantic importance, utilizing a four-way semantic feature codec to obtain the received semantic features; and reconstructing a video P frame by using a semantic information restorer and a de-noising enhancement module based on the received semantic features, the received channel state and the multi-scale reference context features. According to the method, the semantic features of the video frames are extracted and recovered by using the module constructed by the CNN, and the semantic features of the video frames are processed by using the coding and decoding module constructed by the Swinin-Transform, so that the coding and decoding efficiency is improved, the understanding ability of the model to the video content is enhanced, the high-quality video content can be better reconstructed, and the size of the model parameter is controlled.

Description

Technical Field

[0001] The present application relates to the field of semantic communication technology, and in particular to a semantic communication method and system for ultra-high-definition video. Background Art

[0002] The amount of data required to be transmitted for ultra-high-definition video-related services is extremely large, which will bring huge communication overhead. It is necessary to implement a compact coding and transmission solution to reduce the use of channel bandwidth.

[0003] Traditional image / video communication systems with a separate design have the following problems when transmitting ultra-high-definition video: under poor channel conditions or limited bandwidth, once there is a mismatch between the communication transmission bit rate and the channel capacity, it will lead to a cliff effect, that is, the communication performance will drop sharply when the channel capacity is lower than the communication transmission bit rate, and the receiving end is often unable to recover the image / video data; the system is only an independent optimization at the symbol level. Under poor channel conditions or limited communication bandwidth, the accuracy at the symbol level is difficult to effectively guarantee the accuracy at the source semantic level.

[0004] Based on this, a new semantic communication method for ultra-high-definition video transmission is urgently needed. Summary of the invention

[0005] The present application provides a semantic communication method and system for ultra-high-definition video to solve the above-mentioned problems.

[0006] In a first aspect of an embodiment of the present application, a semantic communication method for ultra-high-definition video is provided, the method comprising: Inputting the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; Performing a hyper-prior distribution estimation on the semantic feature to obtain a hyper-prior; Inputting the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior; Evenly dividing the semantic features into four groups along the channel dimension to obtain four groups of feature elements; Obtaining an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; Obtaining four-step semantic importance based on the channel state, the four groups of feature elements and the four-step spatial domain prior estimation; Based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; The semantic feature encoding result of each channel is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping modules of each channel at the receiving end restore the symbols according to their dimensions; For the i-th channel among the four channels, the channel state, the received i-th channel semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th channel semantic feature decoder to obtain the i-th channel semantic feature decoding result, and the four-channel semantic feature decoding results constitute the received semantic feature; Inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features; The noisy semantic feature, the channel state and a first reference context feature in the multi-scale reference context feature of the previous video P frame are input into a denoising enhancement module to reconstruct a video P frame.

[0007] In an optional embodiment of the present application, when a quadtree-based spatial entropy model is used for step-by-step prior estimation, each group of characteristic elements in each step of the step-by-step prior estimation is different; The prior used by the feature element of step 0 is the initial fusion prior, which is generated by the super prior, the time domain prior and the semantic features of the previous video P frame; The feature elements in step 1 will use the priors estimated based on the feature elements in step 0 and the initial fusion priors; The feature elements in step 2 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, and the initial fusion priors; The feature elements in step 3 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, the feature elements in step 2, and the initial fusion priors; The prior used in each step of the characteristic element is used to estimate the symbol length corresponding to the encoding of the characteristic element of this step.

[0008] In an optional embodiment of the present application, based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, including: Using the initial fusion prior as the spatial domain estimation result of the zero-step spatial domain estimation; In the process of the first to third steps of spatial domain estimation, the estimated distribution features obtained in the previous steps and the initial fusion prior are used as references, and the estimation is completed using a network with shared weights to obtain the spatial domain prior for each step.

[0009] In an optional embodiment of the present application, the generation process of the multi-scale reference context features used in the video P frame semantic feature extraction and recovery process is divided into three steps: Step 1: Input the reference video frame into the multi-scale reference feature generation module to extract the outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include the Xth multi-scale reference features; Step 2: inputting the multi-scale reference features and the reference video frame into a coarse-grained reference context generation module to obtain a multi-scale coarse-grained reference context feature, wherein the multi-scale coarse-grained reference context feature includes an Xth multi-scale coarse-grained reference feature; Step 3: Input the multi-scale coarse-grained reference context features and the channel state into a refinement network to obtain the multi-scale reference context features, wherein the multi-scale reference context features include an Xth reference context feature, and the value range of X is 1-3.

[0010] In an optional embodiment of the present application, the method further includes: Input the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame; Inputting the motion feature into a super prior estimation module to obtain a super prior; generating a motion fusion prior based on the super prior and motion features of a reference video frame; Inputting the motion fusion prior and the motion feature into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate; Inputting the motion feature prior distribution estimate, the channel state and the motion feature into a motion importance estimation module to obtain four-step motion importance; For the i-th path among the four paths, the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result; The motion feature encoding result of each channel is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping modules of each channel at the receiving end restore the symbols according to their dimensions; For the i-th path among the four paths, the channel state, the received i-th path motion feature encoding result, and the motion feature decoding results of each path before the i-th path are input into the four-path motion feature decoder to obtain the i-th path motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature; The received motion feature, the channel state and the motion reference feature of the reference video frame are input into a motion information restorer to restore a motion vector.

[0011] In an optional embodiment of the present application, the motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module; Inputting the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, including: Input the to-be-encoded video P frame and the reference video frame into the SpyNet module to perform optical flow estimation to obtain a motion vector; The motion vector, the channel state and the motion reference feature of the reference video frame are input into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

[0012] In an optional embodiment of the present application, the channel perception module includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; Inputting the motion vector, the channel state, and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the P frame of the video to be encoded, including: Inputting the motion vector into the first down-sampling module to obtain a first down-sampling result; Inputting the first downsampling result and the channel state into the channel attention module to obtain a channel attention result; Based on the first downsampling result and the channel attention result, using the spatial attention module, obtaining a spatial attention result; Obtaining the channel perception result based on the first downsampling result and the spatial attention result; The channel sensing result and the motion reference feature of the reference video frame are input into the second down-sampling module to obtain the motion feature of the P frame of the video to be encoded.

[0013] In an optional embodiment of the present application, each motion feature decoder includes: a channel sensing module and a motion feature decoding module, and the motion feature decoding module includes: a first operator layer and a second operator layer; Inputting the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel before the i-th motion feature into a four-channel motion feature decoder to obtain the i-th motion feature decoding result, including: Input the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result; According to the first processing result and the channel state, the channel perception module and the second operator layer are used to obtain the i-th motion feature decoding result.

[0014] In an optional embodiment of the present application, the motion information restorer includes: a channel sensing module, a first upsampling module, and a second upsampling module; Inputting the received motion feature, the channel state and the motion reference feature of the reference video frame into a motion information restorer to restore a motion vector, including: Inputting the received motion feature into the first upsampling module to obtain a first upsampling result; Inputting the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of a P frame of the video to be encoded; The motion vector is restored by utilizing the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

[0015] In a second aspect of an embodiment of the present application, a semantic communication system for ultra-high-definition video is provided, the system comprising: The semantic feature extraction module is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; A semantic super-prior estimation module, used for estimating the super-prior distribution of the semantic features to obtain a super-prior; A time domain prior estimation module, used for inputting the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior; A feature segmentation module, used for evenly segmenting the semantic features into four groups along the channel dimension to obtain four groups of feature elements; An initial fusion module, used to obtain an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; A spatial domain prior estimation module, used for performing step-by-step prior estimation based on the four groups of feature elements and the initial fusion prior using a quadtree-based spatial domain entropy model to obtain a four-step spatial domain prior; A semantic importance estimation module, used for obtaining four-step semantic importance based on the channel state, the four groups of characteristic elements and the four-step spatial domain prior estimation; A semantic feature encoding module, configured to obtain a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; A first wireless transmission module is used to map the semantic feature encoding result of each channel into symbols of different lengths for wireless transmission through the symbol mapping modules corresponding to each channel; A first restoration module, used for each symbol mapping module of the receiving end, restores the symbol according to the dimension of the symbol; The semantic feature decoding module is used for inputting the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel among the four channels, so as to obtain the semantic feature decoding result of the i-th channel, and the four semantic feature decoding results constitute the received semantic feature; A semantic information recovery module, used for inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to recover the noisy semantic features; The reconstruction module is used to input the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context feature of the previous video P frame into the denoising and enhancement module to reconstruct the video P frame.

[0016] The present application includes the following advantages: an embodiment of the present application provides a semantic communication method and system for ultra-high-definition video, wherein the multi-scale reference context features of the video P frame to be encoded, the multi-scale reference context features of the previous video P frame, and the channel state are input into a video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; a super-prior distribution estimation is performed on the semantic features to obtain a super-prior; the feature with the smallest scale in the multi-scale reference context features of the previous video P frame is input into a time domain prior estimation module to obtain a time domain prior; the semantic features are evenly divided into four groups along the channel dimension to obtain four groups of feature elements; an initial fusion prior is obtained based on the super-prior, the time domain prior, and the semantic features of the previous video P frame; based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; based on the channel state, the four groups of feature elements, and the four-step spatial prior estimation, a four-step semantic importance is obtained; based on the initial fusion prior and the four groups of feature elements, a four-way semantic A feature encoder is provided to obtain four-channel semantic feature encoding results, wherein the four-channel semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; each channel semantic feature encoding result is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; each channel symbol mapping module of the receiving end restores the symbol according to the dimension of the symbol; for the i-th channel among the four channels, the channel state, the received i-th channel semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th channel semantic feature decoder to obtain the i-th channel semantic feature decoding result, and the four-channel semantic feature decoding results constitute the received semantic feature; the received semantic feature, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic feature; the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context features of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The module built using CNN is used to extract and restore the semantic features of video frames, and the codec module built using Swin-Transformer is used to process the semantic features of video frames. This not only improves the coding and decoding efficiency, but also enhances the model's ability to understand video content, helps to better reconstruct high-quality video content, and control the size of model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a flowchart of the steps of a semantic communication method of ultra-high-definition video provided in an embodiment of the present application; Figure 2 It is a flowchart of a semantic communication method of ultra-high-definition video provided in an embodiment of the present application; Figure 3 This is an architecture diagram of a semantic communication system for ultra-high-definition video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to solve the following problems caused by the traditional image / video communication system with a separate design during ultra-high-definition video transmission: under poor channel conditions or limited bandwidth, once a mismatch occurs between the communication transmission bit rate and the channel capacity, it will lead to a cliff effect, that is, the communication performance will drop sharply when the channel capacity is lower than the communication transmission bit rate, and the receiving end is often unable to recover the image / video data; the system is only an independent optimization at the symbol level. Under poor channel conditions or limited communication bandwidth, the accuracy of the symbol level is difficult to effectively guarantee the accuracy of the source semantic level. The present application proposes a semantic communication method for ultra-high-definition video to solve the above problems. In video stream transmission and storage, common video compression image frames include I frames, P frames and B frames, among which I frames contain more information and can be independently encoded and decoded. Any I frame can be randomly decoded to play the video stream, while P frames contain relatively less information and cannot be independently encoded and decoded. Since P frames use motion prediction to perform inter-frame prediction coding, only prediction errors and motion vectors are transmitted, so transmission redundancy can be significantly reduced. In this application, the image frames in the ultra-high-definition video transmitted are P frames.

[0021] See also Figure 1 , Figure 1 : is a flowchart of a method for semantic communication of ultra-high-definition video provided in an embodiment of the present application, the method comprising the following steps: Step 101: inputting the multi-scale reference context features of the video P frame to be encoded and the previous video P frame and the channel state into a video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; Step 102: Estimating the hyper-prior distribution of the semantic feature to obtain a hyper-prior; Step 103: inputting the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior; Step 104: Evenly divide the semantic features into four groups along the channel dimension to obtain four groups of feature elements; Step 105: obtaining an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; Step 106: Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; Step 107: Obtain four-step semantic importance based on the channel state, the four groups of feature elements and the four-step spatial domain prior estimation; Step 108: Based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; Step 109: Mapping the semantic feature encoding result of each channel into symbols of different lengths for wireless transmission through the symbol mapping module corresponding to each channel; Step 110: The symbol mapping modules of each channel at the receiving end restore the symbols according to the dimensions of the symbols; Step 111: for the i-th channel among the four channels, the channel state, the received i-th channel semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th channel semantic feature decoder to obtain the i-th channel semantic feature decoding result, and the four-channel semantic feature decoding results constitute the received semantic feature; Step 112: input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features; Step 113: input the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context feature of the previous video P frame into a denoising enhancement module to reconstruct the video P frame.

[0022] In order to clearly illustrate the semantic communication method of ultra-high-definition video proposed in this application, Figure 2 To explain, Figure 2 It is a flow chart of a semantic communication method for ultra-high-definition video provided in an embodiment of the present application.

[0023] When implementing step 101, first obtain a frame of the ultra-high-definition video to be transmitted as a P frame of the video to be encoded, wherein the ultra-high-definition video refers to a video with a resolution of 4k or 8k or above. Combined with the multi-scale reference context features of the previous video P frame and the channel state, the video P frame semantic feature extractor is used to obtain the semantic features of the video P frame to be encoded. The multi-scale reference context features of the previous video P frame correspond to Figure 2 The source residual reference in .

[0024] In an optional embodiment of the present application, the video P frame semantic feature extractor includes multiple Conv2d layers, ResBlock layers and a channel perception module (AF Module). and the first reference context feature in the multi-scale reference context feature of the previous video P frame Input the first Conv2d layer to obtain the first processing result, and combine the first processing result with the second reference context feature in the multi-scale reference context feature of the previous video P frame Input the first ResBlock layer to obtain the second processing result, input the channel state SNR into the channel sensing module (AFModule) to obtain the channel sensing result, and based on the second processing result, the channel sensing result and the second reference context feature in the multi-scale reference context feature of the previous video P frame , using the remaining layers in the video P frame semantic feature extractor to obtain semantic features .

[0025] When the step 102 is specifically implemented, a hyper-prior distribution estimation is performed on the semantic features to obtain a hyper-prior; in order to compact the semantic features in the present application, a hyper-prior is introduced to make a preliminary estimate of the potential distribution of the semantic features, and a hyper-prior distribution estimation is performed through a single neural network model. A hyper-prior distribution estimation is performed on the above-mentioned semantic features to obtain a hyper-prior. Specifically, the above-mentioned semantic features are input into a hyper-prior estimation module to obtain the above-mentioned hyper-prior. The hyper-prior estimation module is only deployed at the transmitting end, and the transmitting end will not transmit the hyper-prior.

[0026] When step 103 is specifically implemented, the feature with the smallest scale among the multi-scale reference context features of the previous video P frame is input into the time domain prior estimation module to obtain the time domain prior. The time domain prior estimation module includes two Conv2d layers and one Leaky Relu layer. Figure 2 The temporal prior estimation module estimates the temporal prior distribution of the semantic features to obtain the temporal prior. Specifically, the feature with the smallest scale among the multi-scale reference context features of the previous video P frame is Input the hyper-prior estimation module to obtain the temporal prior.

[0027] When implementing step 104, the entropy model of the quadtree is introduced in the present application to perform subsequent spatial domain prior distribution estimation and semantic importance estimation. First, the extracted semantic features are evenly divided into four groups along the channel dimension to obtain four groups of feature elements, and the four groups of feature elements are used for subsequent spatial domain prior distribution estimation and semantic importance estimation. The channel dimension of the semantic feature is channel, which can be abbreviated as C.

[0028] When step 105 is specifically implemented, when the video P frame is transmitted, when the previous frame of the video P frame to be encoded is a P frame, an initial fusion prior is obtained according to the super prior, the time domain prior and the semantic features of the previous video P frame; when the previous frame of the video P frame to be encoded is an I frame, the initial fusion prior of the video P frame to be encoded is generated by the super prior and the time domain prior.

[0029] When step 106 is specifically implemented, a quadtree-based spatial entropy model is used to estimate the spatial prior distribution and semantic importance. The quadtree-based spatial entropy model can fully identify the redundancy of spatial information in semantic features. First, based on four groups of feature elements and the above-mentioned initial fusion super-prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior. According to the step-by-step estimation strategy of the entropy model, a four-way semantic feature coding transmission link with mutual reference is realized; the output result of the entropy model is used to estimate the semantic importance to guide the feature encoder to fully reduce the channel bandwidth occupancy and realize unequal error protection transmission.

[0030] In an optional embodiment of the present application, when a quadtree-based spatial entropy model is used for step-by-step prior estimation, each group of feature elements in each step-by-step prior estimation is different; specifically as follows: the prior used by the feature element of step 0 is the initial fusion prior, and the initial fusion prior is generated based on the super prior, the time domain prior, and the semantic features of the previous video P frame; the prior used by the feature element of step 1 is estimated based on the feature element of step 0 and the initial fusion prior; the prior used by the feature element of step 2 is estimated based on the feature element of step 0, the feature element of step 1, and the initial fusion prior; the prior used by the feature element of step 3 is estimated based on the feature element of step 0, the feature element of step 1, the feature element of step 2, and the initial fusion prior; since the prior used by the feature element of each step is different, the feature element of each step is also different. The prior used by the feature element of each step is used to estimate the symbol length corresponding to the feature element of that step after encoding.

[0031] In an optional embodiment of the present application, based on four groups of feature elements and the above-mentioned initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, specifically including: first, using the above-mentioned initial fusion prior as the spatial estimation result of the zeroth step spatial estimation; then, in the process of the first to third steps of spatial estimation, the estimated distribution characteristics obtained in the previous steps and the initial fusion prior are used as references, and a network with shared weights is used to complete the estimation to obtain the spatial prior for each step.

[0032] In an optional embodiment of the present application, the spatial domain prior distribution estimation is divided into four steps, which can be referred to Figure 2 , along the channel dimension C, the dimension is The semantic features are evenly divided into four groups of dimensions: Each group will estimate the feature element at the corresponding index at each step. C represents the channel dimension, H represents the height dimension, and W represents the width dimension. For example, step 0 ( Figure 2 Step 0) Each group will Figure 2 The position with index 0 in the image is estimated. The indexes of each group are different, but the indexes of each group can cover the dimension [H, W] when combined. The indexes of each step are also different. The combination of feature elements used in each step of estimation is all feature elements corresponding to the estimated index of each step. Therefore, each step of spatial prior distribution estimation can cover each spatial position of dimension [H, W], and the estimated feature elements can form features of dimension [C / 4, H, W]. In the subsequent first to third steps, the spatial prior distribution estimation will use the feature elements of the estimated distribution in the previous steps for reference, so that the position to be estimated in the same group in each step can refer to its neighbors in the same group and the same positions in different groups as references, thereby guiding the model to fully identify the spatial information redundancy in the semantic features. The output parameters of the entropy model will also be used to guide the semantic feature encoder to achieve unequal error protection. Based on the semantic importance estimation module, the prior distribution parameters estimated by the entropy model and the semantic features are input to calculate the semantic importance of each feature element. The output result will interact with the semantic feature encoding process through the cross-attention layer in the semantic feature encoder.

[0033] Specifically, when performing semantic importance estimation, the channel state, the above four groups of feature elements and the above four-step spatial domain priors are input into the semantic importance estimation module to estimate the four-step semantic importance. The semantic importance estimation module used in each step includes multiple DepthConvBlock layers, multiple ResBlock layers and a channel perception module (AFModule).

[0034] When step 108 is specifically implemented, based on the above-mentioned initial fusion prior and the above-mentioned four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result. The above-mentioned four-way semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; the feature spatial domain composition dimension estimated according to each step of the spatial domain prior distribution is The feature elements of the spatial domain prior distribution estimation are used to construct a four-way semantic feature encoder, and the features composed of each step of the spatial domain prior distribution estimation are encoded respectively to obtain the four-step semantic feature encoding results. The self-attention layer of Swin-Transformer and the cross-attention layer based on Swin-Transformer-V2 are used to construct a neural network model to remove the information redundancy of the semantic features identified in the spatial domain distribution estimation. The network structure of each semantic feature encoder is the same, and each semantic feature encoder includes: a channel perception module and a semantic feature encoding module, and the above-mentioned semantic feature encoding module includes: a first cross-attention layer and a second cross-attention layer, and also includes a first self-attention layer, a second self-attention layer and a DepthConvBlock layer.

[0035] When implementing step 109 and step 110, after obtaining the i-th semantic feature encoding result, the corresponding symbol mapping module of each channel maps each semantic feature encoding result into symbols of different lengths for wireless transmission; then, the symbol mapping module of each channel at the receiving end restores the symbol according to the dimension of the symbol, and decodes the restored encoding result. The dimension of the output result of the semantic feature after passing through the feature encoder is , for the dimension Each element of has a length of , according to the hyper-prior distribution estimated by the entropy model, the symbol length required for the element is calculated, and a linear layer matching the symbol length is selected for mapping. At the receiving end, the corresponding linear layer will be used to map it back to length , thereby recovering the dimension from the received symbol The coded transmission result is then transmitted to the semantic feature decoder. Based on the symbol mapping module, a semantic communication method for ultra-high-definition video is implemented, which dynamically adjusts the video codec output bit rate according to the channel state and semantic information.

[0036] When step 111 is specifically implemented, for the i-th path among the four paths, the above-mentioned channel state, the received i-th semantic feature encoding result, and the semantic feature decoding results of each path before the i-th path are input into the i-th semantic feature decoder to obtain the i-th semantic feature decoding result, and the four semantic feature decoding results constitute the received semantic feature; each semantic feature decoder includes: a channel perception module and a semantic feature decoding module, and the semantic feature decoding module includes: a third cross-attention layer, and also includes: a third self-attention layer, a fourth self-attention layer and a DepthConvBlock layer.

[0037] When step 112 is specifically implemented, after the semantic features are decoded to obtain the received semantic features, the received semantic features are subjected to semantic information reconstruction processing, which is implemented in the present application by a semantic information restorer and a denoising enhancement module. The received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic features.

[0038] In an optional embodiment of the present application, the semantic information restorer includes: multiple upsampling modules (ResBlock Upsample), two ResBlock layers and a channel perception module (AF Module). First, the received semantic features Input the first upsampling module to obtain the upsampling result; based on the upsampling result and the third reference context feature of the previous video P frame , using the first ResBlock layer, obtain a reference processing result; based on the channel state SNR and the reference processing result, use the channel sensing module to obtain a channel sensing processing result; based on the channel sensing processing result and the second reference context feature , use the remaining modules to obtain noisy semantic features .

[0039] When step 113 is specifically implemented, the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context feature of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The multi-scale reference context feature of the previous video P frame used in steps 112 and 113 corresponds to Figure 2 The receiving end context in .

[0040] In an optional embodiment of the present application, the denoising enhancement module includes: two Conv2d layers, two Unet layers and a channel perception module (AF Module). First, the noisy semantic features are Input the first Conv2d layer to obtain the first denoising result; based on the first denoising result and the channel perception state SNR, use the channel perception module and the residual module to obtain the video P frame .

[0041] In an optional embodiment of the present application, the generation process of the multi-scale reference context features used in the video P frame semantic feature extraction and recovery process is divided into three steps. Specifically, in the first step, the reference video frame The input multi-scale reference feature generation module extracts the output of different network layers to obtain the X-th multi-scale reference feature, where the value range of X is 1-3, and the X-th multi-scale reference feature includes the first multi-scale reference feature , the second multi-scale reference feature and the third multi-scale reference feature ; In the second step, the above multi-scale reference features , and , and the above reference video frame Input the coarse-grained reference context generation module to obtain the Xth multi-scale coarse-grained reference context feature, wherein the multi-scale coarse-grained reference context feature includes the first multi-scale coarse-grained reference context feature , the second multi-scale coarse-grained reference context feature and the third multi-scale coarse-grained reference context feature .

[0042] The roughness reference context features obtained above need to be refined by the neural network before they can be used. The refined network is based on the residual network, which guides the network to learn and supplement the fine-grained reference information, and introduces the channel perception module. In addition, the reference context features of different scales refer to each other during the generation process. In the third step, the multi-scale roughness reference context features and the channel state are input into the refinement network to obtain the Xth reference context feature, where the value range of X is 1-3. The multi-scale reference context features include the first reference context feature , the second reference context feature and the third reference context feature .

[0043] In an optional embodiment of the present application, the method further includes the following steps: Step 201: input the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame; Step 202: input the motion feature into a hyper-prior estimation module to obtain a hyper-prior; Step 203: Generate a motion fusion prior based on the super prior and the motion features of the reference video frame; Step 204: input the motion fusion prior and the motion feature into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate; Step 205: inputting the motion feature prior distribution estimation, the channel state and the motion feature into a motion importance estimation module to obtain four-step motion importance; Step 206: for the i-th path among the four paths, input the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path into the four-path motion feature encoder to obtain the i-th path motion feature encoding result; Step 207: Mapping the motion feature encoding result of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; Step 208: The symbol mapping module of each channel of the receiving end restores the symbol according to the dimension of the symbol; Step 209: for the i-th path among the four paths, input the channel state, the received i-th path motion feature encoding result, and the motion feature decoding results of each path before the i-th path into the four-path motion feature decoder to obtain the i-th path motion feature decoding result, and compose the received motion feature with the four-path motion decoding results; Step 210: Input the received motion feature, the channel state and the motion reference feature of the reference video frame into a motion information restorer to restore the motion vector.

[0044] When step 201 is specifically implemented, the above-mentioned video P frame to be encoded and the reference video frame are input into a motion information extractor to obtain the motion features of the above-mentioned video P frame to be encoded, and the above-mentioned reference video frame is the previous video P frame of the above-mentioned video P frame to be encoded.

[0045] In an optional embodiment of the present application, the network structure of the motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module; the motion information extractor also includes: a ResBlock Downsample layer, a DepthConvBlock layer and a Conv2d layer. Based on the above network structure, the P frame of the video to be encoded and the reference video frame are input into the motion information extractor to obtain the motion features of the P frame of the video to be encoded, specifically including: firstly, the P frame of the video to be encoded is input into the motion information extractor; and reference video frame , input the above SpyNet module to perform optical flow estimation and obtain the motion vector , and then the motion vector , the above channel state (e.g. SNR) and the motion reference features of the reference video frame Input the above downsampling module and the above channel sensing module for downsampling and channel sensing to obtain the motion features of the above video P frame to be encoded .

[0046] In an optional embodiment of the present application, the channel perception module (AF Module) includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; the motion vector, the channel state, and the motion reference feature of the reference video frame are input into the downsampling module and the channel perception module for downsampling and channel perception to obtain the motion feature of the P frame of the video to be encoded, specifically including: the motion vector Input the first downsampling module to obtain a first downsampling result; input the first downsampling result and the channel state (such as SNR) into the channel attention module to obtain a channel attention result; based on the first downsampling result and the channel attention result, use the spatial attention module to obtain a spatial attention result; based on the first downsampling result and the spatial attention result, obtain the channel perception result; and compare the channel perception result and the motion reference feature of the reference video frame. Input the second downsampling module to obtain the motion features of the P frame of the video to be encoded. By integrating the channel perception module, the system can adaptively adjust the encoding and decoding strategy according to the real-time channel conditions (for example, SNR), and strengthen the important features of the video data with the attention mechanism, thereby improving the overall transmission efficiency and quality. Among them, the above motion reference features are the output results of the intermediate network layer of the motion information recovery module.

[0047] When step 202 is specifically implemented, the above motion features are input into the super prior estimation module to obtain super prior; the process of super prior estimation of motion features is the same as the process in step 102, the difference being that the input object is motion features, not semantic features.

[0048] When step 203 is specifically implemented, a motion fusion prior is generated based on the above-mentioned super prior and the motion features of the reference video frame; and the motion fusion prior is used for subsequent motion prior estimation based on the quadtree-based spatial entropy model.

[0049] When the step 204 is specifically implemented, the spatial information redundancy in the motion features can be fully identified by the spatial entropy model based on the quadtree. Based on the above motion fusion prior and the above motion features, the spatial entropy model based on the quadtree is input for step-by-step prior estimation to obtain a four-step motion feature prior distribution estimation. The process of performing distribution prior estimation based on the quadtree cross entropy model is the same as the process of performing distribution prior estimation on the semantic features, except that the motion features and motion fusion prior are input.

[0050] Specifically, when performing motion importance estimation, the motion feature prior distribution estimation, the channel state and the motion feature are input into the motion importance estimation module to estimate the four-step motion importance. The motion importance estimation module used in each step includes multiple DepthConvBlock layers, multiple ResBlock layers and a channel perception module (AF Module).

[0051] When step 206 is specifically implemented, the motion features are divided into four groups of feature elements according to the channels. The index used in each step of estimation is different. The feature element combination used in each step of estimation is all the feature elements corresponding to the estimation index of each step. The feature element combination corresponding to the estimation index of the i-th step is the feature element combination corresponding to the estimation index of the i-th step (the feature elements in the feature element combination are obtained by dividing the motion features according to the channels). Based on the motion importance and the feature element combination, a four-way motion feature encoder is used to obtain the motion feature encoding result of each way.

[0052] Each motion feature encoder includes: a channel perception module and a motion feature encoding module, and the motion feature encoding module includes: a first Conv2d layer, a second DepthConvBlock layer, a third ResBlock layer and a fourth ResBlock layer; based on the above network structure, for the i-th path in the four paths, the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the above channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result, specifically including: the i-th path motion feature encoding result , the decoding results of motion features of each channel before the i-th channel and the importance of the motion at step i , input the first Conv2d layer to obtain the first encoding result; based on the first encoding result and the channel state (such as SNR), use the second DepthConvBlock layer and the channel perception module (AF Module) to obtain the second encoding result; based on the second encoding result, use the third ResBlock layer and the fourth ResBlock layer to obtain the i-th motion feature encoding result .

[0053] When the steps 207 and 208 are specifically implemented, the motion feature coding results of each channel are mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; the symbol mapping modules of each channel at the receiving end restore the symbols according to the dimensions of the symbols and decode the restored coding results. The symbol mapping module is the same as the description of the specific implementation steps 109 and 110, except that the object of transmission restoration is the motion feature coding result.

[0054] When step 209 is specifically implemented, for the i-th path among the four paths, the above-mentioned channel state, the received i-th motion feature encoding result, and the motion feature decoding results of each path before the i-th path are input into the four-path motion feature decoder to obtain the i-th motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature; based on the received motion feature encoding result, a four-path motion feature encoder is used to obtain the motion feature encoding result of each path.

[0055] Each motion feature decoder includes: a channel sensing module and a motion feature decoding module, and the motion feature decoding module includes: a first operator layer and a second operator layer, the first operator layer is a Conv2d layer, and the second operator layer is a DepthConvBlock layer. Based on the above network structure, the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel before the i-th channel are input into the four-channel motion feature decoder to obtain the i-th motion feature decoding result, which specifically includes: the i-th motion feature encoding result received is input into the four-channel motion feature decoder. and the decoding results of motion features of each path before the i-th path Input the first operator layer to obtain a first processing result; according to the first processing result and the channel state (such as SNR), use the channel perception module (AF Module) and the second operator layer to obtain the i-th motion feature decoding result The four-way motion feature decoding results are combined into the received motion feature for subsequent motion information recovery. After the motion vector is obtained, it is processed and the processed motion vector is used as the reference motion vector of the next video.

[0056] When step 210 is specifically implemented, the receiving end recovers the motion vector by upsampling and channel-aware modulation of the received motion features. Specifically, the received motion features, the channel state and the motion reference features of the reference video frame are input into a motion information recovery device to recover the motion vector.

[0057] In an optional embodiment of the present application, the motion information recoverer includes: a channel perception module, a first upsampling module, and a second upsampling module; the first upsampling module and the second upsampling module are both composed of a DepthConvBlock2 layer and a ResBlock Upsample layer. The motion information recoverer also includes: a DepthConvBlock2 layer, a DepthConvBlock layer, and a ResBlock Upsample layer. Based on the above network structure, the above received motion features, the above channel state, and the motion reference features of the reference video frame are input into the motion information recoverer to recover the motion vector, specifically including: the above received motion features Input the first upsampling module to obtain the first upsampling result; input the first upsampling result and the channel state (such as SNR) into the channel perception module (AF Module) to obtain the second upsampling result and the motion reference feature of the P frame of the video to be encoded ; Motion reference features based on reference video frames The second upsampling module is used to recover the motion vector The motion vector recovered by the motion information restorer It contains noise and needs to be applied after offset prediction. The second upsampling result is used to extract motion information of the next video frame.

[0058] The embodiment of the present application provides a semantic communication method for ultra-high-definition video, which inputs the multi-scale reference context features of the P frame of the video to be encoded, the multi-scale reference context features of the previous P frame of the video, and the channel state into a video P frame semantic feature extractor to obtain the semantic features of the P frame of the video to be encoded; performs super-prior distribution estimation on the semantic features to obtain super-prior; inputs the feature with the smallest scale in the multi-scale reference context features of the previous P frame of the video into a time domain prior estimation module to obtain time domain prior; divides the semantic features evenly into four groups along the channel dimension to obtain four groups of feature elements; obtains an initial fusion prior based on the super-prior, the time domain prior, and the semantic features of the previous P frame of the video; based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; based on the channel state, the four groups of feature elements, and the four-step spatial prior estimation, a four-step semantic importance is obtained; based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain To four-way semantic feature encoding results, the four-way semantic feature encoder corresponds to the four-step estimation of the spatial entropy model based on the quadtree; each semantic feature encoding result is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping module of each channel; the symbol mapping modules of each channel at the receiving end restore the symbols according to the dimension of the symbols; for the i-th channel among the four channels, the channel state, the received i-th semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th semantic feature decoder to obtain the i-th semantic feature decoding result, and the four-way semantic feature decoding results constitute the received semantic feature; the received semantic feature, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame are input into the semantic information restorer to restore the noisy semantic feature; the noisy semantic feature, the channel state, and the first reference context feature in the multi-scale reference context features of the previous video P frame are input into the denoising enhancement module to reconstruct the video P frame. The module built using CNN is used to extract and restore the semantic features of video frames, and the codec module built using Swin-Transformer is used to process the semantic features of video frames. This not only improves the coding and decoding efficiency, but also enhances the model's ability to understand video content, helps to better reconstruct high-quality video content, and control the size of model parameters.

[0059] In a second aspect of the embodiment of the present application, a semantic communication system for ultra-high-definition video is provided. Figure 3 , Figure 3 : is an architecture diagram of a semantic communication system for ultra-high-definition video provided in an embodiment of the present application, the system comprising: The semantic feature extraction module 301 is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; A semantic super-prior estimation module 302 is used to estimate the super-prior distribution of the semantic features to obtain a super-prior; A time domain prior estimation module 303 is used to input the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior; A feature segmentation module 304 is used to evenly segment the semantic features into four groups along the channel dimension to obtain four groups of feature elements; An initial fusion module 305, configured to obtain an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; A spatial domain prior estimation module 306 is used to perform step-by-step prior estimation based on the four groups of feature elements and the initial fusion prior using a quadtree-based spatial domain entropy model to obtain a four-step spatial domain prior; A semantic importance estimation module 307, configured to obtain four-step semantic importance based on the channel state, the four groups of characteristic elements and the four-step spatial domain prior estimation; A semantic feature encoding module 308, configured to obtain a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; The first wireless transmission module 309 is used to map the semantic feature encoding result of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The first restoration module 310 is used for each symbol mapping module of the receiving end to restore the symbol according to the dimension of the symbol; The semantic feature decoding module 311 is used for inputting the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel among the four channels, so as to obtain the semantic feature decoding result of the i-th channel, and the four semantic feature decoding results constitute the received semantic feature; The semantic information recovery module 312 is used to input the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information recoverer to recover the noisy semantic features; The reconstruction module 313 is used to input the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context feature of the previous video P frame into a denoising and enhancement module to reconstruct a video P frame.

[0060] Wherein, when the spatial domain entropy model based on quadtree is used for step-by-step prior estimation in the spatial domain prior estimation module, each group of characteristic elements in each step-by-step prior estimation is different; the spatial domain prior estimation module includes: A prior estimation submodule of step 0, wherein the prior used for the feature element of step 0 is an initial fusion prior, and the initial fusion prior is generated by the super prior, the time domain prior and the semantic features of the previous video P frame; The first step prior estimation submodule uses the priors of the feature elements of the first step as estimated based on the feature elements of the 0th step and the initial fusion priors; The second step prior estimation submodule is used for the feature elements of the second step, and the priors used are obtained based on the feature elements of the 0th step, the feature elements of the 1st step, and the initial fusion prior estimation; The prior estimation submodule of step 3 uses the priors of the feature elements of step 3, which are estimated based on the feature elements of step 0, step 1, step 2 and the initial fusion priors; The prior estimation submodule is used to estimate the symbol length corresponding to the encoding of the feature element at each step using the prior information of the feature element at that step.

[0061] Wherein, the spatial domain prior estimation module further includes: An initial prior estimation submodule, used for taking the initial fusion prior as the spatial domain estimation result of the zero-step spatial domain estimation; The first to third step estimation submodule is used to use the estimated distribution features obtained in the previous steps and the initial fusion prior as references during the first to third step spatial domain estimation, and use a network with shared weights to complete the estimation to obtain the spatial domain prior for each step.

[0062] The generation process of the multi-scale reference context features used in the semantic information recovery module and the reconstruction module is implemented by the following submodules: A first generation submodule is used to input the reference video frame into a multi-scale reference feature generation module to extract outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include an Xth multi-scale reference feature; A second generation submodule is used to input the multi-scale reference feature and the reference video frame into a coarse-grained reference context generation module to obtain a multi-scale coarse-grained reference context feature, wherein the multi-scale coarse-grained reference context feature includes an Xth multi-scale coarse-grained reference feature; The third generation submodule is used to input the multi-scale coarse-grained reference context features and the channel state into a refining network to obtain the multi-scale reference context features, wherein the multi-scale reference context features include the Xth reference context features, and the value range of X is 1-3.

[0063] Wherein, the system further comprises: A motion feature extraction module, used for inputting the to-be-encoded video P frame and a reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame; A motion super-prior estimation module, used for inputting the motion features into the super-prior estimation module to obtain a super-prior; A motion fusion prior module, used to generate a motion fusion prior based on the super prior and the motion features of the reference video frame; A motion feature prior distribution estimation module, used for inputting the motion fusion prior and the motion feature into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimation; A motion importance estimation module, used for inputting the motion feature prior distribution estimation, the channel state and the motion feature into the motion importance estimation module to obtain four-step motion importance; A motion feature encoding module is used to input the feature element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path into the four-path motion feature encoder to obtain the i-th path motion feature encoding result; The second wireless transmission module is used to map the motion feature encoding result of each channel into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The second restoration module is used for each symbol mapping module of the receiving end to restore the symbol according to the dimension of the symbol; A motion feature decoding module is used to input the channel state, the received motion feature encoding result of the i-th channel, and the motion feature decoding results of each channel before the i-th channel into a four-channel motion feature decoder to obtain the i-th motion feature decoding result, and the four-channel motion decoding results form the received motion feature; The motion vector recovery module is used to input the received motion feature, the channel state and the motion reference feature of the reference video frame into a motion information recovery device to recover the motion vector.

[0064] The motion information extractor in the motion feature extraction module includes at least a SpyNet module, a channel perception module and a downsampling module; the motion feature extraction module includes: An optical flow estimation submodule, used for inputting the to-be-encoded video P frame and the reference video frame into the SpyNet module for optical flow estimation to obtain a motion vector; The downsampling and channel sensing submodule is used to input the motion vector, the channel state and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

[0065] Among them, the channel perception module in the downsampling and channel perception submodule includes a spatial attention module and a channel attention module, the downsampling module includes a first downsampling module and a second downsampling module; the downsampling and channel perception submodule includes: A first down-sampling unit, configured to input the motion vector into the first down-sampling module to obtain a first down-sampling result; A channel attention unit, configured to input the first downsampling result and the channel state into the channel attention module to obtain a channel attention result; A spatial attention unit, configured to obtain a spatial attention result by using the spatial attention module based on the first downsampling result and the channel attention result; A channel sensing unit, configured to obtain the channel sensing result based on the first downsampling result and the spatial attention result; The second down-sampling unit is used to input the channel sensing result and the motion reference feature of the reference video frame into the second down-sampling module to obtain the motion feature of the P frame of the video to be encoded.

[0066] Wherein, each motion feature decoder in the motion feature decoding module includes: a channel sensing module and a motion feature decoding module, the motion feature decoding module includes: a first operator layer and a second operator layer; the motion feature decoding module includes: A first processing submodule, configured to input the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result; The second processing submodule is used to obtain the i-th motion feature decoding result according to the first processing result and the channel state by using the channel perception module and the second operator layer.

[0067] The motion information restorer in the motion vector recovery module includes: a channel sensing module, a first upsampling module, and a second upsampling module; the motion vector recovery module includes: A first upsampling submodule, configured to input the received motion feature into the first upsampling module to obtain a first upsampling result; A channel sensing submodule, configured to input the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of a P frame of a video to be encoded; The second upsampling submodule is used to restore the motion vector by using the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

[0068] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0069] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0070] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0071] The above is a detailed introduction to the semantic communication method and system for ultra-high-definition video provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A semantic communication method for ultra-high-definition video, characterized in that: The method comprises: Inputting the multi-scale reference context features of the video P frame to be encoded, the previous video P frame, and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; Performing a hyper-prior distribution estimation on the semantic feature to obtain a hyper-prior; Inputting the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into a time domain prior estimation module to obtain a time domain prior; Evenly dividing the semantic features into four groups along the channel dimension to obtain four groups of feature elements; Obtaining an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior; Obtaining four-step semantic importance based on the channel state, the four groups of feature elements and the four-step spatial domain prior estimation; Based on the initial fusion prior and the four groups of feature elements, a four-way semantic feature encoder is used to obtain a four-way semantic feature encoding result, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; The semantic feature encoding result of each channel is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping modules of each channel at the receiving end restore the symbols according to their dimensions; For the i-th channel among the four channels, the channel state, the received i-th channel semantic feature encoding result, and the semantic feature decoding results of each channel before the i-th channel are input into the i-th channel semantic feature decoder to obtain the i-th channel semantic feature decoding result, and the four-channel semantic feature decoding results constitute the received semantic feature; Inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to restore the noisy semantic features; The noisy semantic feature, the channel state and a first reference context feature in the multi-scale reference context feature of the previous video P frame are input into a denoising enhancement module to reconstruct a video P frame.

2. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: When the quadtree-based spatial entropy model is used for step-by-step prior estimation, each group of characteristic elements in each step of the step-by-step prior estimation is different; The prior used by the feature element of step 0 is the initial fusion prior, which is generated by the super prior, the time domain prior and the semantic features of the previous video P frame; The feature elements in step 1 will use the priors estimated based on the feature elements in step 0 and the initial fusion priors; The feature elements in step 2 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, and the initial fusion priors; The feature elements in step 3 will use the priors estimated based on the feature elements in step 0, the feature elements in step 1, the feature elements in step 2, and the initial fusion priors; The prior used in each step of the characteristic element is used to estimate the symbol length corresponding to the encoding of the characteristic element of this step.

3. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: Based on the four groups of feature elements and the initial fusion prior, a quadtree-based spatial entropy model is used to perform step-by-step prior estimation to obtain a four-step spatial prior, including: Using the initial fusion prior as the spatial domain estimation result of the zero-step spatial domain estimation; In the process of the first to third steps of spatial domain estimation, the estimated distribution features obtained in the previous steps and the initial fusion prior are used as references, and the estimation is completed using a network with shared weights to obtain the spatial domain prior for each step.

4. The semantic communication method of ultra-high-definition video according to claim 1, characterized in that: The generation process of multi-scale reference context features used in the video P frame semantic feature extraction and recovery process is divided into three steps: Step 1: Input the reference video frame into the multi-scale reference feature generation module to extract the outputs of different network layers to obtain multi-scale reference features, wherein the multi-scale reference features include the Xth multi-scale reference features; Step 2: inputting the multi-scale reference features and the reference video frame into a coarse-grained reference context generation module to obtain a multi-scale coarse-grained reference context feature, wherein the multi-scale coarse-grained reference context feature includes an Xth multi-scale coarse-grained reference feature; Step 3: Input the multi-scale coarse-grained reference context features and the channel state into a refinement network to obtain the multi-scale reference context features, wherein the multi-scale reference context features include an Xth reference context feature, and the value range of X is 1-3.

5. The semantic communication method of ultra-high definition video according to claim 1, characterized in that: The method further comprises: Input the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, wherein the reference video frame is a previous video P frame of the to-be-encoded video P frame; Inputting the motion feature into a super prior estimation module to obtain a super prior; generating a motion fusion prior based on the super prior and motion features of a reference video frame; Inputting the motion fusion prior and the motion feature into a quadtree-based spatial entropy model to obtain a motion feature prior distribution estimate; Inputting the motion feature prior distribution estimate, the channel state and the motion feature into a motion importance estimation module to obtain four-step motion importance; For the i-th path among the four paths, the characteristic element combination corresponding to the i-th step estimation index, the i-th step motion importance, the channel state and the motion feature encoding results of each path before the i-th path are input into the four-path motion feature encoder to obtain the i-th path motion feature encoding result; The motion feature encoding result of each channel is mapped into symbols of different lengths for wireless transmission through the corresponding symbol mapping modules of each channel; The symbol mapping modules of each channel at the receiving end restore the symbols according to their dimensions; For the i-th path among the four paths, the channel state, the received i-th path motion feature encoding result, and the motion feature decoding results of each path before the i-th path are input into the four-path motion feature decoder to obtain the i-th path motion feature decoding result, and the four-path motion decoding results are combined into the received motion feature; The received motion feature, the channel state and the motion reference feature of the reference video frame are input into a motion information restorer to restore a motion vector.

6. The semantic communication method of ultra-high definition video according to claim 5, characterized in that: The motion information extractor includes at least a SpyNet module, a channel perception module and a downsampling module; Inputting the to-be-encoded video P frame and the reference video frame into a motion information extractor to obtain motion features of the to-be-encoded video P frame, including: Input the to-be-encoded video P frame and the reference video frame into the SpyNet module to perform optical flow estimation to obtain a motion vector; The motion vector, the channel state and the motion reference feature of the reference video frame are input into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the video P frame to be encoded.

7. The semantic communication method of ultra-high-definition video according to claim 6, characterized in that: The channel perception module includes a spatial attention module and a channel attention module, and the downsampling module includes a first downsampling module and a second downsampling module; Inputting the motion vector, the channel state, and the motion reference feature of the reference video frame into the downsampling module and the channel sensing module for downsampling and channel sensing to obtain the motion feature of the P frame of the video to be encoded, including: Inputting the motion vector into the first down-sampling module to obtain a first down-sampling result; Inputting the first downsampling result and the channel state into the channel attention module to obtain a channel attention result; Based on the first downsampling result and the channel attention result, using the spatial attention module, obtaining a spatial attention result; Obtaining the channel perception result based on the first downsampling result and the spatial attention result; The channel sensing result and the motion reference feature of the reference video frame are input into the second down-sampling module to obtain the motion feature of the P frame of the video to be encoded.

8. The semantic communication method of ultra-high-definition video according to claim 5, characterized in that: Each motion feature decoder includes: a channel sensing module and a motion feature decoding module, and the motion feature decoding module includes: a first operator layer and a second operator layer; Inputting the channel state, the received i-th motion feature encoding result, and the i-th motion feature decoding results of each channel before the i-th motion feature into a four-channel motion feature decoder to obtain the i-th motion feature decoding result, including: Input the received i-th motion feature encoding result and the i-th motion feature decoding results of each path before the i-th path into the first operator layer to obtain a first processing result; According to the first processing result and the channel state, the channel perception module and the second operator layer are used to obtain the i-th motion feature decoding result.

9. The semantic communication method of ultra-high definition video according to claim 5, characterized in that: The motion information restorer comprises: a channel sensing module, a first upsampling module, and a second upsampling module; Inputting the received motion feature, the channel state and the motion reference feature of the reference video frame into a motion information restorer to restore a motion vector, including: Inputting the received motion feature into the first upsampling module to obtain a first upsampling result; Inputting the first upsampling result and the channel state into the channel sensing module to obtain a second upsampling result and a motion reference feature of a P frame of the video to be encoded; The motion vector is restored by utilizing the second upsampling module based on the motion reference feature of the reference video frame and the second upsampling result.

10. A semantic communication system for ultra-high definition video, characterized in that: The system comprises: The semantic feature extraction module is used to input the multi-scale reference context features of the video P frame to be encoded, the previous video P frame and the channel state into the video P frame semantic feature extractor to obtain the semantic features of the video P frame to be encoded; A semantic super-prior estimation module, used for estimating the super-prior distribution of the semantic features to obtain a super-prior; A time domain prior estimation module, used for inputting the feature with the smallest scale among the multi-scale reference context features of the previous video P frame into the time domain prior estimation module to obtain a time domain prior; A feature segmentation module, used for evenly segmenting the semantic features into four groups along the channel dimension to obtain four groups of feature elements; An initial fusion module, used to obtain an initial fusion prior according to the super prior, the time domain prior and the semantic features of the previous video P frame; A spatial domain prior estimation module, used for performing step-by-step prior estimation based on the four groups of feature elements and the initial fusion prior using a quadtree-based spatial domain entropy model to obtain a four-step spatial domain prior; A semantic importance estimation module, used for obtaining four-step semantic importance based on the channel state, the four groups of characteristic elements and the four-step spatial domain prior estimation; A semantic feature encoding module, configured to obtain a four-way semantic feature encoding result based on the initial fusion prior and the four groups of feature elements using a four-way semantic feature encoder, wherein the four-way semantic feature encoder corresponds to a four-step estimation of a quadtree-based spatial entropy model; A first wireless transmission module is used to map the semantic feature encoding result of each channel into symbols of different lengths for wireless transmission through the symbol mapping modules corresponding to each channel; A first restoration module, used for each symbol mapping module of the receiving end, restores the symbol according to the dimension of the symbol; The semantic feature decoding module is used for inputting the channel state, the received semantic feature encoding result of the i-th channel, and the semantic feature decoding results of each channel before the i-th channel into the i-th semantic feature decoder for the i-th channel among the four channels, so as to obtain the semantic feature decoding result of the i-th channel, and the four semantic feature decoding results constitute the received semantic feature; A semantic information recovery module, used for inputting the received semantic features, the channel state, and the second and third reference context features in the multi-scale reference context features of the previous video P frame into a semantic information restorer to recover the noisy semantic features; The reconstruction module is used to input the noisy semantic feature, the channel state and the first reference context feature in the multi-scale reference context feature of the previous video P frame into the denoising and enhancement module to reconstruct the video P frame.

Images