Adaptive rate wireless image communication method based on channel attention and mask auto-encoder

Abstract

The invention provides an adaptive rate wireless image communication method based on channel attention and a mask auto-encoder, which comprises the following steps of: preprocessing an original image data set to obtain a preprocessed image; inputting the preprocessed image into a semantic encoder, sequentially performing down-sampling, feature extraction and channel noise self-adaptive encoding, and iterating for a plurality of rounds to obtain encoding features; performing channel-level adaptive mask processing on the coding features according to the compression ratio to obtain unmasked semantic features to be transmitted; carrying out power normalization on the unmasked semantic features, and sending the unmasked semantic features to a physical channel for transmission to obtain channel output features; zero filling is carried out on the channel output features, the channel output features are input into a semantic decoder after dimension recovery, and an output image is obtained through reconstruction; and calculating a loss function according to the output image and the original image, and performing model parameter optimization through the loss function. According to the invention, bandwidth occupation and transmission delay can be significantly reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of wireless image semantic communication systems, and particularly relates to an adaptive rate wireless image communication method based on channel attention and mask autoencoder. BACKGROUND

[0002] In the current rapid development of wireless communication technology, image as the core carrier of information transmission, its transmission quality and efficiency directly affect the application effect of key fields such as space exploration, medical research and high-precision monitoring, and also relate to the quality of public work and life. However, the contradiction between the limited bandwidth resources of the current wireless channel and the massive growth of high-definition image data is increasingly prominent - although traditional communication technology improves transmission capacity through coding optimization, bandwidth allocation and other means, when transmitting high-precision images and videos (such as 4K / 8K), the contradiction between the surge in data volume and the limited channel capacity leads to frequent problems such as transmission delay and picture loss.

[0003] The existing image transmission system is still difficult to balance between semantic information preservation and transmission efficiency. The mainstream scheme relies on deep convolutional neural network (CNN) for source coding and channel adaptation, but the inherent limitations of CNN make it difficult to meet the needs of complex scenarios: first, the local receptive field characteristics of convolution operation result in weak ability to capture global semantics of images (such as object position logic, scene atmosphere), which easily causes loss of key information; second, the mechanism of CNN equally processing all channel features cannot dynamically allocate resources to areas of different semantic importance in images (such as lesion areas in medical images, changing areas in monitoring pictures), which consumes a lot of bandwidth and cannot guarantee the transmission reliability of core information. In addition, the computational overhead of convolution operation is too large, and in terminal devices with limited computing power, the efficiency of image transmission will be greatly reduced.

[0004] In recent years, the Transformer architecture has made breakthroughs in natural language processing and computer vision fields, becoming an important direction to make up for the shortcomings of convolutional networks. Among them, Swin Transformer relies on mobile window multi-head self-attention mechanism, combining the ability of "local efficient modeling" and "integration of cross-window global dependencies", which can effectively process large-size image details and establish long-distance semantic association, alleviating the problems of over-smoothing and detail loss in traditional methods.

[0005] However, the existing feature extraction process generally treats different features "equally", ignoring the significant difference in semantic importance. This processing neither conforms to the importance distribution of images at the spatial region level, nor ignores the unequal contribution of feature channels in decision-making. Therefore, it is necessary to introduce a mechanism that can adaptively emphasize key semantics and suppress redundant information to improve the recoverability and overall transmission efficiency of key content under limited bandwidth.

[0006] Based on the above problems, the present application proposes a feature selection mechanism combining spatial and channel attention, and an adaptive rate wireless image communication method using a mask autoencoder for self-supervised compression and reconstruction of high-frequency redundant semantics. Under different channel conditions and service demands, this method can dynamically allocate encoding resources, highlight key information and suppress non-key content, thereby achieving higher subjective and objective reconstruction quality and lower transmission delay under limited bandwidth conditions. SUMMARY

[0007] The present application proposes an adaptive rate wireless image communication method based on channel attention and mask autoencoder to solve the above problems existing in the prior art.

[0008] To achieve the above purpose, the present application provides an adaptive rate wireless image communication method based on channel attention and mask autoencoder, comprising the following steps:

[0009] Preprocessing the original image dataset to obtain a preprocessed image;

[0010] Inputting the preprocessed image into a semantic encoder, sequentially performing downsampling, feature extraction and channel noise adaptive encoding, and obtaining encoded features after repeating several rounds;

[0011] According to the compression rate, the channel level adaptive mask processing is performed on the encoded features to obtain unmasked semantic features to be transmitted;

[0012] After power normalization, the unmasked semantic features are sent to the physical channel for transmission to obtain channel output features;

[0013] Zero padding is performed on the channel output features, and the restored dimension is input into a semantic decoder to reconstruct an output image;

[0014] According to the output image and the original image, a loss function is calculated, and the model parameter optimization is performed through the loss function.

[0015] Optionally, the preprocessing of the original image dataset comprises:

[0016] Randomly cropping the original image to a predetermined size;

[0017] The cropped image is converted into a tensor form and divided into a training set and a test set.

[0018] Optionally, the semantic encoder comprises:

[0019] a plurality of cascaded stages, each stage comprising a down-sampling layer or an up-sampling layer and at least one SwinTransformer Block (STB);

[0020] After a single stage processing is completed, an attention-noise adaptive module, a channel attention module and a spatial attention module are connected.

[0021] Optionally, the processing of the attention-noise adaptive module comprises:

[0022] The semantic information output by the stage is globally averaged pooled to obtain a pooled feature;

[0023] The signal-to-noise ratio information and the pooled feature are spliced to form a noise-semantic signal;

[0024] The noise-semantic signal is sequentially passed through a fully connected layer, a ReLU activation layer, a fully connected layer and a Sigmoid layer to obtain a weight vector;

[0025] The weight vector is multiplied by the original semantic information to obtain a semantic feature fused with channel information.

[0026] Optionally, the calculation expression of the channel attention module is:

[0027] CA=σ(FC2(ReLU(FC1(Avg_pool(x1))))+FC2(ReLU(FC1(Max_pool(x1)))));

[0028] wherein, x1 is an input vector input to the module, σ represents a Sigmoid function, Avg_pool(·) and Max_pool(·) represent global average pooling and maximum pooling in the channel layer respectively, ReLU represents a rectified linear unit, and FC1, FC2 both represent a fully connected layer.

[0029] Optionally, the calculation expression of the spatial attention module is:

[0030] SA=σ(Conv([AvgPool(x2);MaxPool(x2)]));

[0031] Wherein, x2 represents an input vector input to the module, sigma represents a Sigmoid function, AvgPool(·) and MaxPool(·) represent average pooling and maximum pooling in the spatial level respectively, and Conv(·) represents a convolution operation.

[0032] Optionally, the obtaining of the unmasked semantic feature to be transmitted comprises:

[0033] According to the compression rate, the number of channels to be reserved is calculated.

[0034] A binary mask vector with the same dimension as the semantic feature channel is generated, and the encoded feature is multiplied by point operation to obtain an unmasked semantic feature.

[0035] Optionally, the reconstruction of the output image comprises:

[0036] The channel output feature is filled with a zero vector to the same dimension as the semantic encoder output;

[0037] The filled feature is input into the semantic decoder for feature reconstruction;

[0038] The reconstructed feature is mapped back to the image space to obtain an output image consistent with the original image size.

[0039] Compared with the prior art, the present application has the following advantages and technical effects:

[0040] The application discloses an adaptive rate wireless image communication method based on channel attention and mask autoencoder, comprising the following steps: preprocessing an original image dataset to obtain a preprocessed image; inputting the preprocessed image into a semantic encoder to sequentially perform downsampling, feature extraction and channel noise adaptive encoding, and obtaining encoded features after several rounds of repetition; performing channel-level adaptive mask processing on the encoded features according to a compression rate to obtain unmasked semantic features to be transmitted; sending the unmasked semantic features into a physical channel for transmission after power normalization to obtain channel output features; performing zero padding on the channel output features, inputting the features into a semantic decoder after restoring the dimension, and reconstructing to obtain an output image; and calculating a loss function according to the output image and the original image, and optimizing model parameters through the loss function. Under the same channel condition, the application differentiates and weights key semantic regions through spatial-channel attention, and combines adaptive mask dominated by the compression rate, so that the number of feature channels to be transmitted is reduced on the premise that the peak signal-to-noise ratio (PSNR) or multi-scale structural similarity index (MS-SSIM) is not less than a set threshold, thereby significantly reducing bandwidth occupation and transmission delay. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and of which specific embodiments will be described, are illustrated in the drawings, where:

[0042] Figure 1 Method flowchart of an embodiment of the present application;

[0043] Figure 2 Communication architecture diagram of an embodiment of the present application;

[0044] Figure 3 Channel feature-channel-space self-attention block (FCSAB) schematic diagram of an embodiment of the present application;

[0045] Figure 4 Channel feature self-adaptive block schematic diagram of an embodiment of the present application;

[0046] Figure 5 Channel attention block schematic diagram of an embodiment of the present application;

[0047] Figure 6 Space attention block schematic diagram of an embodiment of the present application;

[0048] Figure 7 Masking strategy schematic diagram of an embodiment of the present application;

[0049] Figure 8 "Zero padding" schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0050] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0052] Embodiment one

[0053] As shown in the present embodiment, a self-adaptive rate wireless image communication method based on channel attention and mask autoencoder is provided, which comprises: Figures 1-2

[0054] The original image data set is preprocessed to obtain a preprocessed image;

[0055] ​The preprocessed image is input into the semantic encoder, and downsampling, feature extraction and adaptive coding of channel noise are performed in sequence. After repeating for several rounds, the encoded features are obtained. The number of repetitions is set according to the image complexity.

[0056] Based on the compression ratio, channel-level adaptive masking is performed on the encoded features to obtain the unmasked semantic features to be transmitted.

[0057] The unmasked semantic features are normalized by power and then sent into the physical channel for transmission to obtain the channel output features.

[0058] Zero-padding is performed on the channel output features, the dimensions are restored, and the data is input into the semantic decoder to reconstruct the output image.

[0059] The loss function is calculated based on the output image and the original image, and the model parameters are optimized using the loss function.

[0060] Specifically, the following steps are included:

[0061] Step S1: Preprocess the original dataset and divide it into training and test sets.

[0062] Step S2: Input the data into the semantic encoder, which includes downsampling, feature extraction, and adaptive coding for channel noise. The semantic decoder simply replaces the downsampling operation with upsampling.

[0063] Step S3: The extracted features are added to the compression ratio-driven masking module for adaptive masking, and then the unmasked semantic features are input into the physical channel after power normalization.

[0064] Step S4: Perform a "zero-padding" operation on the features after the channel to restore the dimension of the semantic information to be consistent with the output of the semantic encoder, and then input it into the semantic decoder.

[0065] Step S5: After the semantic information is decoded, the reconstructed image can be obtained. The model parameters are updated and optimized using loss functions such as minimum mean square error.

[0066] In step S1, the dataset is first cropped randomly to a dimension suitable for the proposed network model, and then converted into tensors. The preprocessed data dimension is: x∈R B ×H×W×C B, H, W, and C represent the batch size, height, width, and number of channels of the input dataset, respectively. The dataset is divided into training and test sets according to a specific ratio.

[0067] In step 2, the training set data is input into the encoding layer of the proposed network model. First, it passes through the patch embedding layer, and then through multiple STBs.

[0068] The Swin Transformer Block (STB) consists of a window-based multi-head self-attention (W-MSA) module, a shifted window-based multi-head self-attention (SW-MSA) module, a multi-layer perceptron (MLP), a linear normalization layer, and a residual layer. Specifically, it can be expressed by the following formula:

[0069]

[0070] The first step is to input x l-1 After layer normalization, the data is input into W-MSA for feature extraction within different windows, followed by residual connection to obtain intermediate features. Then, layer normalization, MLP, and residual connections are performed to obtain the intermediate result x. l The second step is to replace W-MSA with SW-MSA, the purpose of which is to merge the information between different windows. The rest of the process is the same as the first step.

[0071] A stage contains multiple STBs and one downsampling or upsampling operation. After passing through a stage, the system first passes through an attention-noise adaptive module, followed by a channel attention module and a spatial attention module to improve information extraction capabilities. (The attention-noise adaptive module, channel attention module, and spatial attention module constitute an FCSAB, with the structure as follows...) Figure 3 (As shown). For datasets with larger image sizes, apply 4 stages; for smaller datasets, such as those with only (32×32×3) CIFAR10 or CIFAR100 datasets, apply 2 stages.

[0072] Assume the semantic information output after each Stage is H represents the semantic information of the i-th channel. s W s C s This represents the height, width, and number of channels of an image after a certain stage. To make semantic information robust to channel noise, it needs to pass through an attention-noise adaptive module, the mechanism of which is as follows: Figure 4Its specific implementation is to first set M... T Perform global average pooling and define the semantic information after pooling. Its expression can be written as:

[0073]

[0074] in This represents a feature at height i and width j in a given channel. Noise values ​​are written into m as n semantic information. T In this context, n is defined as n = C n / C1, where C1 is the number of output channels in the first stage, C n Let n be the number of output channels in stage n. Assuming that n is calculated to be 2 for a certain stage, the noise-semantic signal can be written as:

[0075]

[0076] Where γ represents the signal-to-noise ratio γ = SNR, in decibels (dB), and then the noise-semantic signal is... The inputs are sequentially fed into a fully connected layer FC1, a ReLU activation layer, another fully connected layer FC1, and a Sigmoid layer. Finally, the output weights are combined with the semantic information m. T Multiplication yields semantic information that incorporates channel information. The expression for this operation is as follows:

[0077]

[0078] Here, σ represents the Sigmoid function. Since the information perceived by the human eye is biased or has a particular focus, we can add a "spatial focus" to the model. This can be achieved by adding a spatial attention module (SAM), such as... Figure 5 Its expression is as follows:

[0079] SA=σ(Conv([AvgPool(x2);MaxPool(x2)]));

[0080] x`2=x2*S A ;

[0081] Where AvgPool(·) and MaxPool(·) represent spatial average pooling and max pooling, respectively, Conv(x) represents the convolution operation, with the dimensions remaining unchanged before and after convolution. SA represents the spatial attention weights. x2, These represent the input and output of the module, respectively. The pooled results are concatenated and then processed by f. 7×7The spatial attention value can be obtained. Then, multiplying the input x2 by it will assign spatial weights.

[0082] Similar to spatial attention mechanisms, we want to assign higher weights to more important channels, which can be achieved using a channel attention module. The principle of the channel attention mechanism is as follows: Figure 6 :

[0083] CA=σ(FC2(ReLU(FC1(Avg_pool(x))))+FC2(ReLU(FC1(Max_pool(x)))))

[0084] X`1 = x1 * CA;

[0085] Here, Avg_pool(·) and Max_pool(·) represent adaptive global pooling and max pooling at the spatial level, respectively, and CA represents the spatial attention weight value. x1, These represent the input and output of the module, respectively. The pooled result is passed through a shared MLP and sigmoid layer to obtain the spatial attention value. The input value is then multiplied by this channel attention value, thus assigning weights to different channels.

[0086] The encoding part of the semantic encoder is defined as:

[0087] z o =E φ (x,γ);

[0088] Where x, z o φ and γ represent the encoder's input vector, output, parameters, and channel features, respectively. The semantic decoder process in this invention is approximately the inverse process of the encoder, i.e., replacing downsampling with upsampling. The dimensionality transformation process is the same, so it will not be described in detail here. The semantic decoder is defined as follows:

[0089]

[0090] in, θ, These represent the model parameters and inputs for the reconstructed image and semantic decoder, respectively. The sampling rate for both the encoder and decoder is 2. Note that both the semantic encoder and decoder use channel features as part of their input. When training the model, the channel parameters, i.e., the signal-to-noise ratio (SNR), can be set as needed. For example, when the channel conditions are known, the SNR can be set to a fixed value; when the channel conditions are unknown, the SNR can be set to a random value. This way, the trained model has a certain robustness to different channel conditions.

[0091] Having efficiently extracted the data features using the preceding steps, the next step is to reduce the data dimensionality to decrease transmission overhead. First, the data features are passed through a fully connected layer to reduce the channel dimensionality. This can be achieved by using a masking strategy to conceal the features of subsequent channels. This can also be combined with the system's channel attention module to concentrate information on the unmasked features. The masking is implemented by applying the semantic encoder's output z... o The dot product with binary vector α is implemented as follows: Figure 7 As shown. z o The expression is as follows:

[0092]

[0093] Where H e W e This represents the final height and width of the image after going through several stages, C. o This represents the output dimension of the fully connected layer. In step 3, similar to the SNR value, a compression ratio can also be set for model training. Here, a random masking ratio scheme is used to select the masking scheme with the least amount of data under a given PSNR. Assuming the masking ratio is CR, and the value range of CR is [0,1], then the number of channels that can be retained is... This is a rounding up operation. The mask value is defined as follows:

[0094]

[0095] The masking operation can then be written as:

[0096] z=(z o ) T ·α;

[0097] Here, · represents the dot product operation, and z represents the unmasked feature. Let the unmasked feature z, carrying more information, pass through the channel. Let the output after passing through the wireless channel be... It can be modeled as:

[0098]

[0099] Where h represents the channel state information coefficient, assumed to follow a circularly symmetric complex Gaussian distribution. It has zero mean and unit variance; noise term The noise is characterized by complex Gaussian noise with a mean of zero and a variance of σ. n 2 If the output power of the signal is set to P, then the formula for calculating the signal-to-noise ratio γ is:

[0100]

[0101] The output after passing through the wireless channel Zero-padding is used to restore its dimension to match the encoder's output z. Specifically, this is achieved by filling the masked portion of the tensor with an all-zero tensor. For example... Figure 8 As shown. Features after "zero padding" The image is input into the semantic decoder to obtain the reconstructed image.

[0102] In step 5, there are two evaluation metrics for the model's performance or the reconstructed image: PSNR and MS-SSIM. One metric can be chosen as the training objective; this embodiment selects PSNR, whose expression is:

[0103]

[0104] Where n represents the image color bit depth, usually 8, and MSE is the mean square error (MSE), which is calculated using the following formula:

[0105]

[0106] As can be seen from the formula, the PSNR depends on the MSE, so the MSE is used as the model's loss function. Backpropagation is used to update the model parameters until convergence. During model training, structural similarity can also be chosen as a metric for updating model parameters. Furthermore, the signal-to-noise ratio, masking rate, and specific channel type (such as additive white Gaussian noise or Rayleigh fading channel) can be set as needed to train the desired model.

[0107] To ensure the learning effectiveness and decision-making accuracy of the model, this invention assigns higher weights to features that play a crucial role. A differentiated weight allocation mechanism strengthens the representation of important information while mitigating the interference of secondary information. Under the same conditions, a higher number of transmitted features results in higher-quality images decoded at the receiver. However, sometimes high image quality is not necessary. Therefore, the number of transmitted features can be adaptively reduced under certain image quality standards (such as PSNR, MS-SSIM) to reduce transmission overhead.

[0108] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An adaptive rate wireless image communication method based on channel attention and mask autoencoder, characterized in that, Includes the following steps: The original image dataset is preprocessed to obtain preprocessed images; The preprocessed image is input into the semantic encoder, and downsampling, feature extraction and adaptive coding of channel noise are performed sequentially. After several iterations, the encoded features are obtained. Based on the compression ratio, channel-level adaptive masking is performed on the encoded features to obtain the unmasked semantic features to be transmitted. The unmasked semantic features are normalized by power and then sent into the physical channel for transmission to obtain the channel output features. Zero-padding is performed on the channel output features, the dimensions are restored, and the data is input into the semantic decoder to reconstruct the output image. The loss function is calculated based on the output image and the original image, and the model parameters are optimized using the loss function.

2. The method according to claim 1, characterized in that, The preprocessing of the original image dataset includes: The original image is randomly cropped to a predetermined size; The cropped images are converted into tensor form and divided into training and test sets.

3. The method according to claim 1, characterized in that, The semantic encoder includes: Several cascaded Stages, each Stage including a downsampling layer or an upsampling layer and at least one SwinTransformer Block; After a single Stage processing is completed, the attention-noise adaptive module, the channel attention module, and the spatial attention module are connected.

4. The method according to claim 3, characterized in that, The processing of the attention-noise adaptive module includes: Global average pooling is performed on the semantic information output by Stage to obtain pooled features; The signal-to-noise ratio information is concatenated with the pooling features to form a noise-semantic signal; The noise-semantic signal is passed sequentially through a fully connected layer, a ReLU activation layer, another fully connected layer, and a Sigmoid layer to obtain a weight vector; Multiplying the weight vector with the original semantic information yields the semantic features of the fused channel information.

5. The method according to claim 3, characterized in that, The calculation expression for the channel attention module is: CA=σ(FC2(ReLU(FC1(Avg_pool(x1))))+FC2(ReLU(FC1(Max_pool(x1))))); Where x1 is the input vector to this module, σ represents the Sigmoid function, Avg_pool(·) and Max_pool(·) represent global average pooling and max pooling at the channel level, respectively, ReLU represents the modified linear unit, and FC1 and FC2 both represent fully connected layers.

6. The method according to claim 3, characterized in that, The calculation expression for the spatial attention module is: SA=σ(Conv([AvgPool(x2);MaxPool(x2)])); Where σ represents the Sigmoid function, AvgPool(·) and MaxPool(·) represent the spatial average pooling and max pooling respectively, and Conv(·) represents the convolution operation.

7. The method according to claim 1, characterized in that, The obtained unmasked semantic features to be transmitted include: Calculate the number of channels to be retained based on the compression ratio; Generate a binary mask vector with the same dimension as the semantic feature channel, and perform a dot product operation on the encoded features to obtain the unmasked semantic features.

8. The method according to claim 1, characterized in that, The reconstructed output image includes: The channel output features are padded with zero vectors to the same dimension as the semantic encoder output; The padded features are input into the semantic decoder for feature reconstruction. The reconstructed features are mapped back to the image space to obtain an output image with the same size as the original image.

Images