High-perception adaptive rate wireless image transmission method based on attention network

Abstract

A high-perception adaptive rate wireless image transmission method based on an attention network belongs to the field of computer image transmission, enables a model to pay more attention to the perception quality in a training process by combining a selection network and using LPIPS loss in a loss function, enables a generated image to be closer to a real image, and improves the image quality. According to the method, the defects of transmission images in perception quality are overcome, the flexibility and the expression ability of the model are improved, a system can focus on key areas when processing different types of images, and low channel resource occupation can still be kept and the images can be truly restored even under the condition of low signal-to-noise ratio. Compared with the traditional joint source channel coding image transmission system, the method disclosed by the invention makes a further progress in the aspect of improving the quality of the transmitted image.

Description

Technical Field

[0001] The present invention belongs to the field of computer image transmission, and in particular to a high-perception adaptive rate wireless image transmission method based on an attention network. Background Art

[0002] In recent years, joint source channel coding (JSCC) has improved transmission performance by jointly optimizing source and channel coding in a single framework. JSCC directly maps source data to channel symbols, saving bandwidth and improving transmission efficiency. In addition, semantic communication further improves transmission by transmitting information at the semantic level.

[0003] However, this model mainly relies on peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) to evaluate image quality, which cannot fully reflect the perception of the human visual system, limiting the visual effect of the model. In addition, the semantic information generated by the model has weak correlation, making it difficult to focus on key areas, resulting in more channel resources being occupied under low signal-to-noise ratio. Summary of the invention

[0004] In order to overcome the shortcomings of human perception of images in the prior art, the present invention provides a high-perception adaptive rate wireless image transmission method based on an attention network. By combining a selection network and adding LPIPS loss to the loss function, the quality of the image at the receiving end can be evaluated from a perceptual perspective, thereby improving visual realism. It can intelligently select active feature groups, so that the model pays more attention to perceptual quality during training, enhances the performance and flexibility of the model in complex scenes, enables it to focus on key areas for different types of input images, and can still maintain low channel resource occupancy and truly restore images under low signal-to-noise ratio conditions, promoting the generated image to be visually closer to the real effect.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions: A high-perception adaptive rate wireless image transmission method based on an attention network, the method comprising the following steps: Step 1: First, preprocess the input image and adjust the image to be transmitted to size, and convert the image into a PyTorch tensor. Finally, the image is standardized. The feature dimension after processing is , denoted as F; Step 2: Input F into a source encoder that contains a reflection padding layer, three convolutional layers, three instance normalization layers, and three ReLU activation layers to obtain the shallow features of the image. , the dimension is ; Step 3: The input is respectively to a selection network including a multi-head attention layer, a pooling layer, a signal-to-noise ratio connection layer, two MLP modules, and a channel encoder including two residual network layers, two signal-to-noise ratio adaptation modules, a convolution layer, and Reshape and Group operations; Step 4 After passing through the channel encoder, the generated dimension is The deep features of , and group them into a selectable feature vector S through the Group operation, with the dimension recorded as , the other is a non-selected feature vector N, with dimensions recorded as 28; Step 5 After passing through the trainable selection network, a binary selection mask is generated, denoted as W, with a dimension of ×1; this selection mask is multiplied by S to control the total number of active feature vectors A, and the dimension of A is , calculate the total number of active feature groups The formula is as follows, (1); in is the number of non-selected feature groups, is the selection mask; Step 6: After power normalization of the eigenvector A, it enters the AWGN channel and the obtained eigenvector is Y. The formula is as follows: (2); Where n is the noise of the channel; Step 7: Input the feature vector Y into a joint source channel decoder that includes two residual network blocks, two signal-to-noise ratio adaptive modules, two convolutional layers, a zero padding layer, a Reshape operation, two transposed convolutional layers, two instantiation normalization layers, two ReLU layer activation layers, a reflection padding layer, and a Sigmoid activation layer for decoding to obtain a visualized image. Step 8: Set the network hyperparameters and train the network parameters by minimizing the loss function.

[0006] Further, in step 3, the workflow in the selected network is as follows: The selection network block contains a multi-head attention layer, a pooling layer, a signal-to-noise ratio connection layer, and two MLP modules; the selection network is responsible for passing the shallow features of the source code through the multi-head self-attention layer and average pooling and cascading with the SNR, and finally passing through two layers of MLP with a Softmax function to generate the probability of each option. The formula is as follows: (3); Where P is the probability of corresponding feature activation, and are the feature vectors generated by two MLP modules respectively.

[0007] Furthermore, in step 5, a trainable selection network is constructed, and the process is as follows: Model the selection feature as a sampling problem of categorical distribution, and the sample space is , through the following formula, discrete sampling can be obtained from the probability distribution to obtain discrete selection categories, (4); in For the probability of each category, is the noise term sampled from the standard Gumbel distribution, is a value sampled from a uniform distribution U(0, 1); in order to make the discrete selection process trainable, the Softmax function is used to Transform to obtain a continuous probability distribution, (5); in Yes Control Discrete temperature parameter; when When it approaches infinity, it converges to a uniform distribution, that is, the probabilities of all categories are close, and hour, Close to a one-hot vector, that is, the output of the selection network is close to selecting a unique category; The output of the selection network after training is a one-hot vector , used to represent the final choice of each discrete choice category, Transformed into a feature mask vector W that satisfies, (6); in Represents a cumulative selection mask.

[0008] Furthermore, in step 7, the loss function is constructed as follows: Set multiple loss functions weighted sum, pixel-level loss function Using mean square error, (7); in and Respectively represent the features of the i-th picture to be transmitted and the features of the i-th picture to be reconstructed; Indicates the norm; Since image transmission needs to ensure that the restored image at the receiving end is similar to the original image, the channel resource usage is reduced as much as possible. The method for measuring the channel usage loss is as follows: (8); Since image transmission is related to human visual perception, by adopting LPIPS loss, we can effectively increase the model's attention to the perceptual characteristics of the human visual system and increase the visual similarity of images in the following way: (9); Where N is the total number of images in a batch during training; The total loss for training is as follows, (10); in, and are weight parameters, respectively and 0.1, The impact of control channel usage on total losses; By controlling the impact of LPIPS loss in the total loss, minimizing the loss function can reduce the use of channel bandwidth and make the image visually closer to the image to be transmitted.

[0009] Preferably, the channel type is set to AWGN channel. In order to improve the convergence and stability of model training and improve the accuracy of the model, a progressive learning strategy is adopted, that is, the network is trained three times. First, the learning rate is Train for 150 rounds and then use the learning rate Train for 150 rounds, and finally freeze the parameters of the source encoder and the selection network with a learning rate of Fine-tune other modules for 100 cycles; , the initial temperature is set to 5, the temperature decay factor is 0.017, the batch size is set to 128, and the signal-to-noise ratio is uniformly sampled between 0dB and 20dB; the model is trained at 0~20dB.

[0010] The beneficial effects of the present invention are as follows: by combining the selection network and using the LPIPS loss in the loss function, the model pays more attention to the perceptual quality during the training process, so that the generated image is closer to the real image, and the flexibility and expression ability of the model are improved, so that the system can focus on its key areas when processing different types of images, and can maintain the integrity of the receiving end image under harsh channel conditions while minimizing the occupancy of channel resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a diagram of the structure of high-sensing adaptive rate wireless image transmission; Figure 2 is to select the network structure diagram; Figure 3 It is a flow chart of high-perception adaptive rate wireless image transmission. DETAILED DESCRIPTION

[0012] The present invention will be further described below in conjunction with the accompanying drawings.

[0013] Reference Figure 1~Figure 3 , a high-perceptual adaptive rate wireless image transmission method based on an attention network, the method comprising the following steps: Step 1: First, preprocess the input image and adjust the image to be transmitted to size, and convert the image into a PyTorch tensor. Finally, the image is standardized. The feature dimension after processing is , denoted as F; Step 2: Input F into a source encoder that contains a reflection padding layer, three convolutional layers, three instance normalization layers, and three ReLU activation layers to obtain the shallow features of the image. , the dimension is ; Step 3: They are respectively input into the selection network including a multi-head attention layer, a pooling layer, a signal-to-noise ratio connection layer, two MLP modules and a channel encoder including two residual network layers, two signal-to-noise ratio adaptive modules, a convolution layer and Reshape and Group operations; the selection network block includes a multi-head attention layer, a pooling layer, a signal-to-noise ratio connection layer, and two MLP modules; the selection network is responsible for passing the shallow features of the source coding through the multi-head self-attention layer and average pooling and cascading with the SNR, and finally passing through two layers of MLP with a Softmax function to generate the probability of each option. The formula is as follows, (3); Where P is the probability of each feature being activated, and are the feature vectors generated by two MLP modules respectively; Step 4 After passing through the channel encoder, the generated dimension is The deep features of , and group them into a selectable feature vector S through the Group operation, with the dimension recorded as , the other is a non-selected feature vector N, with dimensions recorded as 28; Step 5 After passing through the trainable selection network, a binary selection mask is generated, denoted as W, with a dimension of ×1; this selection mask is multiplied by S to control the total number of active feature vectors A, and the dimension of A is , calculate the total number of active feature groups The formula is as follows, (1); in is the number of non-selected feature groups, is the selection mask proposed in formula (6); The process of constructing a trainable selection network is as follows: Model the selection feature as a sampling problem of classification distribution, and the sample space is , through the following formula, discrete sampling can be obtained from the probability distribution to obtain discrete selection categories, (4); in For the probability of each category, is the noise term sampled from the standard Gumbel distribution, is a value sampled from a uniform distribution U(0, 1); in order to make the discrete selection process trainable, the Softmax function is used to Transform to obtain a continuous probability distribution, (5); in Yes Control Discrete temperature parameter; when When it approaches infinity, it converges to a uniform distribution, that is, the probabilities of all categories are close, and hour, Close to a one-hot vector, that is, the output of the selection network is close to selecting a unique category; The output of the selection network after training is a one-hot vector , used to represent the final choice of each discrete choice category, Transformed into a feature mask vector W that satisfies, (6); in Represents a cumulative selection mask; Step 6: After power normalization of the eigenvector A, it enters the AWGN channel and the obtained eigenvector is Y. The formula is as follows: (2); Where n is the noise of the channel; Step 7: Input the feature vector Y into a joint source channel decoder that includes two residual network blocks, two signal-to-noise ratio adaptive modules, two convolutional layers, a zero padding layer, a Reshape operation, two transposed convolutional layers, two instantiation normalization layers, two ReLU layer activation layers, a reflection padding layer, and a Sigmoid activation layer for decoding to obtain a visualized image. Step 8: Set network hyperparameters and train network parameters by minimizing the loss function; the process is as follows: Set multiple loss functions weighted sum, pixel-level loss function Using mean square error, (7); in and Respectively represent the features of the i-th picture to be transmitted and the features of the i-th picture to be reconstructed; Indicates the norm; Since image transmission needs to ensure that the restored image at the receiving end is similar to the original image, the channel resource usage is reduced as much as possible. The method for measuring the channel usage loss is as follows: (8); Since image transmission is related to human visual perception, by adopting LPIPS loss, we can effectively increase the model's attention to the perceptual characteristics of the human visual system and increase the visual similarity of images in the following way: (9); Where N is the total number of images in a batch during training; The total loss for training is as follows, (10); in, and are weight parameters, respectively and 0.1, The impact of control channel usage on total losses; Control the impact of LPIPS loss in the total loss. By minimizing the loss function, the channel bandwidth usage can be reduced and the image can be visually closer to the image to be transmitted. The channel type is set to AWGN channel. In order to improve the convergence and stability of model training and improve the accuracy of the model, a progressive learning strategy is adopted, that is, the network is trained three times. First, the learning rate is Train for 150 rounds and then use the learning rate Train for 150 rounds, and finally freeze the parameters of the source encoder and the selection network with a learning rate of Fine-tune other modules for 100 cycles; , the initial temperature is set to 5, the temperature decay factor is 0.017, the batch size is set to 128, and the signal-to-noise ratio is uniformly sampled between 0dB and 20dB; the model is trained at 0~20dB.

[0014] The high-perception adaptive rate wireless image transmission method based on the attention network of this embodiment, by combining the selection network and adding LPIPS loss in the loss function, makes the generated image closer to the real image, solves the shortcomings of the transmitted image perception, enables it to intelligently select active feature groups, and enhances the flexibility and expression ability of the model to a certain extent, so that the system can focus on its key areas when facing different types of images, and can still maintain low channel resource occupancy and truly restore the image under low signal-to-noise ratio conditions. Compared with the traditional joint source channel coding image transmission system, the present invention further improves the quality of the transmitted image.

[0015] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A high-perception adaptive rate wireless image transmission method based on attention network, characterized in that: The method comprises the following steps: Step 1: First, preprocess the input image and adjust the image to be transmitted to size, and convert the image into a PyTorch tensor. Finally, the image is standardized. The feature dimension after processing is , denoted as F; Step 2: Input F into a source encoder that contains a reflection padding layer, three convolutional layers, three instance normalization layers, and three ReLU activation layers to obtain the shallow features of the image. , the dimension is ; Step 3: The input selection network and the channel encoder including two residual network layers, two signal-to-noise ratio adaptation modules, a convolution layer and Reshape and Group operations are respectively input; Step 4 After passing through the channel encoder, the generated dimension is The deep features of , and group them into a selectable feature vector S through the Group operation, with the dimension recorded as , the other is a non-selected feature vector N, with dimensions recorded as 28; Step 5 After passing through the trainable selection network, a binary selection mask is generated, denoted as W, with a dimension of ×1; this selection mask is multiplied by S to control the total number of active feature vectors A, and the dimension of A is , calculate the total number of active feature groups The formula is as follows, （1）； in is the number of non-selected feature groups, is the selection mask; Step 6: After power normalization of the eigenvector A, enter the AWGN channel and obtain the eigenvector Y. The formula is as follows: （2）； Where n is the noise of the channel; Step 7: Input the feature vector Y into a joint source channel decoder that includes two residual network blocks, two signal-to-noise ratio adaptive modules, two convolutional layers, a zero padding layer, a Reshape operation, two transposed convolutional layers, two instantiation normalization layers, two ReLU layer activation layers, a reflection padding layer, and a Sigmoid activation layer for decoding to obtain a visualized image. Step 8: Set the network hyperparameters and train the network parameters by minimizing the loss function.

2. The high-perception adaptive rate wireless image transmission method based on attention network according to claim 1 is characterized in that: In step 3, when selecting the workflow in the network, the process is as follows: The selection network block contains a multi-head attention layer, a pooling layer, a signal-to-noise ratio connection layer, and two MLP modules; the selection network is responsible for passing the shallow features of the source code through the multi-head self-attention layer and average pooling and cascading with the SNR, and finally passing through two layers of MLP with a Softmax function to generate the probability of each option. The formula is as follows: （3）； Where P is the probability of each feature being activated, and are the feature vectors generated by two MLP modules respectively.

3. The high-perception adaptive rate wireless image transmission method based on attention network according to claim 1 or 2, characterized in that: In step 5, a trainable selection network is constructed, and the process is as follows: Model the selection feature as a sampling problem of categorical distribution, and the sample space is , discrete selection categories are obtained by discrete sampling from the probability distribution through the following formula, （4）； in For the probability of each category, is the noise term sampled from the standard Gumbel distribution, is a value sampled from a uniform distribution U(0, 1); in order to make the discrete selection process trainable, the Softmax function is used to Transform to obtain a continuous probability distribution, （5）； in Yes Control Discrete temperature parameter; when When it approaches infinity, it converges to a uniform distribution, that is, the probabilities of all categories are close, and hour, Close to a one-hot vector, that is, the output of the selection network is close to selecting a unique category; The output of the selected network after training is a one-hot vector , used to represent the final choice of each discrete choice category, Transformed into a feature mask vector W that satisfies, （6）； in Represents a cumulative selection mask.

4. The high-perception adaptive rate wireless image transmission method based on attention network according to claim 1 or 2, characterized in that: In step 7, the process of constructing the loss function is as follows: Set multiple loss functions weighted sum, pixel-level loss function Using mean square error, （7）； in and Respectively represent the features of the i-th picture to be transmitted and the features of the i-th picture to be reconstructed; Indicates the norm; Since image transmission needs to ensure that the restored image at the receiving end is similar to the original image, the channel resource usage is reduced as much as possible. The method for measuring the channel usage loss is as follows: （8）； Since image transmission is related to human visual perception, by adopting LPIPS loss, we can effectively increase the model's attention to the perceptual characteristics of the human visual system and increase the visual similarity of images in the following way: （9）； Where N is the total number of images in a batch during training; The total loss for training is as follows, （10）； in, and are weight parameters, respectively and 0.1, The impact of control channel usage on total losses; By controlling the impact of LPIPS loss in the total loss, minimizing the loss function can reduce the use of channel bandwidth and make the image visually closer to the image to be transmitted.

5. The high-perception adaptive rate wireless image transmission method based on attention network according to claim 4 is characterized in that: The channel type is set to AWGN channel. In order to improve the convergence and stability of model training and improve the accuracy of the model, a progressive learning strategy is adopted, that is, the network is trained three times. First, the learning rate is Train for 150 rounds and then use the learning rate Train for 150 rounds, and finally freeze the parameters of the source encoder and the selection network with a learning rate of Fine-tune other modules for 100 cycles; , the initial temperature is set to 5, the temperature decay factor is 0.017, the batch size is set to 128, and the signal-to-noise ratio is uniformly sampled between 0dB and 20dB; the model is trained at 0~20dB.

Images