An anti-noise and anti-interference image semantic communication system based on an attention mechanism

Abstract

The application provides an anti-noise and anti-interference image semantic communication system based on an attention mechanism, and relates to the fields of communication technology and artificial intelligence. An input picture is sequentially subjected to an encoder of a residual convolutional auto-encoding system, a dimension transformation module, a hybrid attention module, a dimension transformation module, and an encoder of the residual convolutional auto-encoding system to obtain an output image. The system dynamically adjusts the attention weight by calculating the similarity between the received image token and the predefined error code mode, and preferentially processes those more likely to represent the signal part that remains intact in transmission. The method of the application not only increases the accuracy and reliability of image transmission in a harsh communication environment, but also provides a new way for efficient compression and transmission of image semantics. The attention mechanism based on the KL transformation of the image token and the covariance matrix of the channel error code mode provides a new solution strategy for semantic communication, and has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of communication technology and artificial intelligence, and particularly relates to an anti-noise and anti-interference image semantic communication system based on attention mechanism in an interference environment. BACKGROUND

[0002] In the field of image semantic communication, attention mechanism has become an important technology that significantly improves the expressiveness and transmission efficiency of image content by assigning different weights to different image regions or features. Attention mechanism was first used in the Transformer network structure proposed by Google in 2017 ("Attention is all you need, NIPS 2017"), and was first used in natural language processing. Subsequently, in 2021, the document "Transformers for Image Recognition at Scale, ICRL 2021" introduced the Transformer network into image recognition and understanding. This method generates attention weights by using the information of the image sequence itself, that is, by analyzing the internal features and structures of the image content to determine which parts are more important and should be given more attention. This method performs well in handling the consistency and significance of internal information of images, but when it comes to external noise and interference, especially in image semantic communication, its performance is often unsatisfactory.

[0003] The existing attention mechanism used in semantic communication often ignores the impact of external interference factors on communication quality. This means that in a severely interfered communication environment, although certain image regions are highly important without interference, they may become inaccurate or lose information due to noise interference, resulting in unsatisfactory final communication effect. Therefore, the traditional method has limitations in adapting to rapidly changing external noise and interference environment.

[0004] The external attention mechanism has been proposed in documents such as “Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks” (IEEE Transactions On Pattern Analysis And Machine Intelligence, Vol. 45, No. 5, May 2023), but there is no specific research on how to use the external attention mechanism under noise or interference in the existing documents. The present application specifically defines the implementation process of the external attention mechanism in denoising. This new type of external attention mechanism provides a new solution strategy for image semantic communication, not only improves the communication quality and robustness in the interference environment, but also opens up a new way for further optimization of the image transmission system. SUMMARY

[0005] One of the purposes of the present application is at least to provide a method and system for robust and anti-interference transmission of image semantic communication based on external attention mechanism to reduce the influence of interference or noise and improve the probability of correct recovery of semantics at the receiving end. The method of the present application no longer relies only on the information of the image itself to generate attention weights, but introduces external factors such as the covariance matrix of the channel error code mode as the key input for generating attention weights. In this way, the attention mechanism can take into account the influence of external noise and interference, and assign lower weights to image token (i.e. image semantic unit) regions that may be affected by noise, and assign higher weights to regions that are relatively unaffected. This dynamic weight adjustment mechanism based on external information enables the system to more flexibly adapt to complex and variable external interference, thereby minimizing the impact of noise and interference while maintaining the accuracy of image semantics.

[0006] To achieve the above-mentioned purposes, the technical scheme adopted by the present application is as follows:

[0007] An anti-noise and anti-interference image semantic communication system based on attention mechanism, an input picture is sequentially input into an encoder of a residual convolutional auto-encoding system, a dimension transformation module, a hybrid attention module, a dimension transformation module, and an encoder of a residual convolutional auto-encoding system to obtain an output image, and the specific working process is as follows:

[0008] Step 1: Obtain an input image, input the input image into an encoder of a residual convolutional auto-encoding system, and the encoder of the residual convolutional auto-encoding system transforms the input image into a feature subgraph, and the feature subgraph has dimensions of C, H, and W, representing channels, height, and width, respectively;

[0009] Step 2: The image data of the feature subgraph is converted into one-dimensional image feature data by a dimension transformation module in column or row, and then the one-dimensional image feature data is organized into two-dimensional data in batches, with the dimension sizes being CHW and batch number respectively;

[0010] Step 3: The two-dimensional data with the dimension sizes being CHW and batch number respectively are input into a mixed attention module, and two-dimensional image feature data with obtained weights are output; the mixed attention module includes an external attention module and a multi-head attention module, and the specific working process of the mixed attention module is as follows:

[0011] Step 301: The mixed attention module receives the two-dimensional data and inputs them into two convolutional neural networks for dimension transformation to obtain image semantic units token; the two convolutional neural networks are connected to an encoder of the mixed attention mechanism and a decoder of the mixed attention mechanism respectively;

[0012] Step 302: The encoder of the mixed attention mechanism includes an external attention module, two residual normalization modules and a position-based feedforward network module; wherein the image semantic units enter the external attention module first after entering the encoder of the mixed attention mechanism, are given corresponding weights, then enter the residual normalization modules, and are transformed by the position-based feedforward network module, and finally obtain the output vector of the encoder after passing through the residual normalization module once;

[0013] Step 303: The decoder of the mixed attention mechanism includes an external attention module, three residual normalization modules, a multi-head self-attention module and a position-based feedforward network module; wherein the image semantic units enter the external attention module first after entering the decoder, are given corresponding weights, and then enter the residual normalization module once; the obtained output and the output vector of the encoder enter the multi-head self-attention module together, and then pass through the residual normalization modules, the position-based feedforward network module and the residual normalization module in sequence to obtain the output;

[0014] Step 304: The output of the decoder passes through a transform dimension fully connected layer to obtain two-dimensional image feature data;

[0015] The external attention module includes a query module Query, a key module Key, a value module Value and a normalization module Normal; the specific structure of the external attention module is as follows: the vector X corresponding to the input image semantic units unfolded by row or column is input into the query module for KL transformation to obtain a vector A; the normalized vector A is multiplied by the weight vector stored in the key module to obtain an intermediate vector B; the intermediate vector B is input into the normalization module for normalization to obtain an intermediate vector B'; the intermediate vector B' is input into the value module and dot product with the vector in the value module weight matrix to obtain the final weight value D;

[0016] Step 4: convert the two-dimensional image feature data into a one-dimensional vector with dimension CHW, and then send the one-dimensional vector into a dimension transformation module to transform the image semantic unit with dimension (C, H, W);

[0017] Step 5: send the image semantic unit with dimension (C, H, W) into the decoder of the residual convolutional autoencoder to obtain the restored image data;

[0018] Preferably, the multi-head attention module adopts the same method as the Transformer network, uses multiple attention heads, receives multiple image semantic unit transformation vectors with enhanced features, extracts more rich feature information, and obtains the relationship between each feature in the sequence.

[0019] Preferably, the dimension transformation module is a convolutional neural network.

[0020] Preferably, the KL transformation is realized by a multi-layer linear network, and the working process is as follows:

[0021] Step 1011: average the input vector X, and subtract the center value to obtain the centralized vector Xm.

[0022] Step 1012: calculate the covariance matrix C of the vector Xm.

[0023] Step 1013: solve the eigenvalues and eigenvectors of the covariance matrix C, and take the eigenvectors corresponding to the first k largest eigenvalues to form an orthogonal transformation matrix P.

[0024] Step 1014: transform the vector A=X*P.

[0025] Preferably, the key module Key is a linear multi-layer neural network, and the initial value of the weight is the covariance matrix of the error code mode occurring in the communication channel.

[0026] Preferably, the value module Value is a multi-layer linear neural network, and the initial value of the weight is randomly generated and optimized during the training process.

[0027] In summary, due to the adoption of the above technical solutions, the present application has at least the following beneficial effects:

[0028] By using the KL transformation on the vector of the image semantic unit token, the main features of the image can be obtained. When a large number of noise points are contained in the token, the statistical features of the noise points can be captured; after statistical learning of all noise patterns, the key module Key matrix obtained can learn the characteristics of all noise patterns; the matrix in the value module Value can further screen the noise points existing in the original input data. Through the above processing steps, the external attention mechanism can better capture which main content of the input image token is noise, so as to give the token a lower weight. In this way, in the subsequent semantic decoding process, the influence of these noise-containing tokens is reduced, thereby achieving the effect of suppressing image semantic noise.

[0029] The method of the present application not only increases the accuracy and reliability of image transmission in a harsh communication environment, but also provides a new way for efficient compression and transmission of image semantics. This attention mechanism based on the KL transformation of the image token and the covariance matrix of the channel error code mode provides a new solution strategy for semantic communication, which has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the structure diagram of the external attention module based on KL transformation provided in embodiment one;

[0031] Figure 2 is the implementation architecture diagram of an image semantic communication system with anti-noise and anti-interference functions provided in embodiment two of the present application, wherein a hybrid attention module containing an external attention module is used.

[0032] Figure 3 is the structure diagram of the hybrid attention module containing the external attention module in embodiment two of the present application. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in combination with the drawings and embodiments, so that the purpose, technical scheme and advantages of the present application are more clear and explicit. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0034] Embodiment one:

[0035] As shown in Figure 1 , embodiment 1 of the present application discloses the implementation structure of the external attention module, and the working process of the module includes the following steps:

[0036] Step 101: send the row or column unfolded vector X corresponding to the input image semantic unit token into the query module Query to perform KL (Karhunen-Loève) transformation to obtain vector A; specifically, the KL transformation is realized by a multi-layer linear network, and the workflow is as follows:

[0037] Step 1011: decenter the input vector; average the input vector X, and subtract the center value to obtain the centralized vector Xm.

[0038] Step 1012: calculate the covariance matrix C of the vector Xm.

[0039] Step 1013: solve the eigenvalues and eigenvectors of the covariance matrix C, and take the eigenvectors corresponding to the first k largest eigenvalues to form an orthogonal transformation matrix P.

[0040] Step 1014: convert the vector A=X*P.

[0041] Step 102: multiply the normalized vector A with the weight vector stored in the key module Key to obtain the intermediate vector B; wherein the Key module is a linear multi-layer neural network, and the initial value of the weight is the covariance matrix of the error code mode that may occur in the communication channel.

[0042] Step 103: send the intermediate vector B into the normalization module Normal to perform normalization to obtain the intermediate vector B';

[0043] Step 104: send the intermediate vector B' into the value module Value to perform dot product with the vector in the Value weight matrix to obtain the weight value D corresponding to the input token vector; the Value module is a multi-layer linear neural network, and the initial value of the weight is randomly generated and optimized in the training process.

[0044] In the above embodiment, based on the feature extraction characteristics of KL transformation and the blind source separation principle of image noise, the noise features existing in the image token are first identified. In the identification process, the image noise mode that may exist in the entire token transmission process is used, wherein the intermediate vector B represents the vector with different noise correlation, and the Value module is equivalent to filtering, so that the vector position with the largest correlation with the noise characteristics is extracted.

[0045] Example two:

[0046] As Figure 2As shown, the second embodiment of the present application discloses an anti-noise and anti-interference image semantic communication system using a hybrid attention mechanism, wherein the hybrid attention module uses two kinds of attention networks, namely an external attention mechanism and a multi-head self-attention mechanism. The multi-head self-attention mechanism is an attention mechanism used in a Transformer network. The working process of the semantic communication system includes the following steps:

[0047] Step 201: input the input image into the encoder of the residual convolutional auto-encoder system of the transmitting end;

[0048] Step 202: the encoder of the residual convolutional auto-encoder system transforms the image into feature subgraphs. These feature subgraphs are small pictures with very small dimensions, with dimensions (C, H, W), representing channels, height, and width;

[0049] Step 203: the transmitting end of the image semantic communication system converts the three-dimensional (C, H, W) image data into one-dimensional image feature data by column or row, and enters the channel transmission;

[0050] Step 204: the receiving end of the image semantic communication system organizes the received one-dimensional image feature data into two-dimensional data in batches, with dimensions CHW and batch number nmodule, respectively;

[0051] Step 205: send the two-dimensional data with dimensions (CHW, nmodule) into the hybrid attention module, convert the two-dimensional data into three-dimensional image token sequences by the transformation dimension CNN at the front end of the module, send the three-dimensional image token sequences into the hybrid attention module, obtain the weights of each image token, and output the (CHW, nmodule) dimensional image feature data with the obtained weights at the same time;

[0052] Step 206: convert the two-dimensional (CHW, nmodule) image feature data with the obtained weights into a one-dimensional vector with a dimension length of CHW;

[0053] Step 207: send the dimension of the one-dimensional image feature vector with dimensions CHW into a dimension transformation module, and transform it into an image token with dimensions (C, H, W);

[0054] Step 208: send the image token with dimensions (C, H, W) into the decoder of the residual convolutional auto-encoder, and obtain the restored image data;

[0055] The hybrid attention network used in step 205 in the above embodiment is composed of the external attention mechanism and the multi-head attention mechanism in the present application, as shown in Figure 3 Figure 3 The dashed box is a specific attention network, and its working process is as follows:

[0056] ​Step 2051: The image feature data is input into two convolutional neural networks for dimension transformation to obtain image tokens. The outputs of the two CNN convolutional neural networks are respectively connected to the attention modules in the two dashed boxes, where the left is an encoder of the hybrid attention mechanism, and the right is a decoder of the hybrid attention mechanism.

[0057] Step 2052: After the image tokens enter the encoder, they first enter the external attention module, where tokens with noise or interference patterns are identified and given smaller weights, while tokens without noise or interference patterns are given larger weight values.

[0058] Step 2053: The token vector with new weights obtained in the encoder first undergoes residual standardization, then enters a position-based feedforward network for transformation, further enhancing the representation of position information and features, and the output undergoes residual standardization again.

[0059] Step 2054: In the encoder, the transformed vector sequence after residual standardization is output to the input end of the multi-head attention module in the decoder.

[0060] Step 2055: In the decoder, the shifted image token vector also undergoes position encoding before entering the external attention network, where tokens with noise or interference patterns are identified and given corresponding weights.

[0061] Step 2056: In the decoder, the shifted token vector with new weights after residual standardization is input into the multi-head attention module together with the token vector output from the encoder.

[0062] Step 2057: In the decoder, the multi-head attention module uses the same method as the Transformer network, using multiple attention heads to receive multiple token transformation vectors with enhanced features, extract more rich feature information, and obtain the relationship between features in the sequence.

[0063] Step 2058: In the decoder, similar to the Transformer network, the multi-head attention module outputs the features of the output vector and the relationship between features to the feedforward neural network after residual standardization, and the feedforward neural network uses these features and relationships to obtain image features that meet the training purpose, which are then output after standardization.

[0064] In the above method, the hybrid attention module plays a similar role to the Transformer network, but the external attention module limits the features of interest to noise features, allowing the entire structure to have anti-noise and anti-interference effects.

[0065] It is to be understood that the present application is described by way of example only, and that modifications or alterations can be made to the features and embodiments described without departing from the spirit and scope of the application. In addition, modifications can be made to the features and embodiments described to accommodate specific situations and materials without departing from the spirit and scope of the application. Accordingly, the application is not limited to the specific embodiments described herein, but rather only by the claims which follow, and their equivalents.

Claims

1. An anti-noise and anti-interference image semantic communication system based on an attention mechanism, characterized in that, The input picture sequentially passes through the encoder of the residual convolutional auto-encoding system, the dimension transformation module, the hybrid attention module, the dimension transformation module, and the encoder of the residual convolutional auto-encoding system to obtain the output picture, and the specific working process is as follows: Step 1: Obtain the input image, input the input image into the encoder of the residual convolutional auto-encoding system, and transform the input image into a feature subgraph by the encoder of the residual convolutional auto-encoding system, wherein the feature subgraph has dimensions of C, H, and W, respectively representing a channel, a height, and a width; Step 2: Convert the image data of the feature subgraph by column or row into one-dimensional image feature data through the dimension transformation module, and then organize the one-dimensional image feature data into two-dimensional data in batches, wherein the dimensions of the two-dimensional data are CHW and the number of batches, respectively; Step 3: Input the two-dimensional data with dimensions of CHW and the number of batches into the hybrid attention module to output two-dimensional image feature data with weights; the hybrid attention module includes an external attention module and a multi-head attention module, and the specific working process of the hybrid attention module is as follows: Step 301: The hybrid attention module receives the two-dimensional data and inputs the two-dimensional data into two convolutional neural networks for dimension transformation to obtain image semantic units token; the two convolutional neural networks are connected to the encoder of the hybrid attention mechanism and the decoder of the hybrid attention mechanism, respectively; Step 302: The encoder of the hybrid attention mechanism includes an external attention module, two residual normalization modules, and a position-based feedforward network module; wherein the image semantic units enter the encoder of the hybrid attention mechanism first enter the external attention module, are assigned corresponding weights, then enter the residual normalization module, and then enter the position-based feedforward network module for transformation, and finally pass through a residual normalization module to obtain an output vector of the encoder; Step 303: The decoder of the hybrid attention mechanism includes an external attention module, three residual normalization modules, a multi-head self-attention module, and a position-based feedforward network module; wherein the image semantic units enter the decoder first enter the external attention module, are assigned corresponding weights, then enter a residual normalization module, and the obtained output and the output vector of the encoder enter the multi-head self-attention module, and then sequentially pass through a residual normalization module, a position-based feedforward network module, and a residual normalization module to obtain an output; Step 304: The output of the decoder passes through a transform dimension fully connected layer to obtain two-dimensional image feature data; The external attention module includes a query module Query, a key module Key, a value module Value, and a normalization module Normal; the specific structure of the external attention module is as follows: the vector X corresponding to the input image semantic units unfolded by row or column is input into the query module for KL transformation to obtain a vector A; the normalized vector A is multiplied by the weight vector stored in the key module to obtain an intermediate vector B; the intermediate vector B is input into the normalization module for normalization to obtain an intermediate vector B'; the intermediate vector B' is input into the value module and dot-multiplied with the vector in the weight matrix of the value module to obtain the final weight value D. Step 4: convert the two-dimensional image feature data into a one-dimensional vector with dimension CHW, and then send the one-dimensional vector into a dimension transformation module to transform the image semantic unit into an image semantic unit with dimension (C, H, W); Step 5: send the image semantic unit with dimension (C, H, W) into the decoder of the residual convolutional autoencoder to obtain the recovered image data.

2. The anti-noise and anti-interference image semantic communication system based on an attention mechanism according to claim 1, characterized in that, The multi-head attention module adopts the same method as the Transformer network, uses multiple attention heads, receives multiple image semantic unit transformation vectors with enhanced features, extracts more rich feature information, and obtains the relationship between each feature in the sequence.

3. The anti-noise and anti-interference image semantic communication system based on an attention mechanism according to claim 1, characterized in that, The dimension transformation module is a convolutional neural network.

4. The anti-noise and anti-interference image semantic communication system based on an attention mechanism according to claim 1, characterized in that, The KL transformation is realized by a multi-layer linear network, and the workflow is as follows: Step 1011: average the input vector X, and subtract the central value to obtain the centralized vector Xm; Step 1012: calculate the covariance matrix C of the vector Xm; Step 1013: solve the eigenvalues and eigenvectors of the covariance matrix C, and take the eigenvectors corresponding to the first k largest eigenvalues to form an orthogonal transformation matrix P; Step 1014: transform the vector A = X*P.

5. The anti-noise and anti-interference image semantic communication system based on an attention mechanism according to claim 1, characterized in that, The key module Key is a linear multi-layer neural network, and the initial value of the weight is the error code mode covariance matrix occurring in the communication channel.

6. The anti-noise and anti-interference image semantic communication system based on an attention mechanism according to claim 1, characterized in that, The value module Value is a multi-layer linear neural network, and the initial value of the weight is randomly generated and optimized during the training process.

Images