A semantic communication method, device and system for an image classification task

Abstract

The application discloses a kind of semantic communication methods, devices and systems for image classification tasks, the method includes transmitter obtains the semantic feature of given image and is encoded into transmitted image semantic vector based on auto-encoder;Relay node extracts the text feature of given image, and it is multimodal feature fusion with the image feature recovered by image channel decoder, then it is obtained multimodal semantic vector by multimodal channel encoder and is forwarded to receiver;Receiver receives multimodal semantic vector and executes image classification task.The semantic communication method provided by the application introduces the structure of auto-encoder into semantic communication on the one hand, greatly reduces the information redundancy, reduces the occupied communication resources, on the other hand, introduces multimodal semantic relay module, it can supplement complementary text semantic features in relay node, to help receiver better complete image classification task.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to a semantic communication method, apparatus, and system for image classification tasks. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and communication technologies, researchers have focused on how to accurately and efficiently transmit the required information under limited communication resources. The semantic communication paradigm offers an effective solution to this problem.

[0003] Currently, traditional image classification methods based on conventional communication primarily involve encoding and transmitting a bitstream containing all pixel information of an image at the sending end, followed by the receiving end attempting to reconstruct the original image and inputting it into a classifier to predict the category. However, this method suffers from significant information redundancy in the transmitted bitstream, and the classification results are easily affected by channel conditions. Existing semantic communication methods for image classification utilize deep neural networks to extract image features at the sending end and encode them into transmission vectors, while the receiving end decodes the image features to directly perform the image classification task. However, in semantic communication, due to the heterogeneity of user equipment attributes and the inaccuracy of transmitted data, the sending and receiving ends may predict different classification results for similar image features.

[0004] Therefore, how to improve the accuracy of image classification while reducing the consumption of communication resources is an urgent problem to be solved in the communication of image classification tasks. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a semantic communication method, apparatus, and system for image classification tasks. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a semantic communication method for image classification tasks, applied to a transmitter, comprising:

[0007] Obtain the semantic features of a given image;

[0008] The semantic features are encoded into image semantic vectors using an autoencoder and then sent to the relay node.

[0009] In one embodiment of the present invention, encoding the semantic features into an image semantic vector based on an autoencoder includes:

[0010] The semantic features are encoded into a transmitted image semantic vector using an autoencoder-based image channel encoder, the expression of which is:

[0011] x i =Ci (f i ;β i )

[0012] Where, x i f represents the semantic vector of the image. i C represents the semantic features of a given image. i Represents the image channel encoder, β i This represents the learnable parameters.

[0013] Secondly, the present invention provides a semantic communication method for image classification tasks, applied to relay nodes, comprising:

[0014] The image semantic vector sent by the receiver is received and decoded to obtain the recovered image features; wherein, the image semantic vector is obtained by encoding the semantic features of a given image based on an autoencoder;

[0015] Obtain the text features of a given image;

[0016] The text features and the recovered image features are fused using multimodal feature fusion to obtain multimodal features;

[0017] The multimodal features are encoded into multimodal semantic vectors and sent to the receiver.

[0018] In one embodiment of the present invention, receiving and decoding an image semantic vector sent by a transmitter to obtain recovered image features includes:

[0019] The received image semantic vector is used to recover the image semantic information through an autoencoder-based image channel decoder, resulting in the recovered image features, the expression of which is:

[0020]

[0021] Among them, f i The image features to be recovered, x i C represents the image semantic vector received by the relay node. i -1 δ represents the image channel decoder. i This represents the learnable parameters.

[0022] In one embodiment of the present invention, obtaining the text features of a given image includes:

[0023] The stored text data is encoded using a text semantic encoder to obtain text features, the expression of which is:

[0024] f t =S t (T;α t )

[0025] Among them, f t S represents text features. t Represents a text semantic encoder, T = {t1, t2, ..., t} l} represents a sentence embedding representation with l words, α t This represents the learnable parameters of the corresponding text semantic encoder.

[0026] In one embodiment of the present invention, the text features and the recovered image features are fused using multimodal feature fusion to obtain multimodal features, including:

[0027] The text features and the recovered image features are input into the multimodal feature fusion module to obtain multimodal features, the expression of which is:

[0028] f m =W(concat(f) i ,f t ))+b

[0029] Among them, f m f represents multimodal features i f represents the recovered image features. t represents text features, concat represents the concatenation operation, W represents the weight matrix, and b represents the corresponding bias constant.

[0030] In one embodiment of the present invention, encoding the multimodal fusion features into a multimodal semantic vector includes:

[0031] The multimodal features are encoded into a transmitted multimodal semantic vector using a multimodal channel encoder, the expression of which is:

[0032] x m =C m (f m ;β m )

[0033] Where, x m f represents the multimodal semantic vector of the transmission. m Representing multimodal features, C m Represents a multimode channel encoder, β m This represents the learnable parameters.

[0034] Thirdly, the present invention provides a semantic communication device for image classification tasks, applied to a transmitter, comprising:

[0035] An image semantic encoder is used to obtain the semantic features of a given image;

[0036] An image channel encoder is used to encode the semantic features into an image semantic vector based on an autoencoder and send it to a relay node.

[0037] Fourthly, the present invention provides a semantic communication device for image classification tasks, applied to relay nodes, comprising:

[0038] An image channel decoder is used to receive and decode the image semantic vector sent by the transmitter to obtain the recovered image features; wherein, the image semantic vector is obtained by encoding the semantic features of a given image based on an autoencoder;

[0039] A text semantic encoder is used to obtain text features of a given image;

[0040] A multimodal feature fusion module is used to fuse the text features with the recovered image features to obtain multimodal features;

[0041] A multimodal channel encoder is used to encode the multimodal features into multimodal semantic vectors and send them to the receiver.

[0042] Fifthly, the present invention provides a semantic communication system for image classification tasks, comprising a transmitter, a relay node, and a receiver; wherein,

[0043] The transmitter is used to acquire the semantic features of a given image and encode them into a transmitted image semantic vector based on an autoencoder;

[0044] The relay node is used to extract the text features of a given image, fuse them with the image features recovered by the image channel decoder, and then obtain a multimodal semantic vector through the multimodal channel encoder and forward it to the receiver.

[0045] The receiver is used to receive multimodal semantic vectors and perform image classification tasks;

[0046] The transmitter includes the semantic communication device for image classification tasks applied to the transmitter end as described in the above embodiments, and the relay node includes the semantic communication device for image classification tasks applied to the relay node end as described in the above embodiments.

[0047] The beneficial effects of this invention are:

[0048] 1. The semantic communication method provided by this invention introduces an autoencoder structure into semantic communication, mapping the extracted feature vectors of different dimensions into semantic vectors of a certain length transmitted in the wireless channel according to different situations, which greatly reduces information redundancy and significantly reduces the communication resources occupied. On the other hand, it uses a multimodal semantic relay method to replace the traditional relay receiving and forwarding methods. At the relay node, it uses stored text data to supplement semantic knowledge and integrates it into the transmitted semantic vector, resulting in more complete multimodal features and multimodal semantic vectors to combat image data missingness and help improve the accuracy of image classification.

[0049] 2. The autoencoder structure used in the semantic communication method provided by this invention can not only significantly reduce the communication resources occupied, but also resist channel noise to a certain extent, thereby further improving receiver performance.

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the process at the transmitter end of a semantic communication method for image classification tasks provided in an embodiment of the present invention;

[0052] Figure 2 This is a flowchart illustrating the semantic communication method for image classification tasks provided in this embodiment of the invention at the relay node end.

[0053] Figure 3 This is a schematic diagram of the structure of a semantic communication device for image classification tasks applied at the transmitter end, provided by an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of a semantic communication device for image classification tasks applied at a relay node, provided by an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of the structure of a semantic communication system for image classification tasks provided in an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram of the workflow of a semantic communication system for image classification tasks provided in an embodiment of the present invention;

[0057] Figure 7 It is the complete image provided by the transmitter in the simulation experiment;

[0058] Figure 8 It is the masked image given by the transmitter in the simulation experiment. Detailed Implementation

[0059] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the semantic communication method for image classification tasks proposed according to the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0061] Example 1

[0062] Please see Figure 1 , Figure 1 This is a flowchart illustrating the semantic communication method for image classification tasks provided in this embodiment of the invention at the transmitter end, which includes:

[0063] Step 1: Obtain the semantic features of the given image.

[0064] Specifically, at the transmitter, the image is first scaled to a uniform resolution. Then, an image semantic encoder is used to extract semantic features from the given image:

[0065] Specifically, this embodiment uses the Inception-V3 model and a fully connected layer as an image semantic encoder to extract semantic features from a given image, and its expression is as follows:

[0066] f i =S i (I;α i )

[0067] Among them, f i S represents the extracted image semantic features. i Represents an image semantic encoder, I∈R d×d α represents the image input that the transmitter is ready to send. i This represents the learnable parameters.

[0068] Optionally, the Inception-V3 model used in the image feature extraction process is a model pre-trained on the ImageNet dataset to extract 2048-dimensional image features.

[0069] Step 2: Encode the semantic features into image semantic vectors based on the autoencoder and send them to the relay node.

[0070] Specifically, after passing through the image semantic encoder, the resulting semantic feature vector f iThe input is fed into an autoencoder-based image channel encoder, which encodes it into an image semantic vector for transmission over the wireless channel.

[0071] x i =C i (f i ;β i )

[0072] Where, x i f represents the semantic vector of the image. i C represents the image features of a given image. i Represents the image channel encoder, β i This represents the learnable parameters.

[0073] Optionally, the image channel encoder based on the autoencoder can be composed of cascaded multi-layer neural networks, including fully connected layers, batch normalization layers and Leacky ReLU activation layers, and the final image semantic vector dimension is 512.

[0074] Finally, the obtained image semantic vector is sent to the relay node through the channel.

[0075] This embodiment introduces an autoencoder into the transmitter of semantic communication, mapping extracted feature vectors of different dimensions into semantic vectors of a certain length transmitted in the wireless channel, greatly reducing information redundancy. The autoencoder structure not only significantly reduces the communication resources occupied but also resists channel noise to a certain extent, further improving receiver performance.

[0076] Example 2

[0077] This embodiment designs a multimodal semantic relay module at the relay node, which consists of an autoencoder-based image channel decoder, a text semantic encoder, a multimodal feature fusion module, and a multimodal channel encoder. In the multimodal semantic relay module, the image channel decoder is built from a multi-layer neural network, and the text semantic encoder is composed of BERT cascaded Bi-LSTM to extract text modality features. Then, the multimodal feature fusion module fuses the text features with the recovered image features, and finally, the multimodal channel encoder obtains the multimodal semantic vector, which is forwarded to the receiver.

[0078] Specifically, please see Figure 2 , Figure 2 This is a flowchart illustrating a semantic communication method for image classification tasks provided in an embodiment of the present invention at the relay node end, which includes:

[0079] Step 1: Receive and decode the image semantic vector sent by the transmitter to obtain the recovered image features; wherein, the image semantic vector is obtained by encoding the semantic features of the given image based on an autoencoder.

[0080] Specifically, after the image semantic vector emitted by the transmitter is transmitted through a wireless channel, assuming the simulated channel is an additive white Gaussian noise (AWGN) channel, the relay node first recovers the image semantic information from the received image semantic vector using an image channel decoder. The resulting image features are:

[0081]

[0082] Among them, f i The image features to be recovered, x i C represents the image semantic vector received by the relay node. i -1 δ represents the image channel decoder. i This represents the learnable parameters.

[0083] It is understood that this embodiment corresponds to the transmitter in Embodiment 1 above, and adopts an image channel decoder based on an autoencoder. It can be composed of cascaded multi-layer neural networks, including fully connected layers, batch normalization layers and Leacky ReLU activation layers, and finally restores the 512-dimensional image semantic vector to 2048-dimensional image features.

[0084] Step 2: Obtain the text features of the given image.

[0085] Specifically, at the relay node, the stored text data is encoded using a text semantic encoder to obtain text features, the expression of which is:

[0086] f t =S t (T;α t ),

[0087] Among them, f t S represents text features. t Represents a text semantic encoder, T = {t1, t2, ..., t} l} represents a sentence embedding representation with l words, α t This represents the learnable parameters of the corresponding text semantic encoder.

[0088] Optionally, before extracting semantic features from the text, the words in the sentence are first mapped to sentence embedding representations E through an embedding layer. tThe text semantic encoder is constructed using a pre-trained BERT model and a Bi-LSTM model. The BERT model contains a 12-layer encoder-decoder structure, 12 multi-head attention modules, and 110M parameters, resulting in a text feature dimension of 768 dimensions. The Bi-LSTM model has an input dimension of 768 dimensions and an output dimension of 768 dimensions. It can simultaneously learn bidirectional text semantic information at time t, while avoiding long-term dependency issues.

[0089] Step 3: Perform multimodal feature fusion between the text features and the restored image features to obtain multimodal features.

[0090] Specifically, the recovered image features f i and text features f t The input multimodal feature fusion module organically aggregates complementary information from image and text features to obtain multimodal features:

[0091] f m =W(concat(f) i ,f t ))+b

[0092] Among them, f m denoted by , concat represents the concatenation operation, W represents the weight matrix, and b represents the corresponding bias constant.

[0093] More specifically, the multimodal feature fusion module first recovers the image features f i and text features f t The features are compressed into 128 dimensions each, and then concatenated into 256-dimensional features. The resulting features are then passed through a multilayer perceptron to obtain multimodal features.

[0094] Step 4: Encode the multimodal features into multimodal semantic vectors and send them to the receiver.

[0095] Specifically, the multimodal features f obtained in step three m The input is fed into an autoencoder-based multimodal channel encoder, where it is encoded into a transmitted multimodal semantic vector:

[0096] x m =C m (f m ;β m ),

[0097] Where, x m f represents the multimodal semantic vector of the transmission. m Representing multimodal features, C m Represents a multimode channel encoder, β m This represents the learnable parameters.

[0098] Optionally, the autoencoder-based multimodal channel encoder can be composed of cascaded multilayer neural networks, ultimately yielding a 512-dimensional multimodal semantic vector.

[0099] At the receiver, a multimodal channel decoder is first used to recover the multimodal semantic vectors to obtain the multimodal features. The recovered multimodal features are as follows:

[0100]

[0101] Among them, f m The recovered multimodal features, x m C represents the multimodal semantic vector received by the receiver. m -1 Denotes a multimodal channel decoder, δ m This represents the learnable parameters.

[0102] Optionally, the multimodal channel decoder is similar in structure to the image channel decoder, ultimately yielding 512-dimensional multimodal features.

[0103] Then, the recovered target multimodal features are input into the classifier to obtain the predicted target category. Simultaneously, the cross-entropy loss is calculated between the predicted result and the input category label.

[0104]

[0105] Where N represents the number of categories, y ic The sign function is p, which takes the value 1 if the class of sample i is c, and 0 otherwise. ic This represents the predicted probability that observed sample i belongs to category c.

[0106] Optionally, the classifier consists of a single fully connected layer that maps 512-dimensional multimodal features to a 101-dimensional feature vector as the prediction result.

[0107] In this embodiment, after multiple experiments and fine-tuning, the image semantic encoder, channel codec, multimodal feature fusion module and classifier adopted stochastic gradient descent (SGD) as the optimizer for network training, with a learning rate set to 0.001, multiplied by 0.1 every 3 training epochs; the text semantic encoder adopted adaptive moment estimation (Adam) as the optimizer for network training, with the network learning rate set to 0.0001.

[0108] This embodiment utilizes multimodal semantic relay to recover and reconstruct image features and extract text features for modality fusion. Compared with traditional communication technologies and single-modal methods, it enriches the semantic information of feature vectors, improves the problem of modality loss under limited communication resources, and further improves the accuracy of image classification.

[0109] Example 3

[0110] Based on Embodiment 1 above, this embodiment provides a semantic communication device for image classification tasks applied at the transmitter end. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of a semantic communication device for image classification tasks applied at the transmitter end, provided by an embodiment of the present invention, comprising:

[0111] Image semantic encoder 11, used to obtain semantic features of a given image;

[0112] Image channel encoder 12 is used to encode semantic features into image semantic vectors based on an autoencoder and send them to relay nodes.

[0113] Specifically, the image semantic encoder 11 adopts an autoencoder-based structure, which can be composed of cascaded multi-layer neural networks, including fully connected layers, batch normalization layers and Leacky ReLU activation layers, and the final image semantic vector has a dimension of 512.

[0114] The apparatus provided in this embodiment can implement the method provided in Embodiment 1 above. For details of the process, please refer to Embodiment 1 above, which will not be described in detail here.

[0115] Therefore, the device provided in this embodiment can not only significantly reduce the communication resources occupied, but also resist channel noise to a certain extent, thereby further improving receiver performance.

[0116] Example 4

[0117] Based on Embodiment 2 above, this embodiment provides a semantic communication device for image classification tasks applied at relay nodes. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of a semantic communication device for image classification tasks applied at a relay node, provided by an embodiment of the present invention, comprising:

[0118] The image channel decoder 21 is used to receive and decode the image semantic vector sent by the transmitter to obtain the recovered image features; wherein, the image semantic vector is obtained by encoding the semantic features of a given image based on an autoencoder;

[0119] Text semantic encoder 22 is used to obtain text features of a given image;

[0120] The multimodal feature fusion module 23 is used to fuse text features with recovered image features to obtain multimodal features;

[0121] The multimodal channel encoder 24 is used to encode multimodal features into multimodal semantic vectors and send them to the receiver.

[0122] Specifically, in this embodiment, the image channel decoder 21 adopts an autoencoder-based structure, which is built from a multi-layer neural network, including a fully connected layer, a batch normalization layer and a Leacky ReLU activation layer, and finally restores the 512-dimensional image semantic vector to 2048-dimensional image features.

[0123] The text semantic encoder22 consists of a BERT cascaded with a Bi-LSTM to extract text modal features. The BERT model contains a 12-layer encoder-decoder structure, 12 multi-head attention modules, and 110M parameters, resulting in a text feature dimension of 768. The Bi-LSTM model has an input dimension of 768 and an output dimension of 768, allowing it to simultaneously learn bidirectional text semantic information at time t while avoiding long-term dependency issues.

[0124] The multimodal channel encoder 24 also adopts an autoencoder-based structure, which can be composed of cascaded multilayer neural networks to ultimately obtain a 512-dimensional multimodal semantic vector.

[0125] The apparatus provided in this embodiment can implement the method provided in Embodiment 2 above. For details, please refer to Embodiment 2 above, which will not be described in detail here.

[0126] Therefore, the device provided in this embodiment enriches the semantic information of the feature vector, improves the problem of modality loss under limited communication resources, and further improves the accuracy of image classification.

[0127] Example 5

[0128] This embodiment, based on embodiments one, two, three, and four above, provides a semantic communication system for image classification tasks. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the structure of a semantic communication system for image classification tasks provided in an embodiment of the present invention. It includes a transmitter, relay nodes, and a receiver, and its operation is as follows: Figure 6 As shown, where,

[0129] The transmitter is used to acquire the semantic features of a given image and encode them into a transmitted image semantic vector based on an autoencoder;

[0130] The relay node is used to extract the text features of a given image and fuse them with the image features recovered by the image channel decoder for multimodal feature fusion. Then, the multimodal semantic vector is obtained by the multimodal channel encoder and forwarded to the receiver.

[0131] The receiver is used to receive multimodal semantic vectors and perform image classification tasks.

[0132] The transmitter includes the semantic communication device for image classification tasks provided in Embodiment 3 above, and the relay node includes the semantic communication device for image classification tasks provided in Embodiment 4 above.

[0133] The semantic communication method provided in this embodiment can be found in Embodiments 1 and 2 above, and will not be described in detail here.

[0134] Therefore, the semantic communication system provided in this embodiment introduces an autoencoder structure into semantic communication, mapping the extracted feature vectors of different dimensions into semantic vectors of a certain length transmitted in the wireless channel according to different situations, which significantly reduces the communication resources occupied. On the other hand, it uses a multimodal semantic relay method to replace the traditional relay receiving and forwarding methods. At the relay node, it uses stored text data to supplement semantic knowledge and integrates it into the transmitted semantic vector, resulting in more complete multimodal features and multimodal semantic vectors to combat image data loss and help improve the accuracy of image classification.

[0135] Example 6

[0136] The following simulation experiment demonstrates the semantic communication method for image classification tasks proposed in this invention. Existing single-modal classification methods are used as a baseline and compared with these methods to further illustrate the effectiveness of this invention. Furthermore, to verify the robustness of the classification accuracy of this invention in extreme communication environments, the experiment simulates this situation by masking 50% of the transmitted image.

[0137] 1. Simulation experimental conditions:

[0138] Operating system: Ubuntu 20.04, Python 3.8

[0139] Experimental platform: PyTorch-1.7.1

[0140] Graphics card: NVIDIA RTX 2080 Ti GPU

[0141] Video memory: 12GB

[0142] 2. Simulation Experiment Content and Result Analysis:

[0143] Simulation Experiment 1: Image Classification Accuracy Experiment

[0144] It should be noted that the simulated channel is assumed to be an additive white Gaussian noise (AWGN) channel, and the signal-to-noise ratio (SNR) is the same during the training and testing phases. All subsequent experiments were conducted in the same experimental environment. This experiment uses the classic image classification dataset 1. Since there are similar or easily confused images in different categories, it is necessary to accurately recover the semantic features of the target so that the receiver can successfully identify the target category. The classification accuracy using dataset 1 compared to the baseline method and the method proposed in this invention is shown in Table 1 below.

[0145] Table 1. Comparison of classification accuracy between the baseline method and the method proposed in this invention under Dataset 1.

[0146]

[0147]

[0148] As shown in Table 1, the image classification accuracy of the proposed method is 62.86% and 85.56% under signal-to-noise ratio (SNR) conditions of -10dB and -5dB, respectively. In contrast, the baseline method only achieves a classification accuracy of approximately 62% at an SNR of 10dB. Furthermore, the proposed method, with 50% of the image masked, shows only a slight performance decrease, achieving classification accuracies of 60.00%, 85.48%, 86.60%, 86.99%, and 87.02% at different SNRs, representing improvements of 48.98%, 55.68%, 42.96%, 38.61%, and 36.88% respectively compared to the baseline method with 50% image masking. Experiments demonstrate that even with the absence of some image modalities due to extreme channel conditions, the proposed method, utilizing an autoencoder and multimodal semantic relay, can still reliably perform image classification tasks, significantly improving classification accuracy.

[0149] Simulation Experiment 2: Visualization Experiment Comparing the Importance of Multimodal Semantic Relay Modules in the Model

[0150] It should be noted that all the following experiments were conducted in the same experimental environment. The baseline method did not use the multimodal semantic relay module proposed in this invention, but instead used the method of directly forwarding image features in the experimental study. The experimental results are shown in Table 2.

[0151] Table 2 Comparison of the importance of multimodal semantic relay modules in the model (Visualization Experiment)

[0152]

[0153] As shown in Table 2, the baseline method accurately classifies the complete image at a signal-to-noise ratio (SNR) of 10 dB. However, the baseline method exhibits significant errors in its predictions when the SNR is -5 dB or when the input is a masked image. In contrast, the method proposed in this invention consistently produces correct predictions.

[0154] The experimental results above show that introducing a multimodal semantic relay module into semantic communication can significantly improve the accuracy of image classification tasks.

[0155] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A semantic communication method for image classification tasks, applied to a relay node, characterized in that, The method comprises the following steps: receiving and decoding the image semantic vector sent by the transmitter to obtain the recovered image feature; wherein the image semantic vector is obtained by encoding the semantic feature of a given image based on a self-encoder; obtaining the text feature of the given image; performing multi-modal feature fusion on the text feature and the recovered image feature to obtain a multi-modal feature; encoding the multi-modal feature into a multi-modal semantic vector and sending it to the receiver.

2. The semantic communication method for image classification tasks according to claim 1, wherein, The step of receiving and decoding the image semantic vector sent by the transmitter to obtain the recovered image feature comprises the following steps: recovering the image semantic information by using an image channel decoder based on a self-encoder to obtain the recovered image feature, and the expression is as follows: wherein, denotes the recovered image feature, denotes the image semantic vector received by the relay node, denotes the image channel decoder, denotes the learnable parameters.

3. The semantic communication method for image classification task according to claim 1, characterized in that, The step of obtaining the text feature of the given image comprises the following steps: encoding the stored text data by using a text semantic encoder to obtain the text feature, and the expression is as follows: wherein, denotes a text feature, denotes a text semantic encoder, denotes a sentence embedding representation with words, denotes the corresponding text semantic encoder learnable parameters.

4. The semantic communication method for image classification tasks according to claim 1, wherein, The step of performing multi-modal feature fusion on the text feature and the recovered image feature to obtain a multi-modal feature comprises the following steps: inputting the text feature and the recovered image feature into a multi-modal feature fusion module to obtain a multi-modal feature, and the expression is as follows: wherein, denotes a multi-modal feature, denotes a recovered image feature, denotes a text feature, denotes a concatenation operation, denotes a weight matrix, denotes a corresponding bias constant.

5. The semantic communication method for image classification tasks according to claim 1, wherein, The step of encoding the multi-modal fusion feature into a multi-modal semantic vector comprises the following steps: encoding the multi-modal feature into a transmitted multi-modal semantic vector by using a multi-modal channel encoder, and the expression is as follows: wherein, denotes a transmitted multimodal semantic vector, denotes a multimodal feature, denotes a multimodal channel encoder, denotes a learnable parameter.

6. A semantic communication device for image classification tasks, applied to a relay node, characterized in that, The method comprises the following steps: an image channel decoder (21) is used to receive and decode the image semantic vector sent by the transmitter to obtain the recovered image feature; wherein the image semantic vector is obtained by encoding the semantic feature of a given image based on a self-encoder; a text semantic encoder (22) is used to obtain the text feature of the given image; a multi-modal feature fusion module (23) is used to perform multi-modal feature fusion on the text feature and the recovered image feature to obtain a multi-modal feature; a multi-modal channel encoder (24) is used to encode the multi-modal feature into a multi-modal semantic vector and send it to the receiver.

7. A semantic communication system for image classification tasks, characterized in that, The method comprises the following steps: The transmitter is used to obtain the semantic feature of a given image and encode it into a transmitted image semantic vector based on a self-encoder; The relay node is used to extract the text feature of the given image, perform multi-modal feature fusion on the text feature and the image feature recovered by the image channel decoder, and then obtain a multi-modal semantic vector by using a multi-modal channel encoder to forward it to the receiver; The receiver is used to receive the multi-modal semantic vector and perform an image classification task; The relay node comprises the semantic communication device for an image classification task according to claim 6.

Images