A Transformer-based semantic communication rate control method

Abstract

The present invention proposes a Transformer-based semantic communication rate control method. A new training method is adopted when deciding the network and the encoder and decoder of multiple code rates, so that the neural network can still converge and achieve good results under multiple code rates. The utility problem of communication is taken into consideration. Under good channel conditions, good results can be achieved even with a lower code rate. However, compared with high-code rate transmission, the channel resources required for transmission can be saved more when a low code rate is used. AI technology is used to transmit text, and the semantic information of the text is taken into consideration. Compared with traditional encoding methods, encoding is performed according to the semantics of the text. Under low signal-to-noise ratio conditions, the accuracy of text transmission is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of semantic communication technology, and in particular to a semantic communication code rate control method based on Transformer. Background Art

[0002] According to the information theory proposed by Shannon and Weaver, communication can be divided into three levels. The first level is the transmission problem, which mainly studies how to accurately transmit communication symbols. The second level is the semantic problem, which mainly studies how to accurately convey the semantics in communication symbols. The third level is the utility problem, which mainly solves the problem of how the received semantics can effectively affect behavior in the expected way. Due to the limitations of the times, in the more than 70 years since Shannon established information theory, many scholars have made a lot of attempts to approach the Shannon limit, but these works are mainly focused on the first level of communication, that is, how to accurately transmit communication symbols. In recent years, the development of related technologies such as artificial intelligence and natural language processing has made it possible to explore the second level of semantic communication, and semantic communication will gradually become a major research trend in the field of communication.

[0003] In the process of communication, we need to face a variety of channels with different signal-to-noise ratios and types. For different channels, choosing the appropriate bit rate for transmission can often reduce the required channel resources while ensuring the accuracy of communication. The research focus of existing semantic communication systems is still on how to use different neural networks with better performance for transmission, without considering the significant impact of bit rate on communication transmission. Therefore, in the context of semantic communication, this paper explores how to flexibly select the transmission bit rate based on the semantic information of the sentence through the neural network, while ensuring the accuracy of communication, saving the channel resources used, and fully considering the utility of communication. Summary of the invention

[0004] The purpose of the present invention is to provide a Transformer-based semantic communication rate control method to overcome the deficiencies in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] This application discloses a Transformer-based semantic communication bit rate control method, which specifically includes the following steps:

[0007] S1, training of encoders and decoders with multiple bit rates;

[0008] S2, training the decision network;

[0009] S3. Based on the common knowledge of the receiving parties, all the words that may be used in the transmission process between the two parties are numbered to create a dictionary;

[0010] S4, using the input sentence embedding module to embed the input sentence and add the position vector; using the target sentence word embedding module to embed the target sentence and add the position vector; obtaining the word vector with the position vector corresponding to the input sentence and the word vector with the position vector corresponding to the target sentence;

[0011] S5. The word vector with the position vector corresponding to the input sentence passes through the semantic encoding layer and the Encoder layer of Transformrer to complete the semantic encoding and obtain the semantic encoding vector.

[0012] S6, taking the semantic coding vector and the signal-noise condition of the wireless channel to be transmitted as input, and sending them to the decision network, the decision network makes a corresponding code rate selection according to the semantic quantity of the semantic coding vector and the signal-noise condition;

[0013] S7, after completing the code rate selection, the semantic coding vector is sent to the wireless channel;

[0014] S8. The decoding end receives the semantic coding vector through the wireless channel, performs a semantic decoding through the decoding layer of the Transformer, and outputs the decoded sentence text after probabilistic logistic regression processing.

[0015] Preferably, the specific steps in step S1 are as follows:

[0016] S11. Consider the actual usage scenario channel conditions and power consumption conditions to test the performance of different bit rates;

[0017] S12, according to the actual performance of different bit rates in S11, perform bit rate division, determine the highest bit rate to be used, and the lowest bit rate that can be received while ensuring the correct rate, and subdivide between the two bit rates according to usage requirements to determine the number of bit rates to be used;

[0018] S13, selecting the highest bit rate case, and training a high bit rate encoder and decoder;

[0019] S14. Randomly select a code rate from among the several code rates determined in S12. According to the randomly selected code rate, select the text vector dimension corresponding to the code rate, perform a zero-padding operation on the remaining dimensions to pad them to the input and output dimensions corresponding to the high code rate; and obtain the corresponding multi-code rate encoder and decoder.

[0020] Preferably, the specific steps in step S2 are as follows:

[0021] S21. Create a decision network consisting of multiple fully connected layers;

[0022] S22, using the multi-rate encoder and decoder in step S1, a test is performed on the training text, and corresponding labels are given to the training text according to the performance of the training text at different code rates and different channel signal-to-noise ratios;

[0023] S23, using the label result in S22, updating the used training text, adding the channel information and the code rate label corresponding to the channel information to the training text, to obtain a new training text;

[0024] S24, the decision network uses the new training text obtained in S23 to perform a classification task training. The input information of the training is the transmission text and the wireless channel information to be transmitted, and the output information is the corresponding bit rate label. The cross entropy of the bit rate label predicted by the network and its corresponding actual label is used as input to update the decision network parameters, thereby completing the training of the decision network.

[0025] Preferably, the specific steps in step S3 are as follows:

[0026] S31, read in the entire text file for transmission;

[0027] S32, performing word segmentation processing on the entire text, counting the number of times each word in the text is used, numbering each word, and removing words with too low a number of times;

[0028] S33, adding a start or end character to the entire dictionary;

[0029] S34. Output the dictionary.

[0030] Preferably, the specific steps in step S4 are as follows:

[0031] S41. Create an embedding layer, send the sentence to be transmitted into the embedding layer, and convert it into a word vector of corresponding dimension;

[0032] S42, calculate and add the position vector;

[0033] S43. Add the word vector to the position vector to obtain a word vector with position information.

[0034] Preferably, the specific steps in step S5 are as follows:

[0035] S51, define three matrices, and perform three linear changes on the word vector with the position vector corresponding to the input sentence according to the three matrices to obtain a query vector, a key vector, and a value vector;

[0036] S52, performing self-attention calculation on the query vector, the key vector and the value vector to obtain an attention vector;

[0037] S53, performing a residual connection, adding the attention vector to the word vector with the position vector corresponding to the input sentence, and performing a layer normalization operation on the obtained result to obtain a residual vector;

[0038] S54, feed-forward transmission operation is performed on the obtained residual vector, and a semantic coding vector is obtained by activating it through two layers of linear mapping and using the activation function ReLU.

[0039] Preferably, the process of bit rate selection in step S6 is as follows:

[0040] S61, sending the semantic encoding vector to a fully connected layer with an output dimension of 1 to integrate the semantic information and obtain an integrated semantic vector;

[0041] S62, concatenating the above-integrated semantic vector with the signal-noise information of the wireless channel to be transmitted to obtain a concatenated vector;

[0042] S63: Input the concatenated vector into a fully connected layer to perform a classification task, and select the bit rate required for the semantic quantity and signal-to-noise condition.

[0043] Preferably, the decoding process in step S8 is as follows:

[0044] S81, input the word vector with position vector corresponding to the target sentence obtained in S4 into the multi-head self-attention layer for decoding;

[0045] S82, input the encoded information received through the wireless channel and the output information of the multi-head self-attention layer in the previous step into the multi-head attention layer for decoding;

[0046] S83, after the target sentence passes through the multi-head self-attention layer and the multi-head attention layer, it passes through the feed-forward transmission layer to obtain a semantic decoding vector;

[0047] S84. Perform probabilistic logistic regression through the Softmax function and output the sentence.

[0048] Beneficial effects of the present invention:

[0049] 1. A new training method is adopted when deciding the network and the encoder and decoder of multiple bit rates, so that the neural network can still converge and achieve good results under multiple bit rates;

[0050] 2. Considering the utility of communication, when the channel conditions are good, even if a lower bit rate is used, good results can be achieved. However, compared with high bit rate transmission, low bit rate can save more channel resources required for transmission.

[0051] 3. AI technology is used to transmit text, taking into account the semantic information of the text. Compared with traditional encoding methods, encoding is performed based on the semantics of the text. Under low signal-to-noise ratio conditions, the accuracy of text transmission has been significantly improved.

[0052] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of a framework of a Transformer-based semantic communication bit rate control method of the present invention;

[0054] Figure 2 It is a schematic diagram of the overall structure of a Transformer-based semantic communication bit rate control method of the present invention;

[0055] Figure 3 It is a schematic diagram of the training process of the multi-rate encoder and decoder of the present invention;

[0056] Figure 4 It is a schematic diagram of the training process of the decision network of the present invention;

[0057] Figure 5 It is a schematic diagram of the overall performance of the semantic communication system when using different bit rates;

[0058] Figure 6 This is a diagram of the average number of bits used by the Policy network; DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0060] See also Figure 3 , the training process of the multi-rate encoder and decoder of the present invention:

[0061] S11. Consider the actual usage scenario channel conditions and power consumption conditions to test the performance of different bit rates;

[0062] S12, according to the actual performance of different bit rates in S11, perform bit rate division, determine the highest bit rate to be used, and the lowest bit rate that can be received while ensuring the correct rate, and subdivide between the two bit rates according to usage requirements to determine the number of bit rates to be used;

[0063] S13, selecting the highest bit rate case, and training a high bit rate encoder and decoder;

[0064] S14. Randomly select a code rate from among the several code rates determined in S12. According to the randomly selected code rate, select the text vector dimension corresponding to the code rate, perform a zero-padding operation on the remaining dimensions to pad them to the input and output dimensions corresponding to the high code rate; and obtain the corresponding multi-code rate encoder and decoder.

[0065] See also Figure 4 , the training process of the decision network:

[0066] S21. Create a decision network consisting of multiple fully connected layers;

[0067] S22, using the multi-rate encoder and decoder in step S1, a test is performed on the training text, and corresponding labels are given to the training text according to the performance of the training text at different code rates and different channel signal-to-noise ratios;

[0068] S23, using the label result in S22, updating the used training text, adding the channel information and the code rate label corresponding to the channel information to the training text, to obtain a new training text;

[0069] S24. The decision network uses the new training text obtained in S23 to perform training for a classification task. The input information of the training is the transmission text and the wireless channel information to be transmitted, and the output information is the corresponding bit rate label. The cross entropy of the bit rate label predicted by the network and its corresponding actual label is used as input to update the decision network parameters, thereby completing the training of the decision network.

[0070] See also Figure 1 and Figure 2 The specific steps of this program are:

[0071] Based on the common knowledge of the receiving parties, all the words that may be used in the transmission process between the two parties are numbered to create a dictionary;

[0072] The input sentence is embedded in the input sentence using the input sentence embedding module and a position vector is added; the target sentence is embedded in the target sentence using the target sentence embedding module and a position vector is added; the word vector with the position vector corresponding to the input sentence and the word vector with the position vector corresponding to the target sentence are obtained;

[0073] Semantic encoding: After the transmitted sentence passes through the semantic encoding layer, it passes through the Encoder layer of Transformrer to complete semantic encoding according to the semantic information of the sentence;

[0074] Bit rate selection: the transmission sentence and the signal-noise condition of the transmission channel are used as input and sent to the decision network. The decision network makes corresponding bit rate selection according to the semantic content of the transmitted text and the signal-noise condition of the channel.

[0075] After completing the code rate selection, the transmission statement is sent to the wireless channel;

[0076] The decoding end receives the sentence through the wireless channel, performs a semantic decoding through the Transformer decoding layer, and outputs the decoded sentence text after probabilistic logistic regression processing.

[0077] Specifically, in step 1, the process of creating a dictionary based on the common information is as follows:

[0078] Step 1: Read the entire text file for transmission;

[0079] Step 2: Segment the entire text, count the number of times each word is used in the text, number each word, and remove words that are used too few times;

[0080] Step 3: Add characters with special meanings such as start or end to the entire dictionary;

[0081] Step 4: Output the dictionary.

[0082] Word embedding and position encoding, the specific process is:

[0083] Step 1: Create an embedding layer, send the sentence to be transmitted into the embedding layer, and convert it into a word vector of corresponding dimension;

[0084] Step 2: Add the position vector. The formula is as follows:

[0085]

[0086]

[0087] Among them, pos refers to the position of a word in a sentence, and its value range is [0, L), where L is the length of the sentence, i refers to the dimension number of the subvector, and its value range is [0, embedding_dimension / 2), where embedding_dimension is the embedding dimension, and d model Refers to the value of the embedding dimension embedding_dimension of the Embedding layer;

[0088] Step 3: After obtaining the word vector of each word and its corresponding position vector, add the two together to get the word vector X with position information. Embedding .

[0089] The Encoder layer of Transformer is used for semantic encoding. The specific process is as follows:

[0090] Step 1: Define three matrices W Q ,W K ,W V ,According to these three matrices, the word vector X obtained in the previous step Embedding Perform three linear changes to obtain the query vector Q, key vector K, and value vector V;

[0091] Step 2: Perform a self-attention calculation on the three vectors obtained above;

[0092]

[0093] Get vector X attention , where d k Represents the dimension of the vector;

[0094] Step 3: Perform residual connection and convert the X attention and X obtained in the second step Ebedding Add and normalize the result to get the residual vector X' attention , where X' attention =layernorm(X+X attention );

[0095] Step 4: Get the residual vector X' attention Perform feedforward transmission operation, obtain vector X through two layers of linear mapping and activation with activation function ReLU hidden

[0096] X hidden =Linear(ReLU(Linear(X attention )));

[0097] The process of bit rate selection is as follows:

[0098] Step 1: The semantic vector obtained from the encoder is fed into a fully connected layer with an output dimension of 1 to integrate the semantic information in the sentence;

[0099] Step 2: Concatenate the integrated semantic vector obtained in the previous step with the signal-to-noise information of the wireless channel to be transmitted;

[0100] Step 3: Input the concatenated vector obtained in the previous step into a fully connected layer to perform a classification task and select the required bit rate under the semantic quantity and signal-to-noise conditions.

[0101] Decoding process:

[0102] Step 1: Input the word embedding vector of the target sentence obtained in step 2 into the multi-head self-attention layer for decoding;

[0103] Step 2: Input the encoded information received through the wireless channel and the output information of the multi-head self-attention layer in the previous step into the multi-head attention layer for decoding;

[0104] Step 3: After the target sentence passes through the multi-head self-attention layer and the multi-head attention layer, it passes through the feed-forward transmission layer to obtain the semantic decoding vector;

[0105] Step 4: Perform probabilistic logistic regression through the Softmax function and output the sentence.

[0106] See also Figure 5 , where the bit rates corresponding to MODE0, 1, and 2 are MODE0, the lowest, 8 bits, MODE2, the highest, 16 bits, and MODE1, 12 bits, respectively. The Policy curve shows the performance of the semantic communication system after the decision network is used. After the decision network is added, the performance curve of the semantic communication system basically coincides with the case of using the highest bit rate in MODE2. Compared with MODE2 using 16 bits, the average number of bits used by the Policy network is as follows Figure 6 As shown, it is much smaller than the number of bits required in MODE2.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A semantic communication rate control method based on Transformer, characterized in that: The specific steps include: S1, training of encoders and decoders with multiple bit rates; S2, training the decision network; S3. Based on the common knowledge of the receiving parties, all the words that may be used in the transmission process between the two parties are numbered to create a dictionary; S4, using the input sentence embedding module to embed the input sentence and add the position vector; Use the target sentence word embedding module to embed the target sentence and add the position vector; Get the word vector with position vector corresponding to the input sentence and the word vector with position vector corresponding to the target sentence; S5. The word vector with the position vector corresponding to the input sentence passes through the semantic encoding layer and the Encoder layer of Transformrer to complete the semantic encoding and obtain the semantic encoding vector. S6, taking the semantic coding vector and the signal-noise condition of the wireless channel to be transmitted as input, and sending them to the decision network, the decision network makes a corresponding code rate selection according to the semantic quantity of the semantic coding vector and the signal-noise condition; S7, after completing the code rate selection, the semantic coding vector is sent to the wireless channel; S8. The decoding end receives the semantic coding vector through the wireless channel, performs a semantic decoding through the decoding layer of the Transformer, and outputs the decoded sentence text after probabilistic logistic regression processing.

2. A Transformer-based semantic communication rate control method as claimed in claim 1, characterized in that: The specific steps in step S1 are as follows: S11. Consider the actual usage scenario channel conditions and power consumption conditions to test the performance of different bit rates; S12, according to the actual performance of different bit rates in S11, perform bit rate division, determine the highest bit rate to be used, and the lowest bit rate that can be received while ensuring the correct rate, and subdivide between the two bit rates according to usage requirements to determine the number of bit rates to be used; S13, selecting the highest bit rate case, and training a high bit rate encoder and decoder; S14. Randomly select a code rate from among the several code rates determined in S12. According to the randomly selected code rate, select the text vector dimension corresponding to the code rate, perform a zero-padding operation on the remaining dimensions to pad them to the input and output dimensions corresponding to the high code rate; and obtain the corresponding multi-code rate encoder and decoder.

3. A Transformer-based semantic communication rate control method as claimed in claim 1, characterized in that: The specific steps in step S2 are as follows: S21. Create a decision network consisting of multiple fully connected layers; S22, using the multi-rate encoder and decoder in step S1, a test is performed on the training text, and corresponding labels are given to the training text according to the performance of the training text at different code rates and different channel signal-to-noise ratios; S23, using the label result in S22, updating the used training text, adding the channel information and the code rate label corresponding to the channel information to the training text, to obtain a new training text; S24. The decision network uses the new training text obtained in S23 to perform training for a classification task. The input information of the training is the transmission text and the wireless channel information to be transmitted, and the output information is the corresponding bit rate label. The cross entropy of the bit rate label predicted by the network and its corresponding actual label is used as input to update the decision network parameters, thereby completing the training of the decision network.

4. A Transformer-based semantic communication rate control method as claimed in claim 1, characterized in that: The specific steps in step S3 are as follows: S31, read in the entire text file for transmission; S32, performing word segmentation processing on the entire text, counting the number of times each word in the text is used, numbering each word, and removing words with too low a number of times; S33, adding a start or end character to the entire dictionary; S34. Output the dictionary.

5. A Transformer-based semantic communication rate control method as claimed in claim 1, characterized in that: The specific steps in step S4 are as follows: S41. Create an embedding layer, send the sentence to be transmitted into the embedding layer, and convert it into a word vector of the corresponding dimension; S42, calculate and add the position vector; S43. Add the word vector to the position vector to obtain a word vector with position information.

6. A Transformer-based semantic communication bit rate control method as claimed in claim 1, characterized in that: The specific steps in step S5 are as follows: S51, define three matrices, and perform three linear changes on the word vector with the position vector corresponding to the input sentence according to the three matrices to obtain a query vector, a key vector, and a value vector; S52, performing self-attention calculation on the query vector, the key vector and the value vector to obtain an attention vector; S53, performing a residual connection, adding the attention vector to the word vector with the position vector corresponding to the input sentence, and performing a layer normalization operation on the obtained result to obtain a residual vector; S54, feed-forward transmission operation is performed on the obtained residual vector, and a semantic coding vector is obtained by activating it through two layers of linear mapping and using the activation function ReLU.

7. A Transformer-based semantic communication rate control method as claimed in claim 1, characterized in that: The process of bit rate selection in step S6 is as follows: S61, sending the semantic encoding vector to a fully connected layer with an output dimension of 1 to integrate the semantic information and obtain an integrated semantic vector; S62, concatenating the above-integrated semantic vector with the signal-noise information of the wireless channel to be transmitted to obtain a concatenated vector; S63: Input the concatenated vector into a fully connected layer to perform a classification task, and select the bit rate required for the semantic quantity and signal-to-noise condition.

8. The Transformer-based semantic communication rate control method according to claim 1, characterized in that: The decoding process in step S8 is as follows: S81, input the word vector with position vector corresponding to the target sentence obtained in S4 into the multi-head self-attention layer for decoding; S82, input the encoded information received through the wireless channel and the output information of the multi-head self-attention layer in the previous step into the multi-head attention layer for decoding; S83, after the target sentence passes through the multi-head self-attention layer and the multi-head attention layer, it passes through the feed-forward transmission layer to obtain a semantic decoding vector; S84. Perform probabilistic logistic regression through the Softmax function and output the sentence.

Images