Semantic communication method and system, electronic equipment and computer readable storage medium

Abstract

The invention provides a semantic communication method and system, electronic equipment and a computer readable storage medium, and relates to the field of semantic communication and artificial intelligence. According to the specific implementation scheme, the method comprises the steps of inputting information source data transmitted by a transmitting end into a pre-trained semantic encoder for semantic encoding, and obtaining information source semantic features of the information source data; screening the information source semantic features through a pre-obtained importance template, and converting the screened information source semantic features into a bit stream; transmitting the bit stream to the receiving end through the communication channel; reducing the bit stream received by the receiving end into reduced semantic features through the importance template, and inputting the reduced semantic features into a pre-trained semantic decoder for semantic decoding to obtain recovery information; wherein the importance template is obtained by training a target data set; the semantic encoder and the semantic decoder are obtained based on the importance template training.

Description

Technical Field

[0001] The present disclosure relates to the fields of semantic communication and artificial intelligence technology. Specifically, the present disclosure relates to a semantic communication method and system, an electronic device, and a computer-readable storage medium. Background Art

[0002] Traditional communications focus on the accurate transmission of bits. Noise and interference can seriously affect the accurate transmission of information, resulting in information distortion and loss.

[0003] Semantic communication is a communication method that focuses on the accurate transmission of information meaning and semantics. Compared with traditional communication, it pays more attention to semantic understanding and expression rather than relying solely on the accurate transmission of bits.

[0004] The core of semantic communication is to convert natural language or other forms of information into machine-understandable semantic representations in order to more accurately capture the meaning and purpose of the information. In semantic communication, the transmitter sends content with semantic information, while the receiver interprets and understands the meaning of this semantic information. By focusing on the semantic level of information during the communication process, more accurate and targeted transmission and understanding can be achieved. Summary of the invention

[0005] The present disclosure provides a semantic communication method and system, an electronic device, and a computer-readable storage medium.

[0006] According to a first aspect of the present disclosure, a semantic communication method is provided. The method is used in a semantic communication system. The semantic communication system includes a transmitter, a receiver, and a communication channel connecting the transmitter and the receiver. The method includes:

[0007] Inputting the source data transmitted by the transmitting end into a pre-trained semantic encoder for semantic encoding to obtain source semantic features of the source data;

[0008] The information source semantic features are screened by using a pre-acquired importance template, and the screened information source semantic features are converted into a bit stream;

[0009] transmitting the bit stream to the receiving end via the communication channel;

[0010] Restoring the bit stream received by the receiving end into restored semantic features through the importance template, and inputting the restored semantic features into a pre-trained semantic decoder for semantic decoding to obtain recovery information;

[0011] The importance template is obtained by training a target data set; and the semantic encoder and the semantic decoder are obtained by training based on the importance template.

[0012] According to a second aspect of the present disclosure, there is provided a semantic communication system, the system comprising:

[0013] A transmitting end is used to transmit source data, input the source data into a pre-trained semantic encoder for semantic encoding, and obtain source semantic features of the source data; filter the source semantic features through a pre-acquired importance template, and convert the filtered source semantic features into a bit stream;

[0014] a communication channel for transmitting the bit stream from the transmitting end to the receiving end;

[0015] A receiving end, used for receiving the bit stream, restoring the bit stream into restored semantic features through the importance template, and inputting the restored semantic features into a pre-trained semantic decoder for semantic decoding to obtain restored information;

[0016] The importance template is obtained by training a target data set; and the semantic encoder and the semantic decoder are obtained by training based on the importance template.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] A memory in communication with the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the semantic communication method.

[0021] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned semantic communication method.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned semantic communication method when executed by a processor.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1is a flowchart of a semantic communication method provided by an embodiment of the present disclosure;

[0026] Figure 2 It is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0027] Figure 3 It is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0028] Figure 4 It is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0029] Figure 5 It is a flowchart of some steps of another semantic communication method provided by an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of a specific embodiment of a semantic communication method provided by an embodiment of the present disclosure;

[0031] Figure 7 is a structural diagram of a semantic communication device provided by an embodiment of the present disclosure;

[0032] Figure 8 It is a block diagram of an electronic device used to implement the semantic communication method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] The nouns that may appear in the embodiments of the present disclosure and their corresponding explanations are as follows:

[0035] Semantic coding: Acquire the semantic information contained in business information. According to different semantic types, it can be subdivided into text semantic coding, image semantic coding, audio semantic coding, video semantic coding, point cloud semantic coding, etc. The possible categories of semantic coding are related to the possible categories of business information, and can be all possible types of semantic coding for communication transmission. The generation of semantic coding is also related to the semantic coding model, which can adopt all possible models of artificial intelligence, deep learning, pattern recognition and other disciplines.

[0036] Transmitter and receiver: There is a channel connection between the transmitter and the receiver, the transmitter (transmitter) has an encoder, and the receiver (receiver) has a decoder. The present invention is based on semantic coding and semantic decoding, so the transmitter of the present invention has a semantic encoder; the receiver of the present invention has a semantic decoder. The transmitter and receiver of the present invention work at the semantic layer, and the bottom layer is still the Shannon physical layer.

[0037] Model: The model referred to in the present invention includes models in the fields of machine learning, artificial intelligence, neural networks, etc. Models are used to semantically encode and decode business information, and are called semantic encoding models and semantic decoding models. Depending on the type of business information, it can be divided into text models, audio models, image models, video models, point cloud models, one-dimensional waveform models, radar data models, etc. The category of the model is related to the category of semantic encoding and the category of business information, and can be models of all possible types of communication transmission.

[0038] Semantic communication focuses more on the accurate transmission of semantic information rather than the accuracy of bits. In other words, semantic communication allows some bits to be wrong as long as the semantic errors are minimized. Even if the semantic information is affected by noise during transmission and some bits are wrong, the trained model can still recover the correct semantic information. Therefore, semantic communication has strong noise resistance and can still transmit information accurately even if the signal-to-noise ratio fluctuates greatly.

[0039] In some related technologies, based on semantic communication, a semantic importance module is used to calculate the semantic importance of semantic features, and then the parts with lower semantic importance are discarded, and only the parts with higher semantic importance are transmitted, which can further reduce the amount of transmitted data. However, in order to ensure the accurate recovery of semantic features at the receiving end, the index of the retained part in the original semantic feature needs to be accurately transmitted, which not only adds additional overhead, but also puts higher requirements on channel conditions.

[0040] The semantic communication method and system, electronic device, and computer-readable storage medium provided by the embodiments of the present disclosure are intended to solve at least one of the above-mentioned technical problems in the prior art.

[0041] The semantic communication method provided in the embodiments of the present disclosure may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a vehicle-mounted device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method may be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method may be executed by a server.

[0042] Figure 1 FIG. 1 is a flow chart of a semantic communication method provided by an embodiment of the present disclosure. Figure 1 As shown in , the semantic communication method provided by the embodiment of the present disclosure may include step S110, step S120, step S130, and step S140.

[0043] In step S110, the source data transmitted by the transmitting end is input into a pre-trained semantic encoder for semantic encoding to obtain source semantic features of the source data;

[0044] In step S120, the information source semantic features are screened using the pre-acquired importance template, and the screened information source semantic features are converted into a bit stream;

[0045] In step S130, the bit stream is transmitted to a receiving end via a communication channel;

[0046] In step S140, the bit stream received by the receiving end is restored to restored semantic features through the importance template, and the restored semantic features are input into a pre-trained semantic decoder for semantic decoding to obtain recovery information;

[0047] Among them, the importance template is obtained by training the target data set; the semantic encoder and the semantic decoder are trained based on the importance template.

[0048] For example, the semantic communication method provided by the embodiment of the present disclosure is used in a semantic communication system.

[0049] The semantic communication system includes a transmitter, a receiver, and a communication channel connecting the transmitter and the receiver.

[0050] In some possible implementations, in step S110, the source data may be image data, voice data, text data, point cloud data, or a mixture of any of the above data.

[0051] In some possible implementations, the semantic encoder is a model for semantically encoding source data. Therefore, inputting the source data into the semantic encoder can obtain source semantic features of the source data.

[0052] In some possible implementations, the semantic encoder may specifically be a CNN (convolutional neural network), an RNN (recurrent neural network), or the like.

[0053] The embodiments of the present disclosure do not limit the specific type of semantic encoders, and any model that can implement semantic encoding is within the protection scope of the embodiments of the present disclosure.

[0054] In some possible implementations, in step S120, screening the information source semantic features obtained in step S110 by using the pre-acquired importance template may be to obtain the importance of all information source semantic features based on the pre-acquired importance template, and screen out information source semantic features with higher importance based on the importance of the information source semantic features;

[0055] Converting the filtered new energy semantic features into a bit stream may be converting the filtered information source semantic features with higher importance into a bit stream.

[0056] Among them, the bit stream refers to a sequence composed of two binary numbers 0 and 1. Converting the semantic features of the source into the bit stream means converting the semantic features of the source into binary data.

[0057] In some possible implementations, in step S130, since the bit stream can be transmitted through a network or other transmission media, the bit stream can be transmitted through a communication channel.

[0058] Specifically, the transmitting end sends a bit stream, and the receiving end receives the bit stream sent by the transmitting end.

[0059] In some possible implementations, in step S140, the bit stream received by the receiving end is restored to semantic features, the acquired semantic features are restored by using the importance template to obtain restored semantic features, and the restored semantic features are input into a semantic decoder to obtain restored information for restoring the source data. The obtained restored information is the source data received by the receiving end.

[0060] The semantic decoder is a decoder opposite to the semantic encoder, and is a model for semantically decoding semantic features. Therefore, the data corresponding to the semantic features can be obtained by inputting the restored semantic features into the semantic decoder.

[0061] In some possible implementations, the semantic decoder may specifically be a CNN (convolutional neural network), an RNN (recurrent neural network), or the like.

[0062] The embodiments of the present disclosure do not limit the specific type of the semantic decoder, and any model that can implement semantic decoding is within the protection scope of the embodiments of the present disclosure.

[0063] Among them, the importance template used in the embodiment of the present disclosure is obtained by training the target data set. The importance template obtained by training is obtained based on the importance of the features in the target data. The target data set and the source data are of the same type of data and their features are similar. Therefore, the importance template obtained by training the target data is also suitable for the source data.

[0064] At the same time, the semantic encoder and semantic decoder are trained based on the importance template. Therefore, the semantic features obtained by the semantic encoder are consistent with the importance template. After being screened by the importance template, the obtained semantic coding features can preserve important semantic features, minimize the loss of semantic features, and ensure the similarity between the obtained recovery information and the source data, thereby ensuring the quality of the obtained recovery information.

[0065] In some related semantic communication methods, the semantically encoded content of the transmitter is discarded based on the semantic importance and then transmitted. The discarded part is unknown to the receiver and requires additional transmission indexes for correct decoding. And this part of the index needs to be transmitted losslessly, and once an error occurs, it cannot be decoded.

[0066] In the semantic communication method provided by the embodiment of the present disclosure, the transmitter and the receiver use the same importance template. The transmitter discards part of the semantically encoded content of the transmitter based on the semantic importance and then transmits it. The receiver can restore the discarded part based on the importance template, and thus does not need an additional transmission index. This reduces the additional overhead of the transmission index and enables communication at a lower signal-to-noise ratio.

[0067] The semantic communication method provided by the embodiment of the present disclosure is introduced in detail below.

[0068] Figure 2 FIG. 1 shows a flow chart of an implementation method of filtering source semantic features through importance templates and converting the filtered source semantic features into a bit stream, such as Figure 2 As shown, it may include step S210 and step S220.

[0069] In step S210, some elements in the source semantic features are discarded according to a preset ratio using an importance template;

[0070] In step S220, the remaining source semantic features are discarded and converted into a bit stream.

[0071] In some possible implementations, in step S210, the size of the importance template is consistent with the size of the semantic feature output by the semantic encoder, the number of elements included in the importance template is consistent with the number of elements included in the semantic feature output by the semantic feature encoder, and they correspond one-to-one according to position.

[0072] Each element of the importance template is data between 0 and 1. The larger the data value is, the greater the importance of the element of the semantic feature corresponding to the element is.

[0073] In some possible implementations, the elements at each position of the importance template are sorted, and according to a preset ratio, the positions corresponding to the elements before the sorting preset ratio are selected as the filtering positions, the positions corresponding to the remaining data are determined as non-filtering positions, and the elements at the non-filtering positions of the source semantic features are discarded.

[0074] In some possible implementations, in step S220 , the elements at the screening positions are converted into binary data to convert the remaining source semantic features into a bit stream.

[0075] Figure 3 FIG. 1 is a flow chart showing an implementation method of restoring a bit stream received by a receiving end to restore semantic features through an importance template, such as Figure 3 As shown, it may include step S310 and step S320.

[0076] In step S310, the bit stream received by the receiving end is restored to feature data;

[0077] In step S320, the feature data is padded with zeros using an importance template to obtain restored semantic features.

[0078] In some possible implementations, in step S310, binary data received in a bit stream received by a receiving end is restored to data of other bases to obtain feature data.

[0079] In some possible implementations, in step S320, the elements at each position of the importance template are sorted, and the positions corresponding to the elements of the preset proportion before sorting are selected as filtering positions according to a preset proportion, and the positions corresponding to the remaining data are determined as non-filtering positions, the feature data are written into the filtering positions in sequence, and zero-filling operations are performed at the non-filtering positions to generate restored semantic features.

[0080] In some possible implementations, the semantic communication method provided by the embodiments of the present disclosure also includes: a training process of a semantic communication model and an importance template.

[0081] Figure 4 A flowchart of the training process of the semantic communication model and the importance template is shown. Figure 4 As shown, the training process of the semantic communication model and the importance template may include step S410, step S420, step S430, and step S440.

[0082] In step S410, a semantic communication model is trained using a training data set;

[0083] In step S420, the semantic importance of each sample in the target data set is calculated using the trained semantic importance module;

[0084] In step S430, an importance template is generated according to the semantic importance of each sample in the target dataset.

[0085] In step S440, the semantic communication model is fine-tuned according to the importance template to obtain the trained semantic encoder and semantic decoder.

[0086] In some possible implementation manners, in step S410, the semantic communication model includes a semantic encoder, a semantic importance module, and a semantic decoder connected in sequence.

[0087] Among them, the semantic encoder is used to perform semantic encoding on the training data to obtain the semantic features of the training data; the semantic importance module is used to obtain the importance of each element of the semantic features; the semantic decoder is used to perform semantic decoding on the semantic features.

[0088] Training the semantic communication model with the training data can be to simultaneously train the semantic encoder, the semantic importance module, and the semantic decoder with the training data.

[0089] Specifically, the training dataset is input into the semantic encoder to obtain the semantic feature vector corresponding to the training dataset; the semantic feature vector is input into the semantic importance module to obtain the semantic importance of each element in the semantic feature vector; the semantic feature vector is input into the semantic decoder to obtain the decoder output; according to the decoder output, the parameters of the semantic encoder, the semantic importance module, and the semantic decoder are adjusted through backpropagation.

[0090] The embodiments of the present disclosure do not limit the specific types of the models used by the semantic encoder, the semantic importance module, and the semantic decoder. Any model that can implement the corresponding functions is within the protection scope of the embodiments of the present disclosure.

[0091] In some possible implementation manners, in step S420, the target dataset can be input into the semantic encoder to obtain the semantic features of each sample in the target dataset; the semantic features of each sample are input into the semantic importance module to obtain the semantic importance of each element in the semantic features of each sample.

[0092] Among them, inputting the target dataset into the semantic encoder can be to input each sample of the target dataset into the trained semantic encoder respectively to obtain the semantic features of each sample, and input the semantic features of each sample into the semantic importance module to obtain the semantic importance of each element in the semantic features of each sample, so as to obtain the semantic importance of the elements at the same position of different samples in the target dataset.

[0093] In some possible implementation manners, in step S430, the semantic importance of each element is sorted, and an importance template is generated according to the sorting result.

[0094] Specifically, after obtaining the semantic importance of elements at the same position of different samples of the target data set, the mean of the semantic importance of elements at the same position of different samples may be obtained.

[0095] Among them, the mean of the semantic importance of elements at the same position of different samples can be obtained by a method of adding and averaging (i.e. obtaining the sum of the semantic importance of elements at the same position of different samples, and dividing it by the number of samples to obtain the mean), or by a method of calculating the median (i.e. sorting the semantic importance of elements at the same position of different samples, and taking the median as the mean), or by a method of calculating the maximum value (i.e. sorting the semantic importance of elements at the same position of different samples, and taking the maximum value as the mean).

[0096] The mean values ​​of the semantic importance corresponding to different positions are sorted, and the value corresponding to each position of the importance template is determined according to the sorting result.

[0097] Specifically, the mean values ​​of the semantic importance of elements at the same position of different samples can be normalized to values ​​between 0 and 1 through a normalization method, and the normalized values ​​can be filled into corresponding positions of the importance template to generate the importance template.

[0098] In some possible implementations, in step S440, based on the importance template, a trained semantic encoder and a trained semantic decoder are obtained for the semantic communication model.

[0099] Figure 5 A flowchart of an implementation method of obtaining a trained semantic encoder and a semantic decoder of a semantic communication model according to an importance template is shown, as shown in FIG. Figure 5 As shown, it may include step S510, step S520, step S530, and step S540.

[0100] In step S510, the training data set is input into a semantic encoder to obtain a semantic feature vector corresponding to the training data set;

[0101] In step S520, some elements of the semantic feature vector are discarded according to the importance template to obtain the discarded semantic feature vector;

[0102] In step S530, the discarded semantic feature vector is input into a semantic decoder to obtain a decoder output;

[0103] In step S540, the parameters of the semantic encoder and the semantic decoder are adjusted through back propagation according to the decoder output.

[0104] In some possible implementations, in step S510, the training data may be training data different from the training data for training the semantic communication model.

[0105] In some possible implementations, in step S520, similar to step S210, the elements at each position of the importance template are sorted, and according to a preset ratio, the positions corresponding to the elements before the sorting preset ratio are selected as the filtering positions, the positions corresponding to the remaining data are determined as non-filtering positions, and the elements at the non-filtering positions of the semantic feature vector are discarded.

[0106] In some possible implementations, in step S530, similar to step S320, the discarded semantic feature vectors are written into the filter positions in order, and zero padding operations are performed at non-filter positions to generate semantic feature vectors input to the semantic decoder, and the output of the semantic decoder is obtained.

[0107] In some possible implementations, in step S540, a loss function value is determined based on the difference between the output of the semantic decoder and the training data, and the parameters of the semantic encoder and the semantic decoder are adjusted by back-propagating the loss function value.

[0108] By adjusting the parameters of the semantic encoder and the semantic decoder, the positions of important features in the semantic features extracted by the semantic encoder can be matched with the importance template, thereby ensuring that the important features of the source data will not be discarded during the data transmission process, thereby ensuring the similarity between the acquired recovery information and the source data, and ensuring the quality of the data recovered at the receiving end.

[0109] The semantic communication method provided by the embodiment of the present disclosure is specifically introduced below with a specific embodiment.

[0110] This specific embodiment includes a training process and an inference process. Figure 6 It is a schematic diagram of the training process and the inference process, such as Figure 6 The training process includes the following steps:

[0111] 1. Train a semantic communication model with a semantic importance module on other datasets, where the encoder outputs a semantic feature size of 1*96*128*64 when the input is a 2048*1024 image.

[0112] 2. Use the semantic importance module to calculate the semantic importance of each sample in the fine-tuning dataset. The size of semantic importance is 1*96*128*64;

[0113] The fine-tuning dataset used is 5000 images of size 3*2048*1024.

[0114] 3. According to the preset algorithm, the importance template is calculated and saved based on the semantic importance of 5000 samples, with a size of 1*96*128*64.

[0115] 4. Fine-tune the semantic communication network based on the saved importance templates.

[0116] The reasoning process includes the following steps:

[0117] 1. Deploy the fine-tuned model and importance template at both the sending and receiving ends;

[0118] 2. The transmitter uses a semantic encoder to encode a 3*2048*1024 image to obtain a 1*96*128*64 semantic feature;

[0119] 3. The transmitter discards some elements with lower importance based on the importance template and retains a ratio of α=0.5, flattens the semantic features, converts them into binary bit rate, and then transmits them;

[0120] 4. The receiving end receives the binary bit rate and fills the received semantic features in order based on the semantic importance template, and fills the empty positions with zero elements;

[0121] 5. The receiving end inputs the restored semantic features into the semantic decoder to restore the image.

[0122] Based on Figure 1 The same principle as shown in the method, Figure 7 A schematic diagram of the structure of a semantic communication device provided by an embodiment of the present disclosure is shown. Figure 7 As shown, the semantic communication device 70 may include:

[0123] The transmitting end 710 is used to transmit the source data, input the source data into a pre-trained semantic encoder for semantic encoding, and obtain the source semantic features of the source data; filter the source semantic features through the pre-acquired importance template, and convert the filtered source semantic features into a bit stream;

[0124] A communication channel 720 for transmitting a bit stream from a transmitting end to a receiving end;

[0125] The receiving end 730 is used to receive a bit stream, restore the bit stream to restored semantic features through an importance template, and input the restored semantic features into a pre-trained semantic decoder for semantic decoding to obtain restored information;

[0126] Among them, the importance template is obtained by training the target data set; the semantic encoder and the semantic decoder are trained based on the importance template.

[0127] In the semantic communication device provided by the embodiment of the present disclosure, the transmitting end and the receiving end use the same importance template. The transmitting end discards part of the semantically coded content of the transmitting end based on the semantic importance and then transmits it. The receiving end can restore the discarded part based on the importance template, and thus does not need an additional transmission index, which not only reduces the additional overhead of the transmission index, but also enables communication at a lower signal-to-noise ratio.

[0128] In some possible implementations, the transmitter is further configured to: discard some elements of the source semantic features according to a preset ratio using an importance template; and convert the remaining source semantic features into a bit stream.

[0129] In some possible implementations, the receiving end is further used to: restore the bit stream received by the receiving end into feature data; and perform a zero-filling operation on the feature data through an importance template to obtain restored semantic features.

[0130] In some possible implementations, the semantic communication device also includes: a training module for training a semantic communication model using a training data set; the semantic communication model includes a semantic encoder, a semantic importance module, and a semantic decoder connected in sequence; using the trained semantic importance module, the semantic importance of each sample in the target data set is calculated; according to the semantic importance of each sample in the target data set, an importance template is generated; according to the importance template, the semantic communication model is fine-tuned to obtain a trained semantic encoder and semantic decoder.

[0131] In some possible implementations, the training module is also used to: input the training data set into the semantic encoder to obtain the semantic feature vector corresponding to the training data set; input the semantic feature vector into the semantic importance module to obtain the semantic importance of each element in the semantic feature vector; input the semantic feature vector into the semantic decoder to obtain the decoder output; and adjust the parameters of the semantic encoder, the semantic importance module, and the semantic decoder through back propagation according to the decoder output.

[0132] In some possible implementations, the training module is also used to: input the target data set into the semantic encoder to obtain the semantic features of each sample of the target data set; input the semantic features of each sample into the semantic importance module to obtain the semantic importance of each element in the semantic features of each sample.

[0133] In some possible implementations, the training module is further used to: sort the semantic importance of each element, and generate an importance template according to the sorting result.

[0134] In some possible implementations, the training module is also used to: input the training data set into the semantic encoder to obtain the semantic feature vector corresponding to the training data set; discard some elements of the semantic feature vector according to the importance template to obtain the discarded semantic feature vector; input the discarded semantic feature vector into the semantic decoder to obtain the decoder output; and adjust the parameters of the semantic encoder and semantic decoder through back propagation according to the decoder output.

[0135] In some possible implementations, the source data is any one of image data, voice data, text data, and point cloud data.

[0136] It can be understood that the above modules of the semantic communication device in the embodiment of the present disclosure have the function of realizing Figure 1 The functions of the corresponding steps of the semantic communication method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of each module of the above semantic communication device, please refer to Figure 1 The corresponding description of the semantic communication method in the embodiment shown in will not be repeated here.

[0137] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0139] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the semantic communication method provided in the embodiment of the present disclosure.

[0140] Compared with the prior art, the electronic device uses the same importance template at the transmitter and receiver. The transmitter discards part of the semantically coded content of the transmitter based on semantic importance and then retransmits it. The receiver can restore the discarded part based on the importance template, and thus does not need an additional transmission index. This reduces the additional overhead of the transmission index and enables communication at a lower signal-to-noise ratio.

[0141] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the semantic communication method provided in the embodiment of the present disclosure.

[0142] Compared with the prior art, the readable storage medium uses the same importance template at the transmitter and the receiver. The transmitter discards part of the semantically coded content of the transmitter based on semantic importance and then transmits it. The receiver can restore the discarded part based on the importance template, and thus does not need an additional transmission index. This reduces the additional overhead of the transmission index and enables communication at a lower signal-to-noise ratio.

[0143] The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the semantic communication method provided in the embodiment of the present disclosure.

[0144] Compared with the prior art, the computer program product uses the same importance template at the transmitter and the receiver. The transmitter discards part of the semantically coded content of the transmitter based on semantic importance and then retransmits it. The receiver can restore the discarded part based on the importance template, and thus does not need an additional transmission index. This reduces the additional overhead of the transmission index and enables communication at a lower signal-to-noise ratio.

[0145] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0146] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0147] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the semantic communication method. For example, in some embodiments, the semantic communication method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the semantic communication method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the semantic communication method in any other appropriate manner (e.g., by means of firmware).

[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0155] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0156] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A semantic communication method, the method being used in a semantic communication system, the semantic communication system comprising a transmitter, a receiver, and a communication channel connecting the transmitter and the receiver, the method comprising: Inputting the source data transmitted by the transmitting end into a pre-trained semantic encoder for semantic encoding to obtain source semantic features of the source data; The information source semantic features are screened by using a pre-acquired importance template, and the screened information source semantic features are converted into a bit stream; transmitting the bit stream to the receiving end via the communication channel; Restoring the bit stream received by the receiving end into restored semantic features through the importance template, and inputting the restored semantic features into a pre-trained semantic decoder for semantic decoding to obtain recovery information; The importance template is obtained by training a target data set; and the semantic encoder and the semantic decoder are obtained by training based on the importance template.

2. The method according to claim 1, wherein: The method of screening the information source semantic features by using the pre-acquired importance template and converting the screened information source semantic features into a bit stream includes: discarding some elements of the information source semantic features according to a preset ratio using the importance template; The remaining source semantic features are discarded and converted into a bit stream.

3. The method according to claim 1, wherein: The step of restoring the bit stream received by the receiving end to the restored semantic features by using the importance template includes: Restoring the bit stream received by the receiving end into characteristic data; The feature data is padded with zeros using the importance template to obtain the restored semantic feature.

4. The method according to claim 1, further comprising: Training a semantic communication model using a training data set; the semantic communication model includes a semantic encoder, a semantic importance module, and a semantic decoder connected in sequence; Using the trained semantic importance module, calculating the semantic importance of each sample in the target data set; Generating the importance template according to the semantic importance of each sample in the target data set; The semantic communication model is fine-tuned according to the importance template to obtain a trained semantic encoder and semantic decoder.

5. The method according to claim 4, wherein: The step of training the semantic communication model through the training data set includes: Inputting the training data set into the semantic encoder to obtain a semantic feature vector corresponding to the training data set; Inputting the semantic feature vector into the semantic importance module to obtain the semantic importance of each element in the semantic feature vector; Inputting the semantic feature vector into the semantic decoder to obtain a decoder output; The parameters of the semantic encoder, the semantic importance module, and the semantic decoder are adjusted through back propagation according to the decoder output.

6. The method according to claim 4, wherein: The method of using the trained semantic importance module to calculate the semantic importance of each sample in the target data set includes: Inputting the target data set into the semantic encoder to obtain the semantic features of each sample of the target data set; The semantic features of each sample are input into the semantic importance module to obtain the semantic importance of each element in the semantic features of each sample.

7. The method according to claim 6, wherein: The generating the importance template according to the semantic importance of each sample in the target data set includes: The semantic importance of each element is ranked, and the importance template is generated according to the ranking result.

8. The method according to claim 4, wherein: The step of fine-tuning the semantic communication model according to the importance template to obtain a trained semantic encoder and semantic decoder includes: Inputting the training data set into the semantic encoder to obtain a semantic feature vector corresponding to the training data set; Discarding some elements of the semantic feature vector according to the importance template to obtain the semantic feature vector after the discarding; Inputting the discarded semantic feature vector into the semantic decoder to obtain a decoder output; The parameters of the semantic encoder and the semantic decoder are adjusted through back propagation according to the decoder output.

9. The method according to claim 1, wherein: The source data is any one of image data, voice data, text data, and point cloud data.

10. A semantic communication system, comprising: A transmitter, used for transmitting source data, inputting the source data into a pre-trained semantic encoder for semantic encoding, and obtaining source semantic features of the source data; The information source semantic features are screened by using a pre-acquired importance template, and the screened information source semantic features are converted into a bit stream; a communication channel for transmitting the bit stream from the transmitting end to the receiving end; A receiving end, used for receiving the bit stream, restoring the bit stream into restored semantic features through the importance template, and inputting the restored semantic features into a pre-trained semantic decoder for semantic decoding to obtain recovery information; The importance template is obtained by training a target data set; and the semantic encoder and the semantic decoder are obtained by training based on the importance template.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images