Semantic communication system

Abstract

The application provides a semantic communication system, comprising: a semantic encoder configured to extract a semantic feature vector of an input source; a source-channel joint encoder configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector; an elastic feature selector configured to sort the to-be-transmitted semantic feature vector according to importance to obtain a semantic importance-sorted feature vector, and transmit a corresponding number of semantic features with the highest importance after sorting to a source-channel joint decoder; and a semantic decoder configured to receive a decoding result and perform semantic decoding on the decoding result to obtain a classification result. The semantic communication system shown in the embodiment of the application uses one semantic encoder for multiple semantic tasks, uses multiple semantic decoders set to complete the multi-task semantic communication requirement, and sequentially sorts the to-be-transmitted features according to importance, so that important semantic information can be preferentially transmitted, and efficient communication transmission and recovery of semantic features are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic communication, and in particular to a semantic communication system. BACKGROUND

[0002] In the prior art semantic communication technical solution, a new general neural network layer is set to implement the reservation of the first several feature values during transmission, so as to realize the elastic scaling of the feature vector transmission, but at least the feature vector is simply intercepted from the head. In another technical solution, the importance difference of different semantic features is not considered during semantic transmission, and some options transmit all semantic feature vectors, and some options randomly select semantic features for transmission. The above technical solutions do not consider the influence of different features on the implementation of the receiving end task, and do not consider the influence of the advantages and disadvantages of the channel on the semantic communication in the process of semantic communication. In addition, the above technical solutions can only complete one task in one semantic communication process, and cannot complete multiple tasks at the same time. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a semantic communication system.

[0004] To achieve the above purpose, the present application provides a semantic communication system, comprising:

[0005] a semantic encoder, a source-channel joint encoder, an elastic feature selector, a source-channel joint decoder and a semantic decoder;

[0006] The semantic encoder is configured to extract a semantic feature vector of an input source.

[0007] The source-channel joint encoder is configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector.

[0008] The elastic feature selector is configured to sort the to-be-transmitted semantic feature vector according to a preset feature importance measure vector to obtain a semantic importance sorted feature vector, calculate the number of transmittable semantic features according to the obtained channel parameters, and transmit the corresponding number of semantic features with the highest importance in the semantic importance sorted feature vector to the source-channel joint decoder according to the number of transmittable semantic features.

[0009] The semantic decoder is configured to receive the decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result.

[0010] In a possible implementation manner, the preset feature importance measure vector is obtained by the following method:

[0011] The semantic encoder and the semantic decoder are trained, and in response to the training ending, a training classification result output by the semantic decoder at the last training is obtained;

[0012] The parameters of the semantic encoder and the semantic decoder are fixed, the source channel joint encoder and the source channel joint decoder are trained, and in response to the training ending, based on the training classification result, importance of all features of a last layer of the source channel joint encoder is measured to obtain the preset feature importance measurement vector.

[0013] In a possible implementation, the number of the semantic decoders can be multiple;

[0014] Each of the semantic decoders corresponds to a task;

[0015] The semantic decoder is configured to receive a decoding result of the source channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result, including:

[0016] All of the semantic decoders are configured to receive the decoding result of the source channel joint decoder and perform semantic decoding on the decoding result for a task to obtain a classification result of the corresponding task.

[0017] In a possible implementation, the training of the semantic encoder and the semantic decoder includes:

[0018] The semantic encoder is used to perform semantic feature extraction on the obtained training source, and all of the semantic decoders are used to decode the extracted training semantic feature vector to obtain multiple training classification results.

[0019] A loss function is calculated by combining the multiple training classification results and preset real classification results, and the semantic encoder and the semantic decoder are trained by using the loss function.

[0020] In a possible implementation, the loss function is calculated by the following formula:

[0021]

[0022] Wherein, L represents the loss function, i represents the number of tasks, n represents the total number of tasks, λ i represents the weight of the i th task, L i represents the loss function of the i th task.

[0023] In a possible implementation, the importance of all semantic features of the last layer of the source-channel joint encoder is measured based on the training classification result, to obtain the preset feature importance measurement vector, including:

[0024] The partial derivative of the semantic feature vector of the last layer of the source-channel joint encoder is calculated based on the training classification result, to obtain a task sensitivity vector;

[0025] The preset feature importance measurement vector is calculated based on the task sensitivity vector and the importance of the corresponding task.

[0026] In a possible implementation, the preset feature importance measurement vector is calculated based on the task sensitivity vector and the importance of the corresponding task, including:

[0027] In response to the number of tasks being 1, the task sensitivity vector is taken as the preset feature importance measurement vector;

[0028] In response to the number of tasks being greater than 1, the task sensitivity vector is weighted by combining a preset task weight, to calculate the preset feature importance measurement vector.

[0029] In a possible implementation, the semantic feature vector to be transmitted is sorted according to the importance according to the preset feature importance measurement vector, including:

[0030] The numerical value of the preset feature importance measurement vector is calculated;

[0031] The semantic feature vector to be transmitted is sorted according to the numerical value of the corresponding preset feature importance measurement vector from large to small.

[0032] In a possible implementation, the channel parameter includes:

[0033] The transmission bandwidth, transmission time slot, and signal-to-noise ratio of the channel;

[0034] The number of transmittable semantic features is calculated according to the obtained channel parameter, including:

[0035] The number of single-time transmission bits is calculated according to the transmission bandwidth, the transmission time slot, and the signal-to-noise ratio;

[0036] The number of transmittable semantic features is calculated according to the number of single-time transmission bits and the bit length of the semantic feature.

[0037] In a possible implementation, the number of single-time transmission bits is calculated by the following formula:

[0038] B = TW low2 (1 + SNR)

[0039] Wherein, B represents the number of bits of single transmission, T represents the transmission time slot, W represents the transmission bandwidth, and SNR represents the signal-to-noise ratio.

[0040] As can be seen from the above, the semantic communication system provided in the application comprises a semantic encoder, a source-channel joint encoder, an elastic feature selector, a source-channel joint decoder and a semantic decoder. The semantic encoder is configured to extract a semantic feature vector of an input source. The source-channel joint encoder is configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector. The elastic feature selector is configured to sort the to-be-transmitted semantic feature vector according to a preset feature importance measurement vector to obtain a semantic importance sorted feature vector. The number of transmittable semantic features is calculated according to the obtained channel parameters. The corresponding number of semantic features with the highest importance in the semantic importance sorted feature vector after sorting are transmitted to the source-channel joint decoder according to the number of transmittable semantic features. The semantic decoder is configured to receive a decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result. The semantic communication system provided in the application utilizes the auxiliary correlation among multiple intelligent tasks to build a multi-task end-to-end semantic communication system, performs multi-task joint semantic feature compression encoding and decoding, and couples the multi-task features with channel conditions and user semantic requirements. Compared with single-line task encoding and direct transmission, the semantic communication system greatly improves the source compression ratio, reduces the communication processing delay of the transmitting and receiving ends, and realizes efficient communication transmission and recovery of semantic features under the premise of ensuring the task completion performance. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the application or related art, the drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 The semantic communication system structure diagram of the embodiments of the application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the application more clear, the application will be further described in detail below in combination with specific embodiments and with reference to the drawings.

[0044] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall have the common meaning understood by one of ordinary skill in the art to which the embodiments of the present application belong. The terms "first", "second", and similar terms used in the embodiments of the present application do not denote any order, quantity, or importance, but are merely used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean only physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are merely used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships can also change accordingly.

[0045] As described in the background section, the semantic communication technical solution in the related art achieves elastic scaling when transmitting by setting a new general neural network layer to retain the first few feature values, but it simply cuts off the feature vector from the head and does not consider that different features have different effects on the network, i.e., different feature vectors have different importance in the system, resulting in poor flexibility of the existing semantic communication method or system. In addition, the purpose of the general neural network layer is to simplify the neural network structure, and in the semantic communication system, the elastic cutting method of the feature needs to be adapted according to the channel condition, so it is difficult to be directly applied to the semantic communication system.

[0046] In addition, the technical solution of the related art does not consider the importance difference of different semantic features during semantic transmission, and selects to transmit all semantic features, or selects to transmit semantic features randomly, or only transmits features according to their order in the feature vector, which cannot control the transmission code rate according to the communication condition, has low communication efficiency, and cannot select semantic features highly related to the semantic task at the sending end for transmission, and cannot achieve elastic coding highly coupled with the semantic task.

[0047] At the same time, the existing technology can only complete one type of task at a time, and needs to repeat the related steps to achieve multiple tasks, or sets multiple complete semantic communication systems to complete multiple tasks, which has high complexity in the training and use process.

[0048] In view of the above, the embodiment of the present application proposes a semantic communication system, comprising: a semantic encoder, a source-channel joint encoder, an elastic feature selector, a source-channel joint decoder and a semantic decoder; the semantic encoder is configured to extract a semantic feature vector of an input source; the source-channel joint encoder is configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector; the elastic feature selector is configured to sort the to-be-transmitted semantic feature vector according to a preset feature importance metric vector to obtain a semantic importance sorted feature vector; the number of transmittable semantic features is calculated according to the obtained channel parameters; the corresponding number of semantic features with the highest importance in the semantic importance sorted feature vector after sorting are transmitted to the source-channel joint decoder according to the number of transmittable semantic features; and the semantic decoder is configured to receive the decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result.

[0049] The embodiment of the present application builds a multi-task end-to-end semantic communication system by utilizing the auxiliary correlation between multiple intelligent tasks, performs multi-task joint semantic feature compression encoding and decoding, and couples the multi-task features with channel conditions and user semantic requirements. Compared with direct transmission after single-line task encoding, the compression ratio of the source is greatly improved, the processing delay of the communication between the transmitter and the receiver is reduced, and efficient communication transmission and recovery of semantic features are realized under the premise of ensuring the task completion performance.

[0050] In addition, the embodiment of the present application proposes an elastic encoding scheme based on channel conditions, which sequentially sorts the to-be-transmitted features according to importance to ensure that important semantic information is transmitted preferentially. Further, under the premise of maintaining high accuracy of multi-task completion, the efficiency and scalability of semantic information transmission under variable channel conditions and variable user requirements are improved.

[0051] Meanwhile, the embodiment of the present application designs an elastic mapping mechanism of channel state information to the number of transmission features guided by channel capacity, uses multi-task completion degree as an index, and transmits semantic features according to the elastic mechanism of transmission dimension, so that the number of transmission features can be flexibly changed with the channel conditions, and the communication resources are maximally utilized under a certain transmission bandwidth and delay.

[0052] In the following, the technical solutions of the embodiments of the present application will be described in detail through specific embodiments.

[0053] Reference Figure 1 is a structural schematic diagram of the semantic communication system of the embodiment of the present application.

[0054] As Figure 1 shown, the semantic communication system in the embodiment of the present application comprises: a semantic encoder, a source-channel joint encoder, an elastic feature selector, a source-channel joint decoder and a semantic decoder;

[0055] the semantic encoder is configured to extract a semantic feature vector of the input source;

[0056] the source-channel joint encoder is configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector;

[0057] the elastic feature selector is configured to sort the to-be-transmitted semantic feature vector according to a preset feature importance measurement vector to obtain a semantic importance sorted feature vector, calculate a number of transmittable semantic features according to the obtained channel parameter, and transmit a corresponding number of semantic features with the highest importance in the semantic importance sorted feature vector to the source-channel joint decoder according to the number of transmittable semantic features.

[0058] the semantic decoder is configured to receive a decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result.

[0059] The overall process of the embodiment of the application includes five steps: building and training an end-to-end multi-task semantic communication system, generating a semantic feature importance measurement value using the semantic communication system model, coupling the importance measurement value with to-be-transmitted features, adaptively mapping the number of transmitted features according to channel conditions, and selecting high-importance features for transmission.

[0060] The system first builds a multi-task semantic communication system composed of a semantic feature encoder, a source-channel joint encoder, a channel, a source-channel joint decoder, and a semantic feature decoder. Secondly, the gradient of the to-be-transmitted features relative to the classification recognition result of the model output is calculated in reverse to obtain the sensitivity of different features to the intelligent task result, which represents the importance of different features to the intelligent task. Finally, the adaptive number of transmitted features under the guidance of the importance value and the channel capacity is selected to preferentially transmit important features.

[0061] Specifically, a semantic communication system is first constructed. In the embodiment of the application, the number of semantic decoders is taken as an example, but it should be noted that the application can be applied to a single-task scenario mode, and correspondingly, the number of semantic decoders is set to one in this scenario.

[0062] The sending end of the semantic communication system is composed of a semantic encoder, a source-channel joint encoder, and an elastic feature selector, and the receiving end is composed of a source-channel joint decoder and a plurality of task corresponding semantic decoders.

[0063] In addition, the input source is denoted as x, the semantic feature vector to be transmitted obtained after the semantic encoder and the source-channel joint encoder is denoted as f, the sorted semantic feature vector output after the elastic feature selector is denoted as z, and the semantic feature vector received after the channel is denoted as The semantic feature vector decoded after the source-channel joint decoder is denoted as The semantic feature vector recovered after the multiple semantic decoders corresponding to the multiple tasks is denoted as y1, y2, y3, …, and the above feature constitutes a probabilistic graphical model:

[0064]

[0065] The above probabilistic graphical model satisfies the following formula:

[0066]

[0067] where y n represents the semantic feature vector corresponding to a certain task, represents the deep neural network parameters of the semantic encoder, represents the neural network parameters of the source-channel joint encoder, represents the deep neural network parameters of the semantic decoder, represents the neural network parameters of the source-channel joint decoder, and h represents the channel transition parameter.

[0068] The above probabilistic graphical model is only for showing the overall processing flow of the semantic communication system of the embodiments of the present application.

[0069] Further, after the semantic communication system is constructed, the semantic communication system of the embodiments of the present application needs to be trained.

[0070] Specifically, first, the semantic encoder and the semantic decoder are trained separately. In the embodiments of the present application, three common semantic intelligent tasks in the intelligent transportation system are taken as examples, which include vehicle identity re-identification, vehicle color classification, and vehicle type classification.

[0071] In the embodiments of the present application, when the semantic encoder and the semantic decoder are trained, the large-scale benchmark dataset of vehicle Re-Id of “VeRi” established in the real-world city monitoring scene is used for training. It contains more than 50,000 images of 776 vehicles, which are captured by 20 cameras and cover an area of 1.0 square kilometers within 24 hours, which makes the dataset sufficient for vehicle Re-Id and other related researches. The images are captured in a real-world unrestricted monitoring scene and are labeled with different attributes, such as type, color, and brand. Therefore, the three semantic intelligent tasks in the present application can be completed simultaneously using this dataset.

[0072] Further, in training the semantic encoder and decoder, semantic features are extracted from the acquired training source using the semantic encoder, and the extracted semantic feature vectors are decoded using all the semantic decoders to obtain a plurality of training classification results, and a loss function is calculated based on the plurality of training classification results and preset true classification results; the semantic encoder and the semantic decoder are trained using the loss function.

[0073] Specifically, in the embodiment of the present application, DenseNet121 deep learning network is used for semantic feature extraction, and three semantic decoders composed of full connection layers and excitation functions are used to obtain restored semantic feature vectors, and finally three classifiers are used to obtain the prediction results of the tasks.

[0074] Further, the loss function is calculated by the following formula:

[0075]

[0076] Wherein, L represents the loss function, i represents the number of tasks, L i represents the loss function of the i-th task, λ i represents the weight of the i-th task, and n represents the total number of tasks.

[0077] In this embodiment, the loss function of this training process is:

[0078] L = λ ID L ID + λ color L color + λ type L type

[0079] Wherein, L ID represents the loss function of vehicle identity re-identification, L color represents the loss function of vehicle color classification, L type represents the loss function of vehicle type classification, λ ID represents the weight of the vehicle re-identification task, λ color represents the weight of the vehicle color classification task, and λ type represents the weight of the vehicle type classification task.

[0080] In the embodiment of the present application, the weight of the vehicle identity re-identification task is set to 0.75, the weight of the vehicle color classification task is set to 0.125, and the weight of the vehicle type classification task is set to 0.125. Those skilled in the art should know that the above task weights can be freely set according to their own needs, and the number of tasks is not limited to three.

[0081] In addition, in the embodiments of the present application, the learning mode is set to supervised learning, and the loss function of the vehicle identity re-identification mentioned above is set as a weighted sum of a hard-mining triplet loss and a cross-entropy loss:

[0082] L ID = L ht (a, p, n) + L ce (y, y')

[0083] wherein L ID represents the loss function of the vehicle identity re-identification, L ht (a, p, n) represents the triplet loss, L ce (y, y') represents the cross-entropy loss, a represents the reference sample, p represents the positive sample for the reference, and n represents the negative sample for the reference. L ht (a, p, n) is calculated by the following formula:

[0084] L ht (a, p, n) = max{ (max(d ap ) - min(d an ) + a), 0}

[0085] wherein d ap represents the distance between the reference sample and the positive sample, and d an represents the distance between the reference sample and the negative sample. The cross-entropy loss function is calculated by the following formula:

[0086]

[0087] wherein y represents the task real label, y' represents the classification prediction value, and K represents the total number of task categories. The loss function of the vehicle color classification and the loss function of the vehicle type classification are also set as the cross-entropy loss between the classification result estimated by the semantic decoder and the real classification label.

[0088] Further, the parameters of the semantic encoder are trained using the above loss function. It should be noted that the technical solution of using the cross-entropy loss as the loss function for training the semantic encoder can be adaptively changed according to actual needs to adapt to the corresponding task. The training of the semantic encoder is completed when the loss function of the semantic encoder reaches convergence. It should be noted that the training of the semantic encoder does not involve other parts of the semantic communication system. In response to the end of the training, the classification result output by the semantic decoder during the last training is obtained.

[0089] Further, the trained parameters of the semantic encoder-decoder are frozen, and then the whole system is trained, i.e., the source-channel joint encoder and the source-channel joint decoder and the channel parameters are trained. The training process is known to those skilled in the art, and thus is not described here.

[0090] Further, when the training of the source-channel joint encoder and the source-channel joint decoder is completed, the trained parameters of the semantic encoder-decoder are unfrozen, and the whole semantic communication system is further trained and the parameters are adjusted. When the overall training is completed, the gradient of all features in the last layer of the source-channel encoder is derived with respect to the probability of the category in the fully connected layer of the semantic decoder, i.e., the classification result used in the training, which can reflect the sensitivity of the output probability to the feature. If the gradient is large, the feature is very sensitive, indicating that the feature is more likely to be predicted by the classification network. The task sensitivity vector is obtained. The task sensitivity vector includes C elements, i.e., the number of feature vectors.

[0091] Further, based on the task sensitivity vector and the importance of the corresponding task, the preset feature importance measure vector is calculated. The importance of the corresponding task is the task weight described above. Each task sensitivity vector is weighted and calculated according to the task weight to obtain the preset feature importance measure vector. It should be noted that if the number of tasks is 1, the task sensitivity vector can be directly used as the preset feature importance measure vector.

[0092] Further, the channel parameters are changed with the preset feature importance measure vector as a constant value, and the parameters of the whole semantic communication system are adjusted based on different channel parameter values.

[0093] Specifically, in the embodiments of the present application, the channel parameters include the transmission bandwidth, the transmission time slot, and the signal-to-noise ratio of the channel. According to the transmission bandwidth, the transmission time slot, and the signal-to-noise ratio, the number of single transmission bits is calculated. According to the number of single transmission bits and the bit length of the semantic feature to be transmitted, the number of transmittable semantic feature vectors is calculated. The above steps are performed by the elastic feature selector.

[0094] Specifically, the number of single transmission bits is calculated by the following formula:

[0095] B = TW log2(1 + SNR)

[0096] where B represents the number of single transmission bits, T represents the transmission time slot, W represents the transmission bandwidth, and SNR represents the signal-to-noise ratio.

[0097] In the subsequent training process, the channel parameters of the semantic communication system are changed so that the semantic communication system of the present application can adapt to almost all channel conditions.

[0098] When the training of the semantic communication system is completed, in the prediction process, the semantic feature vector of the input source is extracted by the semantic encoder, and further, the semantic feature vector is received by the source-channel joint encoder, and the semantic feature vector is encoded to obtain the to-be-transmitted semantic feature vector.

[0099] Further, the to-be-transmitted semantic feature vector is input to the elastic feature selector according to the preset feature importance measurement vector, which has been described in the foregoing training process and will not be repeated here. In the elastic feature selector, the to-be-transmitted semantic feature vector is sorted according to the importance, specifically, the values of the preset feature importance measurement vector are calculated, and the semantic features in the corresponding positions of the to-be-transmitted semantic feature vector are sorted according to the importance from large to small according to the values of the preset feature importance measurement vector. Further, the channel parameters are obtained, and the number of transmittable semantic features is calculated according to the obtained channel parameters. The calculation steps of the number of transmittable semantic features have been described in the foregoing technical solutions and will not be repeated here. Further, according to the number of transmittable semantic features calculated in the foregoing steps, the corresponding number of semantic features with the highest importance in the sorted semantic importance sorting feature vector are transmitted to the source-channel joint decoder.

[0100] Further, the received semantic features are decoded in the source-channel joint decoder, and the decoding process is known to those skilled in the art and will not be repeated here.

[0101] Further, the decoding result of the source-channel joint decoder is received by the semantic decoder, and the decoding result is decoded semantically to obtain a classification result. It should be noted that the number of semantic decoders in the present application is set according to the number of tasks, and each task corresponds to a semantic decoder, so each semantic decoder will decode the decoding result of the foregoing source-channel joint decoder for the task to obtain the classification result of the corresponding task. The process of semantic decoding of each semantic decoder for a single task is known to those skilled in the art and will not be repeated here.

[0102] As can be seen from the above embodiments, the semantic communication system provided in the embodiments of the present application comprises a semantic encoder, a source-channel joint encoder, an elastic feature selector, a source-channel joint decoder and a semantic decoder; the semantic encoder is configured to extract a semantic feature vector of an input source; the source-channel joint encoder is configured to encode the semantic feature vector to obtain a to-be-transmitted semantic feature vector; the elastic feature selector is configured to sort the to-be-transmitted semantic feature vector according to a preset feature importance measurement vector to obtain a semantic importance sorted feature vector; the number of transmittable semantic features is calculated according to the obtained channel parameters; the corresponding number of semantic features with the highest importance in the semantic importance sorted feature vector after sorting are transmitted to the source-channel joint decoder according to the number of transmittable semantic features; and the semantic decoder is configured to receive a decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result.

[0103] By utilizing the auxiliary correlation between multiple intelligent tasks, the embodiments of the present application build a multi-task end-to-end semantic communication system, perform multi-task joint semantic feature compression encoding and decoding, and couple the multi-task features with channel conditions and user semantic requirements. Compared with direct transmission after single-line task encoding, the embodiments of the present application greatly improve the source compression ratio, reduce the transmission and reception end communication processing delay, and realize efficient communication transmission and recovery of semantic features under the premise of ensuring task completion performance.

[0104] In addition, the embodiments of the present application propose an elastic encoding scheme based on channel conditions, sequentially sort the to-be-transmitted features according to importance, couple the semantic information extraction process with the semantic task completion index, ensure that important semantic information can be transmitted preferentially, and improve the system performance under limited communication resources. Further, the embodiments of the present application improve the efficiency and scalability of semantic information transmission under variable channel conditions and variable user requirements under the premise of maintaining high accuracy of multi-task completion.

[0105] Meanwhile, the embodiments of the present application design an elastic mapping mechanism of channel state information to the number of transmission features guided by channel capacity, take the multi-task completion degree as an index, perform transmission of semantic features according to the elastic mechanism of transmission dimension, so that the number of transmission features can be elastically changed with the channel conditions, and the communication resources are maximally utilized under a certain transmission bandwidth and transmission time slot.

[0106] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server. The method of the embodiments of the present application can also be applied to a distributed scenario and completed by multiple devices in cooperation. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices will interact with each other to complete the method.

[0107] It is noted that some embodiments have been described above. Other embodiments are within the scope of the following claims. In some cases, an act or step can be performed in a different order from the described implementation or implementation can be performed concurrently or in different order. In addition, the process depicted in the accompanying drawings does not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0108] It will be apparent to one of ordinary skill in the art that any of the above-described embodiments can be combined with other embodiments or aspects of the application, or techniques thereof, in accordance with the teachings of the present application. In addition, it will be apparent to one of ordinary skill in the art that steps, or aspects of steps, of the disclosed processes can be performed in any order without affecting the overall results. Also, many of the embodiments described herein are amenable to parallel, or concurrent, processing and / or implementation. Moreover, those of ordinary skill in the art will recognize that the foregoing description and examples have been presented for the purposes of illustration and description only. They are not intended to be exhaustive or to limit the application to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. Any and all modifications and variations are intended to be included within the scope of the present application.

[0109] In addition, for the purpose of simplicity and clarity, the power / ground connections of the integrated circuit (IC) chips and other components can or can not be shown in the provided figures. Further, aspects of a device can be shown in block diagram form in order to avoid obscuring the concepts of the present application, and this is done to the extent that one of ordinary skill in the art, with the benefit of this disclosure, would understand the function of the illustrated device. Finally, those of ordinary skill in the art will recognize that the foregoing description and examples have been presented for the purposes of illustration and description only. They are not intended to be exhaustive or to limit the application to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. Any and all modifications and variations are intended to be included within the scope of the present application.

[0110] While the present application has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0111] It is intended that the present application encompass all such variations and modifications as fall within the scope of the appended claims and their equivalents. Accordingly, any one of the steps of the disclosed processes can be performed in any order or concurrently, and the order of steps in the processes should not be construed as a limitation of the present application.

Claims

1. A semantic communication system, characterized in that, include: Semantic encoder, source-channel joint encoder, flexible feature selector, source-channel joint decoder, and semantic decoder; The semantic encoder is used to extract the semantic feature vector of the input source; The source-channel joint encoder is used to encode the semantic feature vector to obtain the semantic feature vector to be transmitted; The elastic feature selector is used to sort the semantic feature vectors to be transmitted according to their importance based on a preset feature importance metric vector, thereby obtaining a semantic importance ranked feature vector; and to calculate the number of transmittable semantic features based on the acquired channel parameters. Based on the number of transmitable semantic features, the number of semantic features with the highest importance in the sorted semantic importance feature vector is transmitted to the source-channel joint decoder. The semantic decoder is used to receive the decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result; The preset feature importance metric vector is obtained using the following method: The semantic encoder and the semantic decoder are trained, and in response to the end of the training, the training classification result output by the semantic decoder at the last training time is obtained; With the parameters of the semantic encoder and the semantic decoder fixed, the source-channel joint encoder and the source-channel joint decoder are trained. Upon completion of training, based on the classification results used in the training, the importance of all semantic features in the last layer of the source-channel joint encoder is measured to obtain the preset feature importance measurement vector, including: Based on the classification results used for training, the partial derivative of the semantic feature vector of the last layer of the source-channel joint encoder is calculated to obtain the task sensitivity vector. Based on the task sensitivity vector and the importance of the corresponding task, the preset feature importance metric vector is calculated.

2. The system according to claim 1, characterized in that, The number of semantic decoders can be multiple; Each of the semantic decoders corresponds to one task; The semantic decoder is used to receive the decoding result of the source-channel joint decoder and perform semantic decoding on the decoding result to obtain a classification result, including: All of the semantic decoders are used to receive the decoding results of the source-channel joint decoder, and perform semantic decoding on the decoding results for the task to obtain the classification results of the corresponding task.

3. The system according to claim 2, characterized in that, The training of the semantic encoder and the semantic decoder includes: The semantic encoder is used to extract semantic features from the acquired training source, and all the semantic decoders are used to decode the extracted training semantic feature vectors to obtain multiple training classification results. A loss function is calculated by combining the multiple training classification results and the preset real classification results; the semantic encoder and the semantic decoder are trained using the loss function.

4. The system according to claim 3, characterized in that, The loss function is calculated using the following formula: Where L represents the loss function, i represents the task number, n represents the total number of tasks, and λ i L represents the weight of the i-th task. i Let represent the loss function for the i-th task.

5. The system according to claim 1, characterized in that, The step of calculating the preset feature importance metric vector based on the task sensitivity vector and the importance of the corresponding task includes: In response to the fact that the number of tasks is 1, the task sensitivity vector is used as the preset feature importance metric vector; In response to the number of tasks being greater than 1, the task sensitivity vector is weighted in combination with the preset task weights to calculate the preset feature importance metric vector.

6. The system according to claim 1, characterized in that, The step of sorting the semantic feature vectors to be transmitted according to their importance based on a preset feature importance metric vector includes: Calculate the value of the preset feature importance metric vector; The semantic feature vectors to be transmitted are sorted from largest to smallest according to the values ​​of the corresponding preset feature importance metric vectors.

7. The system according to claim 1, characterized in that, The channel parameters include: The channel's transmission bandwidth, transmission time slots, and signal-to-noise ratio; The step of calculating the number of transmittable semantic features based on the acquired channel parameters includes: The number of bits transmitted in a single transmission is calculated based on the transmission bandwidth, the transmission time slot, and the signal-to-noise ratio. The number of transmittable semantic features is calculated based on the number of bits transmitted in a single transmission and the bit length of the semantic feature.

8. The system according to claim 7, characterized in that, The number of bits transmitted in a single transmission is calculated using the following formula: B = TW log2(1 + SNR) Where B represents the number of bits transmitted in a single transmission, T represents the transmission time slot, W represents the transmission bandwidth, and SNR represents the signal-to-noise ratio.

Images