Multitask-oriented semantic communication method and device, equipment and storage medium

Abstract

The invention discloses a multi-task-oriented semantic communication method and device, equipment and a storage medium. The method comprises the following steps: performing semantic information representation, information coding and information conversion processing on source information based on a semantic communication sending end in a mutual information maximization semantic communication basic framework; the semantic communication receiving end carries out semantic information recovery, information decoding and task reasoning processing on the received semantic information; the semantic representation quantification model carries out preset precision quantification on semantic information representation at a semantic communication sending end, a semantic symbol obtained after quantification is expressed by adopting a bit number, and the semantic symbol obtained after quantification is mapped to a constellation point; and the adaptive variable-length semantic representation coding model selectively performs sequential and continuous activation on specified dimensions represented by semantic information based on a current channel state and a target task demand condition. According to the method, on the premise of ensuring the multi-task performance, the transmission efficiency of semantic information is improved, and the communication and calculation overhead is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a multi-task oriented semantic communication method, device, equipment and storage medium. Background Art

[0002] Task-oriented semantic communication is an intelligent communication method that completes task requirements by transmitting semantic information. Traditional grammatical communication requires that the decoded information at the receiving end is strictly consistent with the encoded information at the sending end, that is, to achieve bit-level error-free transmission. However, task-oriented semantic communication does not require the encoding and decoding sequences of the sending and receiving ends to match strictly, but only requires that the receiving end information can complete the corresponding task as required after processing.

[0003] Currently, task-oriented semantic communication research is gradually becoming a hot topic in the academic community. Data transmission based on semantic information will be a very competitive key technology. The task-oriented semantic communication system is designed so that the signal transmitted by the sender has the semantic information required by multiple tasks at the same time, and then the needs of multiple tasks are met at the receiving end. How to improve the transmission efficiency of semantic information and reduce communication and computing overhead while ensuring multi-task performance has practical application significance. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a multi-task oriented semantic communication method, apparatus, device and storage medium, so as to optimize the end-to-end semantic communication system by maximizing the mutual information between source information and semantic representation, so as to transmit semantic representations that are both generative and discriminative in the network, and provide the required semantic information for multiple target tasks at the receiving end.

[0005] Based on the above objectives, in a first aspect, the present invention provides a multi-task oriented semantic communication method, comprising:

[0006] Constructing a semantic communication basic framework based on mutual information maximization, the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end, the semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on received semantic information;

[0007] Constructing a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point;

[0008] An adaptive variable-length semantic representation coding model is constructed. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

[0009] As a preferred solution of the multi-task oriented semantic communication method, the semantic communication sending end uses a convolutional neural network and a deep neural network to represent the semantic information of the source information;

[0010] The semantic communication sending end performs information encoding on the source information including semantic encoding, source encoding and channel encoding.

[0011] As a preferred solution for a multi-task semantic communication method, the semantic representation quantization model utilizes an artificial intelligence model and maximizes global and local mutual information, so that the semantic information representation is both generative and discriminative.

[0012] As a preferred solution of the multi-task oriented semantic communication method, the semantic communication receiving end uses an artificial intelligence model to reconstruct the received semantic information and restore the source information during the semantic information recovery process of the received semantic information;

[0013] The semantic communication receiving end performs information decoding on the received semantic information, including semantic decoding, information source decoding and channel decoding.

[0014] As a preferred solution for the multi-task oriented semantic communication method, in the process of mapping the semantic symbols obtained after quantization to constellation points, an asymmetric quantization algorithm is used to map the quantized semantic symbols to constellation points with amplitude and phase intervals to adapt to the RF hardware design in actual digital communication systems.

[0015] As a preferred solution of the multi-task oriented semantic communication method, the current channel state based on the adaptive variable length semantic representation coding model includes the signal-to-noise ratio and noise variance at the current moment;

[0016] The target task requirement conditions on which the adaptive variable-length semantic representation coding model is based include a semantic representation pruning threshold and a target task performance threshold.

[0017] In a second aspect, the present invention provides a multi-task oriented semantic communication device, comprising:

[0018] A semantic communication basic processing module is used to construct a semantic communication basic framework based on mutual information maximization, wherein the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end. The semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on received semantic information;

[0019] A semantic representation quantization module, used to construct a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point;

[0020] An adaptive variable-length semantic representation coding module is used to construct an adaptive variable-length semantic representation coding model. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

[0021] As a preferred solution of a multi-task oriented semantic communication device, in the semantic communication basic processing module:

[0022] The semantic communication sending end uses a convolutional neural network and a deep neural network to represent the semantic information of the source information;

[0023] The semantic communication sending end performs information encoding on the source information, including semantic encoding, source encoding and channel encoding;

[0024] Reconstructing the received semantic information and restoring the source information by using an artificial intelligence model;

[0025] The semantic communication receiving end performs information decoding on the received semantic information, including semantic decoding, information source decoding and channel decoding.

[0026] As a preferred solution of a multi-task oriented semantic communication device, in the semantic representation quantization module:

[0027] The semantic representation quantization model utilizes an artificial intelligence model and maximizes global and local mutual information to make the semantic information representation both generative and discriminative;

[0028] An asymmetric quantization algorithm is used to map the quantized semantic symbols to constellation points with amplitude and phase intervals to adapt to the RF hardware design in actual digital communication systems.

[0029] As a preferred solution of a multi-task oriented semantic communication device, in the adaptive variable-length semantic representation encoding module:

[0030] The current channel state based on the adaptive variable length semantic representation coding model includes the signal-to-noise ratio and noise variance at the current moment;

[0031] The target task requirement conditions on which the adaptive variable-length semantic representation coding model is based include a semantic representation pruning threshold and a target task performance threshold.

[0032] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-task oriented semantic communication method according to the first aspect or any possible implementation thereof is implemented.

[0033] In a fourth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause the computer to execute a multi-task oriented semantic communication method of the first aspect or any possible implementation thereof.

[0034] From the above, it can be seen that the technical solution provided by the present invention constructs a semantic communication basic framework based on mutual information maximization, and the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end, and the semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on the source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on the received semantic information; constructs a semantic representation quantization model, and the semantic representation quantization model performs preset precision quantization on the semantic information representation at the semantic communication sending end, and the semantic symbol obtained after quantization is represented by the number of bits, and the semantic symbol obtained after quantization is mapped to the constellation point; constructs an adaptive variable-length semantic representation coding model, and the adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead under the premise of ensuring the target task performance, and the target task includes at least one of classification, recognition, reconstruction, clustering, and retrieval. The present invention can complete multiple tasks simultaneously through a semantic information extraction, thereby improving the transmission efficiency of semantic information and reducing communication and computing overhead under the premise of ensuring multi-task performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic flow chart of a multi-task oriented semantic communication method provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a simultaneous multi-task joint source channel encoding and decoding framework based on DNN provided in an embodiment of the present invention;

[0038] Figure 3 A joint source-channel coding and decoding framework training process based on mutual information maximization provided in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of an adaptive variable-length semantic representation coding model provided by an embodiment of the present invention;

[0040] Figure 5 An architecture diagram of a multi-task semantic communication device provided by an embodiment of the present invention;

[0041] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0043] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The words "include" or "comprise" and the like used in the embodiments of the present invention mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects.

[0044] In the related technology, task-oriented semantic communication is an intelligent communication method that completes task requirements by transmitting semantic information. Traditional grammatical communication requires that the decoded information at the receiving end is strictly consistent with the encoded information at the sending end, that is, to achieve bit-level error-free transmission. Task-oriented semantic communication does not require strict matching of the encoding and decoding sequences at the sending and receiving ends, but only requires that the information at the receiving end be able to complete the corresponding task on demand after processing. Since task-oriented semantic communication can further compress the transmission content, it redefined the error requirements for information transmission in the system, increased the fault tolerance of information transmission, and thus broke through the transmission bottleneck of the classic grammatical communication system, further improved the transmission efficiency, and provided a new solution for the evolution of the next generation of wireless communication systems.

[0045] At present, task-oriented semantic communication research is gradually becoming a hot topic of people's attention. Data transmission based on semantic information will be a very competitive key technology. Further design of task-oriented semantic communication system can make the signal transmitted by the sender have semantic information required by multiple tasks at the same time, and then meet the needs of multiple tasks at the receiving end, so that the semantic communication system has richer functions and higher transmission efficiency.

[0046] In the related art, Mutual Information Neural Estimation (MINE) is a newly proposed method that uses neural networks to assist semi-supervised or unsupervised representation learning. Using traditional methods to calculate the mutual information of high-dimensional features is a very difficult task because it involves complex probability distributions and high computational costs. However, MINE overcomes this problem by leveraging the powerful representation ability of neural networks to effectively estimate the mutual information between high-dimensional features. Specifically, MINE captures the potential valuable information representation in the data by maximizing the lower bound of the mutual information estimated by the neural network. This method can not only improve the estimation accuracy of mutual information, but also estimate the mutual information of high-dimensional data, making semi-supervised and unsupervised learning on complex data sets more feasible and efficient. In short, MINE provides an effective tool for estimating the mutual information between high-dimensional features, which greatly promotes the research and application of related fields such as representation learning. By integrating mutual information estimation into the design of a multi-task semantic communication framework, the sender can actively extract useful information from the source, rather than just passively generating intermediate features through label matching or pixel restoration. The semantic representation extracted by this method can simultaneously promote the performance of generation tasks and discrimination tasks, thereby more effectively handling various complex task requirements.

[0047] In view of this, the embodiments of the present invention propose a multi-task oriented semantic communication method, apparatus, device and storage medium, which utilizes the global and local mutual information maximization to make the semantic representation extracted by the sender both generative and discriminative, so that it can be directly used to complete the generative task and the discriminative task; in addition, by minimizing the mean square error end-to-end training of the sender and the receiver, the receiver can complete the reconstruction of the original content according to the received semantic representation, and can complete multiple tasks simultaneously through a single semantic information extraction, thereby improving the transmission efficiency of semantic information and reducing communication and computing overheads while ensuring multi-task performance. The following is the specific content of the embodiments of the present invention.

[0048] See also Figure 1 , Figure 2 , Figure 3 and Figure 4 The embodiment of the present invention provides a multi-task oriented semantic communication method, comprising the following steps:

[0049] S1. Construct a semantic communication basic framework based on mutual information maximization, wherein the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end. The semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on the source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on the received semantic information;

[0050] S2. Constructing a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point;

[0051] S3. Construct an adaptive variable-length semantic representation coding model. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

[0052] In this embodiment, the end-to-end semantic communication system is optimized by maximizing the mutual information between the source information and the semantic representation, so that the semantic representation that is both generative and discriminative is transmitted in the network, and the required semantic information is provided for multiple target tasks at the receiving end. Among them, the network includes wireless networks, satellite networks, Wi-Fi networks, etc. The source information includes text, voice, pictures, videos, etc. The target tasks include classification, recognition, reconstruction, clustering, retrieval, etc. The semantic representation includes the intermediate parameters output by various artificial intelligence models, and the artificial intelligence models may include convolutional neural networks, deep neural networks, etc.

[0053] In this embodiment, a semantic communication basic framework of a joint source channel is constructed based on mutual information maximization, so that the extracted semantic representation has both generative and discriminative properties; based on an asymmetric quantization method, the quantized semantic symbols can be mapped to fewer constellation points with clearer amplitude and phase intervals; based on an adaptive semantic representation encoding method of a multi-layer perceptron, the semantic representation adapts to different channel states and activates different dimensions in sequence and continuously. The present invention is applicable to a variety of scenarios including single-user transmission and multi-user transmission, and the semantic communication basic framework based on mutual information maximization can be implemented based on a variety of infrastructures such as deep neural networks (DNNs), convolutional neural networks (CNNs), and attention mechanisms.

[0054] For details, see Figure 2 In this embodiment, DNN is used to illustrate the scheme. The semantic communication basic framework based on mutual information maximization can be applied to various tasks such as classification tasks, retrieval tasks, and reconstruction tasks. In this embodiment, image classification tasks and image reconstruction tasks are used to illustrate the scheme.

[0055] At the transmitter, the joint source-channel coding network E a (·) Source information Mapped to a 2n-dimensional semantic representation vector z, expressed as follows:

[0056]

[0057] Then, the semantic representation vector z is transformed into n complex symbols as the modulation signal And perform power normalization as follows:

[0058]

[0059] Different from traditional information compression, semantic representation contains not only the semantic information of the source information, but also the compensation information to combat the randomness of the wireless channel. The transformed semantic symbol has a continuously changing amplitude and phase, which can be demodulated by the joint source-channel decoding network based on deep learning, or the semantic representation can be quantized before modulation.

[0060] Since the receiving end has orthogonal time-frequency resource blocks, the impact of inter-signal interference on transmission can be ignored, and the received signal y is obtained as follows:

[0061] y=hx+n

[0062] Among them, h is the channel parameter, n represents the independent and identically distributed channel noise vector, which follows the mean of 0 and the variance of δ 2 Symmetric complex Gaussian distribution By using channel estimation and channel equalization technology, the receiver can obtain the channel parameter h and then restore the received signal as follows:

[0063]

[0064] Therefore, it can be extended to Gaussian, Rayleigh and Rice channels. The receiving end uses the same transformation strategy as the transmitting end to transform the noisy received signal Transform back to noisy semantic representation It serves as the input of the joint source-channel decoding network at the receiving end to simultaneously complete image reconstruction tasks and image classification.

[0065] For the image reconstruction task, the receiver combines the source and channel decoding network D β (·) Using noisy semantic representation Get the reconstructed image as follows:

[0066]

[0067] Among them, the joint source channel coding network E at the transmitter α (·) and the joint source channel decoding network D at the receiving end β (·) is jointly trained end-to-end. The performance of the image reconstruction task is measured by the peak signal-to-noise ratio (PSNR), which represents the visual difference between two images:

[0068]

[0069] Among them, MAX represents the maximum value of all pixels in the image, and MSE(,) represents the mean square error operation.

[0070] For the image classification task, the noisy semantic representation is input into a pre-designed classifier ψ(·). As a function, it can be designed according to the specific task of the receiver and can be implemented in different forms, including DNN model, K-nearest neighbor classifier, support vector machine, etc. Since the received semantic representation is discriminative and can be directly used for classification tasks, the classifier ψ(·) does not need to be combined with the joint source channel coding network E at the transmitter. α (·) End-to-end joint training. The performance of the image classification task is measured by accuracy (ACC), which represents the ratio of correctly classified samples to the total number of test samples:

[0071]

[0072] In the traditional single-task semantic communication framework, the above two tasks cannot be completed simultaneously. This is because the task-specific joint source-channel codec model is trained in an end-to-end manner with a task-specific loss function, which aims to make the output closer to the expected value of the task by adjusting the model parameters. The goals of different tasks usually lead to different optimization directions, so it is difficult to achieve multiple goals at the same time. In addition, in the traditional end-to-end training paradigm, the output results are only actively optimized according to the task goals, while the semantic representation of the intermediate layer output is obtained by unsupervised passive learning. Therefore, the transmitted semantic representation is only an intermediate result of the specific target task and is not discriminative or interpretable.

[0073] To this end, the joint source-channel encoding and decoding framework based on mutual information maximization in the present invention aims to simultaneously optimize the semantic representation and task results, thereby realizing the extraction of discriminative semantic information while improving the generation capabilities of the semantic representation and the joint source-channel decoder.

[0074] In this embodiment, at the transmitting end, the global mutual information is first maximized by maximizing the Jensen-Shannon (JS) divergence between the source image s and the semantic representation z. The JS divergence is defined as follows:

[0075]

[0076] Among them, T θ (·) represents the discriminator, which is composed of a neural network with parameters θ, and s′ represents a pseudo image sample. The JS divergence is maximized in an adversarial manner. The goal of the discriminator is to maximize the discrimination score T of the real sample. θ (s, z) while minimizing the discrimination score T of pseudo samples θ (s′, z), thus, the trained discriminator can distinguish whether the semantic representation z comes from the real source image, thereby maximizing the correlation between the semantic representation and the original sample (through the joint distribution term) and improving the independence between the semantic representation and irrelevant samples (through the marginal distribution term).

[0077] like Figure 3 As shown in Figure 1, by focusing on different parts of the image (global or local), the trained semantic representation can contain information for different tasks. Among them, the global mutual information takes the entire image as the receptive field and maximizes the JS divergence between the source image s and the semantic representation z by optimizing the joint source channel coding network and the global discriminator:

[0078]

[0079] Among them, θ grepresents the neural network parameters of the global discriminator. Since the global mutual information focuses on all parts of the image at the same time, maximizing the global mutual information can optimize the performance of the image reconstruction task.

[0080] For image classification tasks, we should focus on local image blocks rather than the entire image, because the edges detected in local image blocks contain the structural knowledge of the image and can improve the performance of classification tasks. Figure 3 As shown, the joint source channel coding network consists of two parts, satisfying Part IE ω The function of (·) is to extract feature maps from the source image The N×N feature vectors of depth C′ of the feature map correspond to the structural information of the N×N local image blocks describing the source image, which are expressed as follows:

[0081] f=E ω (s)

[0082] Part IIE φ The function of (·) is to aggregate all feature vectors into semantic representations.

[0083] Therefore, in order to enhance the discriminative performance of semantic representation in classification tasks, each local image block of the image is used as the receptive field, and the average JS divergence between the N×N local image blocks of the source image and the semantic representation z is maximized by optimizing the joint source channel coding network and the local discriminator as follows:

[0084]

[0085] Where i represents the local image block index, θ l Represents the neural network parameters of the local discriminator. Unlike traditional image classification training methods based on label supervised learning, the semantic representation obtained by joint source-channel coding based on mutual information maximization training through local mutual information maximization contains the edge structure knowledge detected in each local image block, which can be used to distinguish image samples of different categories. The classification method is based on the similarities and differences in visual features between image samples, rather than relying on the data matching between each sample and its label. Therefore, it is an unsupervised learning method, and the training process of semantic representation does not require the participation of labels.

[0086] By jointly training the joint source-channel codec network based on mutual information maximization, the global discriminator, and the local discriminator, the global mutual information and the local mutual information can be optimized simultaneously, thereby learning a semantic representation that is both generative and discriminative. In addition, the performance of the image reconstruction task is not only affected by the quality of the semantic representation, but also depends on the influence of the joint source-channel decoding network. Therefore, the joint source-channel codec based on mutual information maximization simultaneously minimizes the end-to-end mean square error to reduce the pixel-level difference between the source image and the reconstructed image. The final loss function is defined as follows:

[0087]

[0088] Among them, λ is used to adjust the balance between image reconstruction performance and semantic representation quality, and μ1 and μ2 control the degree of attention paid to global and local mutual information during training. By adjusting the above hyperparameters, the joint source channel communication framework based on mutual information maximization can focus only on image classification tasks or image reconstruction tasks, or can weigh the performance of the two tasks to simultaneously meet the needs of the receiving end for both tasks.

[0089] In this embodiment, the semantic representation output by the joint source channel coding network is composed of 32-bit floating point numbers, and the semantic symbols directly mapped have amplitudes and phases that are continuously changing and randomly distributed over a large range. This nearly continuous constellation mapping requires the system to be able to identify high-resolution amplitude offsets and phase offsets, which cannot be met by existing RF hardware designs. In order to enable the joint source channel coding and decoding based on mutual information maximization to be applied to actual wireless communication systems, the semantic representation quantization method is based on asymmetric quantization, which can quantize the semantic representation output by the joint source channel coding network into fewer bits, thereby mapping it to a constellation diagram with fewer bits, more regular distribution, and easier to distinguish amplitude and phase offsets.

[0090] Specifically, first, each floating point number z in the semantic representation z j quantized to q-bit integers

[0091]

[0092] Among them, ρ s Represents a scaling factor that maps a 32-bit floating point number from its original range to a smaller range [0, 2 q -1]. The scaling factor is calculated as follows:

[0093]

[0094] The round function is used to round up, and the clamp function is used to remove the range exceeding [0, 2 q -1], defined as:

[0095]

[0096] Therefore, the quantized semantic representation can be mapped into discretely distributed semantic symbols, which is only composed of 2 q -1 constellation point mapping with easily identifiable amplitude and phase differences between constellation points.

[0097] Since quantization will cause a sudden change in the input parameters of the subsequent joint source channel decoding network, which will in turn lead to a decrease in learning performance, the quantized value cannot be used directly. Therefore, a dequantization operation is required to make the quantized semantic representation have a similar numerical distribution as before quantization while maintaining the discrete characteristics. The dequantization operation is expressed as follows:

[0098]

[0099] The semantic representation vector after dequantization is expressed as It can still be mapped to 2 q -1 constellation point, but numerically similar to the semantic representation vector z before quantization. The larger the number of quantization bits q, the closer the semantic representation after quantization is to the original semantic representation, the less information loss, and the better the target task performance at the receiving end, but with higher communication overhead. By adjusting the number of quantization bits q, the semantic transmission overhead and target task performance can be balanced. A few rounds of fine-tuning under the trained joint source channel coding and decoding framework based on mutual information maximization can make the joint source channel coding and decoding framework based on mutual information maximization adapt to different quantization bit numbers.

[0100] In this embodiment, since the channel environment in the actual application scenario is dynamically changing, the length of the semantic representation should be adjusted according to the changing channel conditions. Specifically, more dimensions should be transmitted under poor channel conditions, and conversely, fewer dimensions should be activated under good channel conditions to save communication resources. Therefore, the simultaneous multi-task semantic communication method introduces adaptive variable-length semantic representation coding, which can sequentially activate a corresponding number of semantic representation dimensions according to the current channel state, such as Figure 4 Therefore, in addition to transmitting the shortened semantic representation vector after masking, the sender only needs to transmit an index indicating the end position to let the receiver know the activation status of the semantic representation, thereby ensuring the normal progress of the subsequent process.

[0101] First, the transmitter receives information about the channel state through the feedback channel, including signal-to-noise ratio, channel fading, etc. The channel state information can be input into the joint source channel coding network as auxiliary knowledge to support adaptive semantic representation variable length coding. 2The input is sent to the lightweight multi-layer perceptron g(·) deployed at the sender to generate an activation mask m(·) with the same dimension as the original semantic representation. In order to activate more dimensions when the channel state is poor, each dimension of the multi-layer perceptron output g i (δ 2 ) should be about the noise variance δ 2 In addition, in order to achieve the continuous activation characteristics of semantic representation, the activation mask should be multiplied by the upper triangular unit matrix to ensure that the value of each dimension of the mask is less than the value of the previous dimension, that is, the value is monotonically decreasing starting from the first dimension. The calculation method of any dimension k of the activation mask is as follows:

[0102]

[0103]

[0104]

[0105] Therefore, the larger the noise variance, the larger the value of each dimension of the activation mask, and it decreases from the first dimension. The pruning threshold η can be predefined according to the trade-off between task performance and communication overhead. The mask dimension with a value greater than η is assigned a value of 1, indicating that the semantic representation of the dimension is temporarily activated, and vice versa, it is assigned a value of 0, indicating that the semantic representation of the dimension is temporarily pruned:

[0106]

[0107] Therefore, the larger the noise variance, the more semantic representation dimensions are activated, and the distribution is continuous. Finally, at the sending end, the Hadamard product of the assigned activation mask and the original semantic representation is used to obtain the semantic representation after variable length coding.

[0108]

[0109] The multi-layer perceptron is integrated into the joint source channel coding network for joint training. The activation mask obtained can continuously activate important dimensions in the semantic representation. The importance of each dimension can be automatically learned during the joint training process. A few rounds of fine-tuning are performed under the trained framework. In each small batch training process, a noise variance value is randomly selected from the possible noise variance value range as the training input, so that the framework can realize adaptive variable-length semantic representation coding under different channel states.

[0110] In summary, the embodiment of the present invention constructs a semantic communication basic framework based on mutual information maximization, and the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end, and the semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on the source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on the received semantic information; a semantic representation quantization model is constructed, and the semantic representation quantization model performs preset precision quantization on the semantic information representation at the semantic communication sending end, and the semantic symbol obtained after quantization is represented by the number of bits, and the semantic symbol obtained after quantization is mapped to the constellation point; an adaptive variable-length semantic representation coding model is constructed, and the adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead under the premise of ensuring the target task performance, and the target task includes at least one of classification, recognition, reconstruction, clustering, and retrieval. The present invention can complete multiple tasks simultaneously through a semantic information extraction, thereby improving the transmission efficiency of semantic information and reducing communication and computing overhead under the premise of ensuring multi-task performance.

[0111] It should be noted that the method of the embodiment of the present invention can be performed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present invention, and the multiple devices will interact with each other to complete the described method.

[0112] It should be noted that some embodiments of the present invention are described above. In some cases, the actions or steps recorded can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] See also Figure 5 Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present invention further provides a multi-task oriented semantic communication device, including:

[0114] The semantic communication basic processing module 100 is used to construct a semantic communication basic framework based on mutual information maximization, wherein the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end, wherein the semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on source information; and the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on received semantic information;

[0115] A semantic representation quantization module 200 is used to construct a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point;

[0116] The adaptive variable-length semantic representation coding module 300 is used to construct an adaptive variable-length semantic representation coding model. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

[0117] In this embodiment, in the semantic communication basic processing module 100:

[0118] The semantic communication sending end uses a convolutional neural network and a deep neural network to represent the semantic information of the source information;

[0119] The semantic communication sending end performs information encoding on the source information, including semantic encoding, source encoding and channel encoding;

[0120] Reconstructing the received semantic information and restoring the source information by using an artificial intelligence model;

[0121] The semantic communication receiving end performs information decoding on the received semantic information, including semantic decoding, information source decoding and channel decoding.

[0122] In this embodiment, in the semantic representation quantization module 200:

[0123] The semantic representation quantization model utilizes an artificial intelligence model and maximizes global and local mutual information to make the semantic information representation both generative and discriminative;

[0124] An asymmetric quantization algorithm is used to map the quantized semantic symbols to constellation points with amplitude and phase intervals to adapt to the RF hardware design in actual digital communication systems.

[0125] In this embodiment, in the adaptive variable-length semantic representation encoding module 300:

[0126] The current channel state based on the adaptive variable length semantic representation coding model includes the signal-to-noise ratio and noise variance at the current moment;

[0127] The target task requirement conditions on which the adaptive variable-length semantic representation coding model is based include a semantic representation pruning threshold and a target task performance threshold.

[0128] The device of the above embodiment is used to implement a corresponding multi-task oriented semantic communication method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0129] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-task oriented semantic communication method described in any of the above embodiments is implemented.

[0130] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other through the bus 450 in the device.

[0131] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0132] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 420 and are called and executed by the processor 410.

[0133] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0134] The communication interface 440 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0135] The bus 450 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the memory 420, the input / output interface 430, and the communication interface 440).

[0136] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0137] The electronic device of the above embodiment is used to implement a corresponding multi-task oriented semantic communication method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0138] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present invention also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute a multi-task oriented semantic communication method as described in any of the above embodiments.

[0139] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0140] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute a multi-task oriented semantic communication method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0141] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of the embodiments of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0142] In addition, to simplify the description and discussion, and in order not to obscure the embodiments of the present invention, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present invention, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present invention will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present invention, it will be apparent to those skilled in the art that embodiments of the present invention may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0143] Although the invention has been described in conjunction with specific embodiments of the invention, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0144] The embodiments of the present invention are intended to cover all such substitutions, modifications and variations that fall within the scope of the claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task oriented semantic communication method, wherein: include: Constructing a semantic communication basic framework based on mutual information maximization, the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end, the semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on received semantic information; Constructing a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point; An adaptive variable-length semantic representation coding model is constructed. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

2. A multi-task oriented semantic communication method according to claim 1, wherein: The semantic communication sending end uses a convolutional neural network and a deep neural network to represent the semantic information of the source information; The semantic communication sending end performs information encoding on the source information including semantic encoding, source encoding and channel encoding.

3. A multi-task oriented semantic communication method according to claim 1, wherein: The semantic representation quantization model utilizes an artificial intelligence model and maximizes global and local mutual information, so that the semantic information representation is both generative and discriminative.

4. A multi-task oriented semantic communication method according to claim 1, wherein: In the process of restoring the received semantic information, the semantic communication receiving end uses an artificial intelligence model to reconstruct the received semantic information and restore the source information; The semantic communication receiving end performs information decoding on the received semantic information, including semantic decoding, information source decoding and channel decoding.

5. A multi-task oriented semantic communication method according to claim 1, wherein: In the process of mapping the semantic symbols obtained after quantization to constellation points, an asymmetric quantization algorithm is used to map the quantized semantic symbols to constellation points with amplitude and phase intervals to adapt to the RF hardware design in actual digital communication systems.

6. A multi-task oriented semantic communication method according to claim 1, wherein: The current channel state based on the adaptive variable length semantic representation coding model includes the signal-to-noise ratio and noise variance at the current moment; The target task requirement conditions on which the adaptive variable-length semantic representation coding model is based include a semantic representation pruning threshold and a target task performance threshold.

7. A multi-task oriented semantic communication device, wherein: include: A semantic communication basic processing module is used to construct a semantic communication basic framework based on mutual information maximization, wherein the semantic communication basic framework based on mutual information maximization includes a semantic communication sending end and a semantic communication receiving end. The semantic communication sending end performs semantic information representation, information encoding, and information transformation processing on source information; the semantic communication receiving end performs semantic information recovery, information decoding, and task reasoning processing on received semantic information; A semantic representation quantization module, used to construct a semantic representation quantization model, wherein the semantic representation quantization model quantizes the semantic information representation with a preset accuracy at the semantic communication sending end, represents the semantic symbol obtained after quantization with a number of bits, and maps the semantic symbol obtained after quantization to a constellation point; An adaptive variable-length semantic representation coding module is used to construct an adaptive variable-length semantic representation coding model. The adaptive variable-length semantic representation coding model selectively activates the specified dimensions of the semantic information representation in sequence and continuously based on the current channel state and the target task requirements, and adaptively reduces the semantic representation transmission overhead while ensuring the performance of the target task. The target task includes at least one of classification, recognition, reconstruction, clustering and retrieval.

8. A multi-task oriented semantic communication device according to claim 7, wherein: In the semantic communication basic processing module: The semantic communication sending end uses a convolutional neural network and a deep neural network to represent the semantic information of the source information; The semantic communication sending end performs information encoding on the source information, including semantic encoding, source encoding and channel encoding; Reconstructing the received semantic information and restoring the source information by using an artificial intelligence model; The semantic communication receiving end performs information decoding on the received semantic information, including semantic decoding, source decoding and channel decoding; In the semantic representation quantization module: The semantic representation quantization model utilizes an artificial intelligence model and maximizes global and local mutual information to make the semantic information representation both generative and discriminative; An asymmetric quantization algorithm is used to map the quantized semantic symbols to constellation points with amplitude and phase intervals to adapt to the RF hardware design in actual digital communication systems. In the adaptive variable-length semantic representation coding module: The current channel state based on the adaptive variable length semantic representation coding model includes the signal-to-noise ratio and noise variance at the current moment; The target task requirement conditions on which the adaptive variable-length semantic representation coding model is based include a semantic representation pruning threshold and a target task performance threshold.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-task oriented semantic communication method described in any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the multi-task oriented semantic communication method as described in any one of claims 1 to 6.

Images