Channel-adaptive sparse semantic visual communication method and system based on alternating learning

Abstract

The invention discloses a channel-adaptive sparse semantic visual communication method and system based on alternating learning, comprising: inputting an image to be processed into a pre-trained encoder to obtain a first semantic embedding representation; expanding the first semantic embedding representation to obtain a first tensor; multiplying the first tensor with a first semantic basis to quantize the representation to obtain a two-dimensional sparse representation; transmitting the two-dimensional sparse representation through a channel to obtain a noise signal; when the signal-to-noise ratio of the channel is less than a threshold, denoising the noise signal through a pre-trained denoising module to obtain a second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, using the noise signal as the second tensor; multiplying the second tensor with a second semantic basis to dequantize the representation to obtain a third tensor; reducing the dimension of the third tensor to obtain a second semantic embedding representation; inputting the second semantic embedding representation into a pre-trained decoder to obtain a restored image.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to a channel-adaptive sparse semantic visual communication method and system based on alternating learning. Background Art

[0002] With the emergence of the smart Internet, the classic bit-level precision-oriented communication technology has approached the Shannon limit and cannot meet the stringent requirements of the smart Internet era. Semantic communication focuses on semantics rather than bit errors in communication transmission, which has promoted the application of artificial intelligence technology in communication in recent years. In this regard, a series of joint source-channel coding (JSCC) communication systems have emerged, and the communication quality has been improved compared with traditional systems. However, in large-scale practical wireless communications, the gradient cannot be fully back-propagated through the physical channel. Therefore, the strong assumption of microchannels in JSCC communication systems may not hold in practice, which poses a major challenge to existing models.

[0003] Due to the widespread application needs of images and videos, signal processing techniques specifically for image and video compression have also developed significantly. Compressed sensing (CS), as a sub-Nyquist sampling framework, has been introduced into communications for effective compression. Unfortunately, there are few works that explicitly and effectively combine sparsity and semantic communication in semantic images and videos. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a channel-adaptive sparse semantic visual communication method and system based on alternating learning.

[0005] In a first aspect, an embodiment of the present invention provides a channel-adaptive sparse semantic visual communication method based on alternating learning, the method comprising:

[0006] Input the input image into the pre-trained encoder to obtain the first semantic embedding representation;

[0007] Expanding the first semantic embedding representation to obtain a first tensor; multiplying the first tensor by the first semantic basis to quantize the first tensor to obtain a two-dimensional sparse representation;

[0008] The two-dimensional sparse representation is transmitted through a channel to obtain a noise signal;

[0009] When the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by the pre-trained denoising module to obtain the second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor;

[0010] The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation;

[0011] The second semantic embedding representation is input into the pre-trained decoder to restore the input image.

[0012] In a second aspect, an embodiment of the present invention provides a channel-adaptive sparse semantic visual communication system based on alternating learning, the system comprising:

[0013] The transmitting end is used to input the image to be processed into a pre-trained encoder to obtain a first semantic embedding representation; expand the first semantic embedding representation to obtain a first tensor; multiply the first tensor by the first semantic basis to quantize it to obtain a two-dimensional sparse representation; and transmit the two-dimensional sparse representation to the receiving end through a channel;

[0014] The receiving end is used to receive the noise signal transmitted by the sending end; when the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by a pre-trained denoising module to obtain a second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor; the second tensor is multiplied by the second semantic basis for dequantization to obtain a third tensor; the third tensor is reduced in dimension to obtain a second semantic embedding representation; the second semantic embedding representation is input into a pre-trained decoder to obtain a restored image.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned channel-adaptive sparse semantic visual communication method based on alternating learning.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned channel-adaptive sparse semantic visual communication method based on alternating learning.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned channel-adaptive sparse semantic visual communication method based on alternating learning.

[0018] The beneficial effects of the present invention are:

[0019] (1) The present invention converts the semantic embedding representation into the semantic basis space at the transmitting end, and quantizes the result by establishing a nonlinear mapping. At the receiving end, the semantic basis space is converted into the semantic embedding for dequantization to achieve efficient channel transmission.

[0020] (2) The present invention trains the encoder, decoder and denoising module based on alternating learning; fix the decoder and train the encoder; fix the encoder and train the decoder; fix the encoder and decoder and train the denoising module; the present invention effectively combines the concept of compressed sensing and the advantages of deep neural networks, and introduces a "self-critic" mechanism in the training process to achieve better exploration and stable learning and training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of a flow chart of a channel-adaptive sparse semantic visual communication method based on alternating learning provided in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the training of the encoder, decoder, and denoising module provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the U-Net structure of the denoising module provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a channel adaptive sparse semantic visual communication method based on alternating learning, and the method specifically includes the following steps:

[0029] Step S1, input the input image I into a pre-trained encoder to obtain a first semantic embedding representation (Semantic Embedding).

[0030] Step S2, expand the first semantic embedding representation to obtain a first tensor x; multiply the first tensor x by the first semantic basis to quantize and obtain a two-dimensional sparse representation in

[0031] Furthermore, the matrix size of the first tensor x is 1×M, the matrix size of the first semantic basis is M×N, and the two-dimensional sparse representation The matrix size is 1×N.

[0032] It should be noted that each image can be completely described by a semantic basis. Through the processing of the quantization module, the image can be sparsely represented by a set of specific bases, thereby being converted into a bit sequence with a higher compression ratio for transmission.

[0033] Step S3, the two-dimensional sparse representation is transmitted through the channel to obtain a noise signal; the expression is as follows:

[0034]

[0035] Where h is the channel coefficient and n is the channel noise impact.

[0036] Step S4: When the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by a pre-trained denoising module to obtain a second tensor When the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor

[0037] Furthermore, in this example, the threshold is 12 dB.

[0038] Step S5: convert the second tensor Multiply it with the second semantic basis to dequantize and obtain the third tensor; reduce the dimension of the third tensor to obtain the second semantic embedding representation z;

[0039] Step S6: input the second semantic embedding representation into the pre-trained decoder to restore the input image.

[0040] It should be noted that if Figure 2 As shown, the encoder includes several convolutional layers with LeakyReLU activation function to extract semantic features from the image; the decoder includes several transposed convolutional layers with LeakyReLU activation function to restore the image.

[0041] It should be noted that in the process of multiplying the first tensor with the first semantic basis for quantization, and multiplying the second tensor with the second semantic basis for dequantization, the DNNs of the quantization module and the dequantization module are composed of a fully connected layer with a Tanh activation function, which converts the semantically encoded bits into a digital sequence from -1 to 1, and then performs binary quantization to obtain a binary bit vector of length N (i.e., only contains 0 or 1).

[0042] It should be noted that if Figure 3 As shown in the figure, referring to the Denoising Diffusion Probabilistic Models (DDPM), this example selects the U-Net of the Encoder-Decoder structure as the backbone network of the denoising module. The two sides of the U-Net are stacked by the same number of encoder blocks and decoder blocks, and also include attention blocks and residual blocks. The downsampling block contains convolutional layers and attention blocks, which can better compress the data, while the input of the upsampling block consists of the output of the previous decoder block and the output of the encoder block at the same level, ensuring that the decoder does not lose information during reasoning and reconstruction. In addition, time embedding is obtained through a multi-layer perceptron (MLP). Considering the large number of parameters in the general U-Net, this example reduces the number of encoder blocks and decoder blocks to adapt to the communication system. In this example, the number of encoders and decoders is set to 2.

[0043] Furthermore, the training process of the encoder, decoder and denoising module includes:

[0044] Step S100, fixing the decoder and training the encoder using a self-critic technique based on Gaussian policy gradient.

[0045] The sender trains an optimal strategy, which is expressed as follows:

[0046]

[0047] In the formula, Represents a decoder, represents the channel, Represents an encoder.

[0048] The input image is fed into the encoder, and a two-dimensional sparse representation of the m-fold repeated encoding is generated according to the Gaussian distribution;

[0049]

[0050] Among them, S means it obeys Gaussian distribution Parallel sampling. It is a covariance matrix consisting of the identity matrix E and the exploration factor σ.

[0051] The two-dimensional sparse representation is transmitted through the channel to obtain a noise signal, and the noise signal is used as the second tensor;

[0052] The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation;

[0053] The second semantic embedding representation is input into the decoder to obtain m rewards. For any j, j∈{1,…,m}, the average of the remaining m-1 outputs is used as the bias term, and the difference between the jth reward and the bias term is calculated to update the encoder based on forward propagation.

[0054] Step S200, fixing the encoder and training the decoder based on the self-criticism technique of Gaussian policy gradient.

[0055] The receiving end trains an optimal strategy, which is expressed as follows:

[0056]

[0057] The input image is fed into the encoder to generate a two-dimensional sparse representation

[0058] The two-dimensional sparse representation is transmitted through the channel to obtain a noise signal, and the noise signal is used as the second tensor;

[0059] The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation;

[0060] The second semantic embedding representation is input into the decoder, and the result representation of m repeated decoding is generated according to the Gaussian distribution:

[0061]

[0062] In the formula, S means it obeys Gaussian distribution Parallel sampling.

[0063] m rewards are calculated from these m results; for any j, j∈{1,…,m}, the average of the remaining m-1 outputs is used as the bias term, and the difference between the jth reward and the bias term is calculated to update the decoder based on forward propagation.

[0064] Step S300, fixing the encoder and decoder, and training the denoising module.

[0065] Forward process: The input signal is incrementally perturbed by Gaussian noise ∈ when transmitted through the channel, and the t-times distribution of the noise signal is obtained.

[0066] The transfer function of step t can be expressed as:

[0067]

[0068] in, is Gaussian noise;

[0069] That is, the tth data can be distributed using Gaussian distribution definition.

[0070] The t-times distribution of the noise signal is given by the original data To express:

[0071]

[0072] Reverse process: reconstruct the input signal from the t-times distribution of the noise signal.

[0073] In this example, a variational lower bound (VLB) is used to optimize the negative log-likelihood function of the target:

[0074]

[0075] Among them, D KL represents the Kullback-Leibler (KL) divergence, which is used to measure the similarity between two Gaussian distributions. In addition, It can also be expressed as sum. is a constant, is considered as a decoder that recovers the data, and they can all be omitted during training and approximated and simplified by a constant C. In this case, by ignoring the weighted terms, the loss function can be modified to:

[0076]

[0077] This guides the gradient descent and updates the network weights to iteratively update the denoising model by minimizing the prediction error.

[0078] Step S400, fine-tuning the encoder, decoder and denoising module.

[0079] The embodiment of the present invention further provides a channel-adaptive sparse semantic visual communication system based on alternating learning, the system comprising:

[0080] The transmitting end is used to input the image to be processed into a pre-trained encoder to obtain a first semantic embedding representation; expand the first semantic embedding representation to obtain a first tensor; multiply the first tensor by the first semantic basis to quantize it to obtain a two-dimensional sparse representation; and transmit the two-dimensional sparse representation to the receiving end through a channel;

[0081] The receiving end is used to receive the noise signal transmitted by the sending end; when the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by a pre-trained denoising module to obtain a second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor; the second tensor is multiplied by the second semantic basis for dequantization to obtain a third tensor; the third tensor is reduced in dimension to obtain a second semantic embedding representation; the second semantic embedding representation is input into a pre-trained decoder to obtain a restored image.

[0082] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0083] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0084] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned channel-adaptive sparse semantic visual communication method based on alternating learning. Figure 4 As shown, a hardware structure diagram of any device with data processing capability in which the channel adaptive sparse semantic visual communication method based on alternating learning provided by an embodiment of the present invention is located, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0085] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the channel-adaptive sparse semantic visual communication method based on alternating learning as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.

[0087] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A channel-adaptive sparse semantic visual communication method based on alternating learning, characterized in that: The method comprises: Input the input image into the pre-trained encoder to obtain the first semantic embedding representation; Expanding the first semantic embedding representation to obtain a first tensor; multiplying the first tensor by the first semantic basis to quantize the first tensor to obtain a two-dimensional sparse representation; The two-dimensional sparse representation is transmitted through a channel to obtain a noise signal; When the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by the pre-trained denoising module to obtain the second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor; The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation; Inputting the second semantic embedding representation into a pre-trained decoder to restore the input image; The training process of the encoder, decoder and denoising module includes: fixing the decoder and training the encoder based on the self-criticism technique of Gaussian policy gradient; fixing the encoder and training the decoder based on the self-criticism technique of Gaussian policy gradient; fixing the encoder and decoder and training the denoising module; Among them, the fixed decoder and the encoder trained by the self-criticism technique based on Gaussian policy gradient include: The input image is fed into the encoder, and a two-dimensional sparse representation of the m-fold repeated encoding is generated according to the Gaussian distribution; The two-dimensional sparse representation is transmitted through the channel to obtain a noise signal, and the noise signal is used as the second tensor; The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation; The second semantic embedding representation is input into the decoder to obtain m rewards. For any j, j∈{1,…,m}, the average value of the remaining m-1 outputs is used as the bias term, and the difference between the jth reward and the bias term is calculated to update the encoder based on the forward propagation. Among them, the fixed encoder and the decoder trained by the self-criticism technology based on Gaussian policy gradient include: The input image is fed into a fixed encoder to generate a two-dimensional sparse representation; The two-dimensional sparse representation is transmitted through the channel to obtain a noise signal, and the noise signal is used as the second tensor; The second tensor is multiplied by the second semantic basis to be dequantized to obtain a third tensor; the third tensor is dimensionally reduced to obtain a second semantic embedding representation; The second semantic embedding representation is input into the decoder, and the restored image is generated m times of repeated decoding according to the Gaussian distribution to obtain m rewards. For any j, j∈{1,…,m}, the average of the remaining m-1 outputs is used as the bias term, and the difference between the jth reward and the bias term is calculated, and the decoder is updated based on the forward propagation.

2. According to claim 1, a channel-adaptive sparse semantic visual communication method based on alternating learning is characterized in that: The encoder includes several convolutional layers with LeakyReLU activation function; the decoder includes several transposed convolutional layers with LeakyReLU activation function.

3. The channel-adaptive sparse semantic visual communication method based on alternating learning according to claim 1, characterized in that: Fixed encoder and decoder, training denoising module includes: Forward process: The input signal is perturbed by Gaussian noise ε increments through the channel transmission, and the t-times distribution of the noise signal is obtained; Reverse process: reconstruct the input signal from the t-times distribution of the noise signal.

4. A channel-adaptive sparse semantic visual communication system based on alternating learning, characterized in that: To implement the channel-adaptive sparse semantic visual communication method based on alternating learning as described in any one of claims 1 to 3, the system comprises: The transmitting end is used to input the image to be processed into a pre-trained encoder to obtain a first semantic embedding representation; expand the first semantic embedding representation to obtain a first tensor; multiply the first tensor by the first semantic basis to quantize it to obtain a two-dimensional sparse representation; and transmit the two-dimensional sparse representation to the receiving end through a channel; The receiving end is used to receive the noise signal transmitted by the sending end; when the signal-to-noise ratio of the channel is less than the threshold, the noise signal is denoised by a pre-trained denoising module to obtain a second tensor; when the signal-to-noise ratio of the channel is greater than the threshold, the noise signal is used as the second tensor; the second tensor is multiplied by the second semantic basis for dequantization to obtain a third tensor; the third tensor is reduced in dimension to obtain a second semantic embedding representation; the second semantic embedding representation is input into a pre-trained decoder to obtain a restored image.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the channel-adaptive sparse semantic visual communication method based on alternating learning as described in any one of claims 1-3 above.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the channel-adaptive sparse semantic visual communication method based on alternating learning as described in any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the channel-adaptive sparse semantic visual communication method based on alternating learning described in any one of claims 1 to 3 is implemented.

Images