Semantic communication method and device based on progressive channel feedback, equipment and medium

Abstract

The invention relates to a semantic communication method and device based on progressive channel feedback, equipment and a medium. The method comprises the following steps: constructing a semantic communication framework based on progressive channel feedback, wherein the framework consists of a transmitting end, a wireless channel, a feedback channel and a receiving end; at a sending end, through a fusion module and an encoder, the content of the current transmission can be optimized by fusing and utilizing the semantics information of historical transmission, and the accuracy of encoder semantics feature extraction is improved; at a receiving end, through a decoder, a channel cache structure, a decoder cache module and a semantic fusion module, historically transmitted semantic information and channel information can be gradually fused to optimize semantic decoding and reconstruction, and the reliability of semantic image reconstruction is improved. By adopting the method, resource utilization can be improved, and the reliability of semantic communication can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic communication, in particular to a semantic communication method and device based on progressive channel feedback, equipment and medium. BACKGROUND

[0002] Semantic communication (SemCom) is a new type of communication technology that encodes and transmits information semantic features, breaks through the theoretical limitations of traditional syntax communication, and is becoming a key technology for efficient and robust transmission in future networks.

[0003] To improve the reliability of semantic communication, the hybrid automatic repeat request (HARQ) scheme is attracting more and more attention. However, the existing semantic HARQ method usually relies on redundant retransmission and fails to effectively utilize past semantic information and channel information, resulting in low resource utilization efficiency and further improvement of communication reliability. SUMMARY

[0004] Therefore, it is necessary to provide a semantic communication method and device based on progressive channel feedback to solve the technical problems of low resource utilization and poor reliability of the above-mentioned semantic communication.

[0005] A semantic communication method based on progressive channel feedback, the method comprising:

[0006] A semantic communication framework based on progressive channel feedback is constructed, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end. The sending end includes a fusion module and an encoder, and the receiving end includes a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module.

[0007] At the sending end, the image vector sequence input in the current transmission and the reconstructed semantic image output in the previous transmission transmitted back through the feedback channel are obtained and input into the fusion module for feature fusion to obtain the input of the encoder in the current transmission, and the semantic features extracted by the encoder are transmitted to the wireless channel for transmission.

[0008] At the receiving end, the outputs of the wireless channel in the historical several rounds of transmission are buffered by the channel buffer structure, and the input of the decoder in the current transmission is obtained by splicing, and the semantic decoding is performed by the decoder to obtain the preliminary semantic estimation result. The reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output by the decoder in the current transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission, and the current transmission is completed. The reconstructed semantic image output in each round of transmission is buffered in the decoder buffer module.

[0009] In one of the embodiments, the image vector sequence input of the current transmission is obtained, including:

[0010] Before the start of each transmission, the source image is obtained and is divided into a grid composed of l1=p×p non-overlapping image blocks, each image block is regarded as a token, and all tokens are arranged in order from left to right and from top to bottom, to obtain the image vector sequence input of each transmission wherein h, w and 3 represent the height, width and RGB three color channels of the source image respectively, is a real set; the hyperparameter p is used to control the granularity of the source image division; the token is defined as the embedding vector of the image block, x i represents the i-th token in the image vector sequence.

[0011] In one of the embodiments, the image vector sequence input of the current transmission and the reconstructed semantic image of the output of the previous transmission transmitted back through the feedback channel are obtained, and are input into the fusion module for feature fusion, to obtain the input of the encoder in the current transmission, including:

[0012] The reconstructed semantic image of the output of the i-1th transmission transmitted back through the feedback channel is obtained is represented as:

[0013]

[0014] wherein represents the reconstructed semantic image of the output of the i-1th transmission; H f represents the feedback channel modeled as an ideal signal or an additive white Gaussian noise feedback channel; W f is the additive white Gaussian noise in the feedback channel, and W f each element in W independently obeys a complex Gaussian distribution with a mean of 0 and a variance of , that is, wherein is the i-th element in W f .

[0015] and the image vector sequence S s input of the i-th transmission are input into the fusion module. In the fusion module, firstly, S is embedded and mapped to a feature space with the same feature dimension as the token in S s for feature alignment in the spatial dimension, then S after the feature alignment is spliced with S s in the channel dimension, and after fusion through a linear projection layer, the input S ini of the encoder in the i-th transmission is generated, represented as:

[0016]

[0017] where f e denotes the embedding mapping process, Concat denotes the feature concatenation, and Linear denotes the linear projection layer.

[0018] In one embodiment, the semantic feature extracted by the encoder is transmitted to the wireless channel, including:

[0019] The input S ini of the encoder in the i-th transmission round is processed by the encoder to obtain the semantic feature representation X i output by the encoder in the i-th transmission round, expressed as:

[0020]

[0021] where denotes the encoder in the i-th transmission round, and a is the learnable parameter of the encoder.

[0022] The X i processed by the power normalization is transmitted to the wireless channel for transmission to obtain the output Y i of the wireless channel in the i-th transmission round, expressed as:

[0023]

[0024] where denotes the wireless channel modeled as an additive white Gaussian noise channel or a slow fading channel; W i is the additive white Gaussian noise in the wireless channel, and W i each element of which is independently subject to a complex Gaussian distribution with mean 0 and variance , i.e. η denotes the channel gain and η is subject to a complex Gaussian distribution with mean 0 and variance , i.e.

[0025] In one embodiment, the outputs of the wireless channel in the historical transmission rounds are cached by the channel cache structure, and the input of the decoder in the current transmission round is obtained by concatenation, the semantic decoding is performed by the decoder to obtain the preliminary semantic estimation result, including:

[0026] The output Y i of the wireless channel in the i-th transmission round is received by the channel cache structure, and the outputs of the wireless channel in the historical transmission rounds cached by itself are concatenated to obtain the input ch buffer,i of the decoder in the i-th transmission round, expressed as:

[0027] chbuffer,i = [Y1, Y2, …, Y i ];

[0028] wherein, i > 1;

[0029] performing semantic decoding on ch buffer,i to obtain a preliminary semantic estimation result of the decoder output in the ith transmission round is denoted as:

[0030]

[0031] wherein, denotes the decoder in the ith transmission round, and β is a learnable parameter of the decoder.

[0032] In one of the embodiments, the reconstructed semantic image output in the previous transmission round and the preliminary semantic estimation result output by the decoder in the current transmission round are input into a semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission round, including:

[0033] the reconstructed semantic image output in the (i-1)th transmission round cached in the decoder cache module and the preliminary semantic estimation result output by the decoder in the ith transmission round are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission round, denoted as:

[0034]

[0035] wherein, Combiner denotes the semantic fusion module, which is composed of a feature concatenation layer, a first linear projection layer and a second linear projection layer connected in sequence, and the two linear projection layers are connected through an activation function ReLU.

[0036] In one of the embodiments, the above method further includes:

[0037] After each transmission round, an automatic repeat request feedback signal fed back by the receiving end is obtained, if the signal is NACK, the sending end is notified to continue retransmission, and the reconstructed semantic image output in the current transmission round cached in the decoder cache module is transmitted back to the sending end through the feedback channel for the next transmission round; if the signal is ACK, the retransmission is terminated, and the information cached in the channel cache structure and the decoder cache module is emptied.

[0038] A semantic communication device based on progressive channel feedback, the device comprising:

[0039] A framework construction unit is configured to construct a semantic communication framework based on progressive channel feedback, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end. The sending end comprises a fusion module and an encoder, and the receiving end comprises a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module.

[0040] A semantic extraction unit is configured to, at the sending end, acquire an image vector sequence input in a current transmission and a reconstructed semantic image output in a previous transmission transmitted back through the feedback channel, input the image vector sequence and the reconstructed semantic image into the fusion module for feature fusion, obtain an input of the encoder in the current transmission, and send semantic features extracted by the encoder to the wireless channel for transmission.

[0041] A semantic reconstruction unit is configured to, at the receiving end, buffer outputs of the wireless channel in a plurality of historical transmissions through the channel buffer structure, splice the outputs to obtain an input of the decoder in the current transmission, perform semantic decoding by using the decoder to obtain a preliminary semantic estimation result, input the reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output in the current transmission by the decoder into the semantic fusion module for fusion, obtain a reconstructed semantic image output in the current transmission, and complete the current transmission. The reconstructed semantic image output in each transmission is buffered in the decoder buffer module.

[0042] A computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0043] A semantic communication framework based on progressive channel feedback is constructed, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end. The sending end comprises a fusion module and an encoder, and the receiving end comprises a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module.

[0044] At the sending end, an image vector sequence input in a current transmission and a reconstructed semantic image output in a previous transmission transmitted back through the feedback channel are acquired, and the image vector sequence and the reconstructed semantic image are input into a fusion module for feature fusion to obtain an input of an encoder in the current transmission. Semantic features extracted by the encoder are sent to the wireless channel for transmission.

[0045] At the receiving end, outputs of the wireless channel in a plurality of historical transmissions are buffered through a channel buffer structure, and the outputs are spliced to obtain an input of a decoder in the current transmission. The decoder is used for semantic decoding to obtain a preliminary semantic estimation result. The reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output in the current transmission by the decoder are input into a semantic fusion module for fusion to obtain a reconstructed semantic image output in the current transmission, and the current transmission is completed. The reconstructed semantic image output in each transmission is buffered in a decoder buffer module.

[0046] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the following steps:

[0047] A semantic communication framework based on progressive channel feedback is constructed, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end; wherein the sending end comprises a fusion module and an encoder, and the receiving end comprises a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module;

[0048] At the sending end, the image vector sequence input in the current transmission and the reconstructed semantic image output in the previous transmission transmitted back through the feedback channel are obtained and input into the fusion module for feature fusion to obtain the input of the encoder in the current transmission, and the semantic features extracted by the encoder are transmitted to the wireless channel for transmission;

[0049] At the receiving end, the output of the wireless channel in the historical several rounds of transmission is buffered through the channel buffer structure, and the input of the decoder in the current transmission is obtained by splicing, and the semantic decoding is performed by using the decoder to obtain the preliminary semantic estimation result; the reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output by the decoder in the current transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission, and the current transmission is completed; wherein the reconstructed semantic image output in each round of transmission is buffered in the decoder buffer module.

[0050] The above-mentioned semantic communication method, device, equipment and medium based on progressive channel feedback construct a wireless communication framework composed of a sending end, a wireless channel, a feedback channel and a receiving end, at the sending end, the reconstructed semantic image output in the previous transmission transmitted back through the feedback channel is received by the fusion module, and is fused with the image vector sequence input in the current transmission, so that the semantic information of the historical transmission can be effectively utilized to optimize the content of the current transmission, while improving the utilization rate of the semantic information, a more complete input is provided for subsequent semantic encoding, and the accuracy of semantic feature extraction is improved; at the receiving end, the information output by the channel in the historical transmission is progressively buffered by the channel buffer structure, while improving the utilization rate of the channel information, a more complete input is provided for the subsequent semantic decoding process, the accuracy of semantic decoding is improved, and the semantic estimation result output by the decoder in the current transmission and the reconstructed semantic image output in the previous transmission are progressively fused by the semantic fusion module, the reconstruction of the semantic image can be gradually optimized, the reliability of the semantic image reconstruction is improved, and thus the reliability of the semantic communication can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a flowchart of the semantic communication method based on progressive channel feedback in one embodiment;

[0052] Figure 2Fig. 1 is a schematic diagram of a semantic communication framework based on progressive channel feedback in one embodiment;

[0053] Figure 3 Fig. 2 is a schematic diagram of the structure of an encoder in one embodiment;

[0054] Figure 4 Fig. 3 is a schematic diagram of the structure of a decoder in one embodiment;

[0055] Figure 5 Fig. 4 is a schematic diagram of the structure of a semantic fusion module in one embodiment;

[0056] Figure 6 Fig. 5 is a schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0058] In one embodiment, a semantic communication method based on progressive channel feedback is provided, comprising the following steps:

[0059] Step S1, a semantic communication framework based on progressive channel feedback is constructed, which is composed of a sending end, a wireless channel, a feedback channel and a receiving end; wherein the sending end includes a fusion module and an encoder, and the receiving end includes a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module.

[0060] As shown in Fig. 1, the semantic communication framework based on progressive channel feedback iteratively optimizes the training parameters through a cross-training mechanism, and the channel buffer structure in the framework is realized based on the accumulation of the wireless channel output in the multi-round transmission process as shown in Fig. 2. Figure 2 Figure 2

[0061] Step S2, at the sending end, the image vector sequence input in the current round of transmission and the reconstructed semantic image output in the previous round of transmission transmitted back through the feedback channel are obtained and input into the fusion module for feature fusion to obtain the input of the encoder in the current round of transmission, and the semantic features extracted by the encoder are transmitted to the wireless channel for transmission.

[0062] As shown in Fig. 4, the fusion module can effectively utilize the semantic information of the historical transmission to optimize the content of the current round of transmission, which improves the utilization rate of semantic information and provides more complete input for subsequent semantic encoding, thereby improving the accuracy of the semantic features extracted by the encoder.

[0063] ​​Step S3, at the receiving end, the output of the wireless channel in the historical several rounds of transmission is cached through the channel cache structure, and spliced to obtain the input of the decoder in the current round of transmission, semantic decoding is performed by using the decoder to obtain the preliminary semantic estimation result; the reconstructed semantic image output in the previous round of transmission and the preliminary semantic estimation result output by the decoder in the current round of transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current round of transmission, and the current round of transmission is completed; wherein the reconstructed semantic image output in each round of transmission is cached in the decoder cache module.

[0064] Wherein, the information output by the channel in the historical transmission is progressively cached through the channel cache structure, which not only improves the utilization rate of channel information, but also provides more complete input for the subsequent semantic decoding process, improves the accuracy of semantic decoding, and progressively fuses the semantic estimation result output by the decoder in the current round and the reconstructed semantic image output in the previous round of transmission by using the semantic fusion module, which can gradually optimize the reconstruction of the semantic image and improve the reliability of the semantic image reconstruction, thereby improving the reliability of the semantic communication.

[0065] The above-mentioned semantic communication method based on progressive channel feedback progressively fuses the semantic information and channel information of the historical transmission, improves the resource use efficiency in the semantic communication process, and can realize more reliable semantic communication based on the more perfect fused information, thereby improving the performance of the semantic communication.

[0066] In one of the embodiments, the image vector sequence of the input of the current round of transmission is obtained, including:

[0067] Before each round of transmission starts, the source image is obtained and is divided into a grid composed of l1=p×p non-overlapping image blocks, each image block is regarded as a token, and all tokens are arranged in order from left to right and from top to bottom to obtain the image vector sequence of the input of each round of transmission Wherein, h, w and 3 respectively represent the height, width and RGB (red, green and blue) three color channels of the source image, is a real set; the hyperparameter p is used to control the granularity of the source image division, and the larger the value of p is, the finer the image can be divided, which is beneficial to learning more fine features, but also brings higher computational complexity; the token is defined as the embedding vector of the image block, x i represents the i-th token in the image vector sequence.

[0068] In one of the embodiments, the image vector sequence of the input of the current round of transmission and the reconstructed semantic image output in the previous round of transmission transmitted back through the feedback channel are obtained, and are input into the fusion module for feature fusion to obtain the input of the encoder in the current round of transmission, including:

[0069] First, obtain the reconstructed semantic image of the i-1th round of transmission output transmitted through the feedback channel Expressed as:

[0070]

[0071] in, represents the reconstructed semantic image output from the i-1th round of transmission; H f represents the feedback channel modeled as an ideal signal or as an additive white Gaussian noise (AWGN) feedback channel; W f is the additive white Gaussian noise in the feedback channel, and W f Each element in the equation has a mean of 0 and a variance of The complex Gaussian distribution of in W f The i-th element in .

[0072] Then The image vector sequence S of the i-th round transmission input s Input to the fusion module, in the fusion module, first Embedded map to S s The feature space with the same token feature dimension is used to align the features in the spatial dimension, and then the aligned features are aligned. With S s After concatenation in the channel dimension and fusion through a linear projection layer, the encoder input S transmitted in the i-th round is generated. ini , expressed as:

[0073]

[0074] Among them, f e Represents embedding mapping processing, which usually consists of two linear projection layers and a downsampling module; Concat represents feature concatenation, and Linear represents a linear projection layer.

[0075] In one embodiment, sending the semantic features extracted by the encoder to a wireless channel for transmission includes:

[0076] First, the encoder is used to transform the input S transmitted in round i ini Perform semantic feature extraction to obtain the semantic feature representation X output by the encoder in the i-th round of transmission i , the expression is:

[0077]

[0078] in, represents the encoder of the i-th round transmission, and α is the learnable parameter of the encoder. The encoder result is as followsFigure 3 As shown, the encoding process consists of multiple stages of stacking, the first stage contains a patch embedding layer for dividing the input image into multiple image patches and converting each image patch into a feature vector that can be processed by the subsequent SwinTransformer block; each of the other stages includes a patch merging layer followed by N i Swin Transformer blocks for gradually projecting the input tokens into semantic feature representations. The Swin Transformer block contains a standard multi-head self-attention (MSA) module and a feed-forward neural network. The sliding window-based self-attention mechanism enables the encoder to capture long-range dependencies in the image, which divides the image into multiple windows and performs self-attention operations independently within each window.

[0079] Secondly, the power normalized X i is sent to the wireless channel for transmission, and the output Y i of the wireless channel in the i-th round of transmission is obtained, which is represented as:

[0080]

[0081] wherein, represents the wireless channel modeled as an additive white Gaussian noise channel or a slow fading channel; W i is the additive white Gaussian noise in the wireless channel, W i each element of which independently obeys a complex Gaussian distribution with mean 0 and variance , i.e. η represents the channel gain, η obeys a complex Gaussian distribution with mean 0 and variance , i.e. and η remains unchanged in the transmission process of a single image but varies independently between different images. Specifically, for static AWGN, let η = 1, and the signal-to-noise ratio (SNR) is defined as in dB.

[0082] In one embodiment, the outputs of the wireless channel in the historical rounds of transmission are cached through a channel caching structure, and are spliced to obtain the input of the decoder in the current round of transmission, the decoder is used for semantic decoding to obtain a preliminary semantic estimation result, including:

[0083] First, the output Y i of the wireless channel in the i-th round of transmission is received through the channel caching structure, and the outputs of the wireless channel in the historical rounds of transmission cached by itself are spliced to obtain the input chbuffer,i , expressed as:

[0084] ch buffer,i =[Y1,Y2,…,Y i ];

[0085] Among them, i>1.

[0086] Secondly, use the decoder to ch buffer,i Perform semantic decoding to obtain the initial semantic estimation results of the decoder output in the i-th round of transmission Expressed as:

[0087]

[0088] in, Denotes the decoder of the i-th round transmission, and β is the learnable parameter of the decoder. The structure of the decoder is as follows Figure 4 As shown, the decoding process is also stacked by multiple stages, each stage includes an image block division (PatchDivision) layer, followed by N i A series of Swin Transformer blocks are used to decode and reconstruct the estimated semantic representation. Figure 3 and Figure 4 Middle C i Indicates the channel dimension of the i-th recording stage, that is, the number of channels of the current layer feature and the representation dimension of each image block feature.

[0089] In one embodiment, the reconstructed semantic image output from the previous round of transmission and the preliminary semantic estimation result output from the decoder in the current round of transmission are input into a semantic fusion module for fusion to obtain the reconstructed semantic image output from the current round of transmission, including:

[0090] The reconstructed semantic image of the i-1th round output cached in the decoder cache module The initial semantic estimation result of the decoder output in the i-th round Input the semantic fusion module for fusion and obtain the reconstructed semantic image output by the i-th round of transmission, which is expressed as:

[0091]

[0092] Among them, Combiner represents the semantic fusion module, and the structure of the semantic fusion module is as follows: Figure 5 As shown in the figure, it consists of a feature concatenation layer, a first linear projection layer, and a second linear projection layer connected in sequence, and the two linear projection layers are connected by an activation function ReLU. This module aims to gradually refine and enhance the semantic representation through retransmission between rounds.

[0093] In one of the embodiments, the method further comprises:

[0094] After each round of transmission, the automatic retransmission request feedback signal fed back by the receiving end is acquired, if the signal is NACK (negative acknowledgement), the sending end is informed to continue retransmission, and the reconstructed semantic image output by the decoder cache module in the current round of transmission is transmitted back to the sending end through the feedback channel to perform the next round of transmission; if the signal is ACK (acknowledgement), the retransmission is terminated, and the information cached in the channel cache structure and the decoder cache module is emptied.

[0095] Further, the workflow of the semantic communication framework based on the progressive channel feedback constructed by the present application comprises the following steps:

[0096] Initialization: transmission round number indicator: i = 1; maximum retransmission number: N max = 3; ACK indicator: ζ = 0; channel cache structure and decoder cache module are initialized to be empty.

[0097] Input: image vector sequence S s .

[0098] Output: reconstructed semantic image

[0099] The framework starts working:

[0100] 1. According to , the semantic feature representation X1 output by the encoder in the first round of transmission is calculated;

[0101] 2. The power normalized X1 is transmitted through the wireless channel to obtain the output Y1 of the wireless channel in the first round of transmission;

[0102] 3. The Y1 is semantically decoded by the decoder to obtain the preliminary semantic estimation result output by the decoder in the first round of transmission, which is taken as the reconstructed semantic image in the first round of transmission;

[0103] 4. When i ≤ N max , the following loop is executed:

[0104] 4.1. The reconstructed semantic image output in the previous round of transmission is transmitted back through the feedback channel to obtain

[0105] 4.2. The fusion module and the encoder are utilized to generate the semantic feature representation X i output by the encoder in the i-th round of transmission;

[0106] 4.3. The power normalized X iand concatenate the outputs of the wireless channel buffered in the channel buffer structure for i rounds [Y1, Y2, …, Y i ] into ch buffer,i ;

[0107] 4.4, semantic decoding of ch buffer,i using the decoder to obtain the preliminary semantic estimation result of the i-th round transmission output by the decoder

[0108] 4.5, inputting the preliminary semantic estimation result of the i-th round transmission output and the reconstructed semantic image of the i-1-th round transmission output buffered in the decoder buffer module into the semantic fusion module for fusion to obtain the reconstructed semantic image of the i-th round transmission output

[0109] 4.6, updating the decoder buffer module using

[0110] 4.7, increasing i by 1 and updating the ACK indicator ζ;

[0111] 5, if i > N max or ζ = 1 (i.e. an ACK is received), the loop is exited.

[0112] In one embodiment, a semantic communication device based on progressive channel feedback is provided, comprising:

[0113] a framework construction unit configured to construct a semantic communication framework based on progressive channel feedback, the framework comprising a sending end, a wireless channel, a feedback channel and a receiving end; wherein the sending end comprises a fusion module and an encoder, and the receiving end comprises a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module;

[0114] a semantic extraction unit configured to, at the sending end, obtain an image vector sequence input for the current round of transmission and a reconstructed semantic image of the previous round of transmission output transmitted back through the feedback channel, and input the image vector sequence and the reconstructed semantic image into the fusion module for feature fusion to obtain the input of the encoder for the current round of transmission, and send the semantic features extracted by the encoder to the wireless channel for transmission;

[0115] a semantic reconstruction unit configured to, at the receiving end, buffer the outputs of the wireless channel in historical rounds of transmission through the channel buffer structure, concatenate the outputs to obtain the input of the decoder for the current round of transmission, perform semantic decoding using the decoder to obtain a preliminary semantic estimation result, input the reconstructed semantic image of the previous round of transmission output and the preliminary semantic estimation result of the current round of transmission output into the semantic fusion module for fusion to obtain the reconstructed semantic image of the current round of transmission output, and complete the current round of transmission; wherein the reconstructed semantic image of each round of transmission is buffered in the decoder buffer module.

[0116] ​​The specific limitation of the semantic communication device based on progressive channel feedback can refer to the limitation of the semantic communication method based on progressive channel feedback in the above, which will not be described here. Each module in the above semantic communication device based on progressive channel feedback can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in the form of software, so that the processor calls to execute the operation corresponding to each module.

[0117] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 6 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a semantic communication method based on progressive channel feedback. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0118] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0119] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the following steps:

[0120] A semantic communication framework based on progressive channel feedback is constructed, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end. The sending end includes a fusion module and an encoder, and the receiving end includes a decoder, a channel buffer structure, a decoder buffer module and a semantic fusion module.

[0121] At the sending end, the image vector sequence input in the current transmission and the reconstructed semantic image output in the previous transmission fed back through the feedback channel are input into the fusion module for feature fusion to obtain the input of the encoder in the current transmission, and the semantic features extracted by the encoder are sent to the wireless channel for transmission.

[0122] At the receiving end, the outputs of the wireless channel in the historical several rounds of transmission are cached through the channel cache structure, and the input of the decoder in the current transmission is obtained by splicing, the semantic decoding is performed by using the decoder to obtain the preliminary semantic estimation result, the reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output by the decoder in the current transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission, and the current transmission is completed; wherein the reconstructed semantic image output in each round of transmission is cached in the decoder cache module.

[0123] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps:

[0124] A semantic communication framework based on progressive channel feedback is constructed, and the framework is composed of a sending end, a wireless channel, a feedback channel and a receiving end; wherein the sending end includes a fusion module and an encoder, and the receiving end includes a decoder, a channel cache structure, a decoder cache module and a semantic fusion module;

[0125] At the sending end, the image vector sequence input in the current transmission and the reconstructed semantic image output in the previous transmission fed back through the feedback channel are input into the fusion module for feature fusion to obtain the input of the encoder in the current transmission, and the semantic features extracted by the encoder are sent to the wireless channel for transmission.

[0126] At the receiving end, the outputs of the wireless channel in the historical several rounds of transmission are cached through the channel cache structure, and the input of the decoder in the current transmission is obtained by splicing, the semantic decoding is performed by using the decoder to obtain the preliminary semantic estimation result, the reconstructed semantic image output in the previous transmission and the preliminary semantic estimation result output by the decoder in the current transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output in the current transmission, and the current transmission is completed; wherein the reconstructed semantic image output in each round of transmission is cached in the decoder cache module.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0128] Any combination of the technical features of the above embodiments can be made, and in order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0129] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the protection scope of the present application.

Claims

1. A semantic communication method based on progressive channel feedback, characterized in that: The method comprises: A semantic communication framework based on progressive channel feedback is constructed, comprising a transmitter, a wireless channel, a feedback channel, and a receiver; wherein the transmitter comprises a fusion module and an encoder, and the receiver comprises a decoder, a channel buffer structure, a decoder buffer module, and a semantic fusion module; At the transmitting end, the image vector sequence input in the current round of transmission and the reconstructed semantic image output from the previous round of transmission transmitted back through the feedback channel are obtained and input into the fusion module for feature fusion. This obtains the input of the encoder in the current round of transmission, and the semantic features extracted by the encoder are sent to the wireless channel for transmission. At the receiving end, the output of the wireless channel in several historical transmission rounds is cached through the channel cache structure, and the decoder's input for the current transmission round is spliced ​​together. The decoder performs semantic decoding to obtain a preliminary semantic estimation result. The reconstructed semantic image output from the previous transmission round and the preliminary semantic estimation result output from the decoder in the current transmission round are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output from the current transmission round, completing the current transmission round. The reconstructed semantic image output from each transmission round is cached in the decoder cache module.

2. The semantic communication method based on progressive channel feedback according to claim 1, characterized in that Get the image vector sequence of this round of transmission input, including: Before each round of transmission begins, obtain the source image Divide it into a grid consisting of l1=p×p non-overlapping image blocks, regard each image block as a token, and arrange all tokens in order from left to right and from top to bottom to obtain the image vector sequence of each round of transmission input Among them, h, w and 3 represent the height, width and RGB color channels of the source image respectively. is a set of real numbers; the hyperparameter p is used to control the granularity of the source image division; token is defined as the embedding vector of the image block, x i Represents the i-th token in the image vector sequence.

3. The semantic communication method based on progressive channel feedback according to claim 2, characterized in that The image vector sequence of the input of this round of transmission and the reconstructed semantic image output of the previous round of transmission returned through the feedback channel are obtained and input into the fusion module for feature fusion to obtain the input of the encoder in this round of transmission, including: Get the reconstructed semantic image of the i-1th round of transmission output returned through the feedback channel Expressed as: in, represents the reconstructed semantic image output from the i-1th round of transmission; H f represents the feedback channel modeled as an ideal signal or as an additive white Gaussian noise feedback channel; W f is the additive white Gaussian noise in the feedback channel, and W f Each element in the equation has a mean of 0 and a variance of The complex Gaussian distribution of in W f The i-th element in ; Will The image vector sequence S of the i-th round transmission input s Input to the fusion module, in which first Embedded map to S s The feature space with the same token feature dimension is used to align the features in the spatial dimension, and then the aligned features are aligned. With S s After concatenation in the channel dimension and fusion through a linear projection layer, the encoder input S transmitted in the i-th round is generated. ini , expressed as: Among them, f e Represents embedding mapping processing, Concat represents feature concatenation, and Linear represents the linear projection layer.

4. The semantic communication method based on progressive channel feedback according to claim 3, characterized in that The semantic features extracted by the encoder are sent to the wireless channel for transmission, including: Use the encoder to transmit the input S in the i-th round ini Perform semantic feature extraction to obtain the semantic feature representation X output by the encoder in the i-th round of transmission i , the expression is: in, represents the encoder of the i-th round transmission, α is the learnable parameter of the encoder; The power-normalized X i , sent to the wireless channel for transmission, and the output Y of the wireless channel in the i-th round transmission is obtained i , expressed as: in, represents the wireless channel modeled as an additive white Gaussian noise channel or a slow fading channel; W i is the additive white Gaussian noise in the wireless channel, W i The elements in it are all independent and have a mean of 0 and a variance of The complex Gaussian distribution of η represents the channel gain and η has a mean of 0 and a variance of The complex Gaussian distribution of 5. The semantic communication method based on progressive channel feedback according to claim 4, characterized in that: The channel buffer structure caches the output of the wireless channel in several historical transmission rounds and concatenates them to obtain the input of the decoder in the current transmission round. The decoder performs semantic decoding to obtain preliminary semantic estimation results, including: The output Y transmitted by the wireless channel in round i is received through the channel buffer structure i , and concatenate it with the output of the wireless channel cached in several historical rounds of transmission to obtain the input ch of the decoder in the i-th round of transmission buffer,i , expressed as: ch buffer,i =[Y1,Y2,…,Y i ]; Where i>1; Use the decoder to ch buffer,i Perform semantic decoding to obtain the initial semantic estimation results of the decoder output in the i-th round of transmission Expressed as: in, represents the decoder of the i-th round transmission, and β is the learnable parameter of the decoder.

6. The semantic communication method based on progressive channel feedback according to claim 5, characterized in that: The reconstructed semantic image output from the previous round of transmission and the preliminary semantic estimation result output from the decoder in this round of transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output from this round of transmission, including: The reconstructed semantic image of the i-1th round output cached in the decoder cache module The initial semantic estimation result of the decoder output in the i-th round Input the semantic fusion module for fusion and obtain the reconstructed semantic image output by this round of transmission, which is expressed as: Among them, Combiner represents the semantic fusion module, which consists of a feature concatenation layer, a first linear projection layer, and a second linear projection layer connected in sequence, and the two linear projection layers are connected by the activation function ReLU.

7. The semantic communication method based on progressive channel feedback according to any one of claims 1 to 6, characterized in that: The method further comprises: After each round of transmission, the automatic retransmission request feedback signal from the receiving end is obtained. If the signal is NACK, the sending end is notified to continue retransmission, and the reconstructed semantic image output by the current round of transmission cached by the decoder cache module is transmitted back to the sending end through the feedback channel for the next round of transmission; if the signal is ACK, the retransmission is terminated, and the information cached in the channel cache structure and the decoder cache module is cleared.

8. A semantic communication device based on progressive channel feedback, characterized in that: The device comprises: A framework construction unit, configured to construct a semantic communication framework based on progressive channel feedback, the framework comprising a transmitter, a wireless channel, a feedback channel, and a receiver; wherein the transmitter comprises a fusion module and an encoder, and the receiver comprises a decoder, a channel buffer structure, a decoder buffer module, and a semantic fusion module; The semantic extraction unit is used to obtain the image vector sequence input in the current round of transmission and the reconstructed semantic image output in the previous round of transmission through the feedback channel at the transmitting end, and input them into the fusion module for feature fusion to obtain the input of the encoder in the current round of transmission. The semantic features extracted by the encoder are then sent to the wireless channel for transmission; The semantic reconstruction unit is used to cache the output of the wireless channel in several historical rounds of transmission through the channel cache structure at the receiving end, and splice it to obtain the input of the decoder in the current round of transmission, use the decoder to perform semantic decoding, and obtain preliminary semantic estimation results; the reconstructed semantic image output by the previous round of transmission and the preliminary semantic estimation result output by the decoder in the current round of transmission are input into the semantic fusion module for fusion to obtain the reconstructed semantic image output by the current round of transmission, thereby completing the current round of transmission; wherein, the reconstructed semantic image output by each round of transmission is cached in the decoder cache module.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images