Channel environment adaptive sampling-semantic-channel coding joint optimization method and system

Abstract

The invention provides a channel environment adaptive sampling-semantic-channel coding joint optimization method and system, and relates to the technical field of semantic communication, a sending end performs semantic sampling on an image needing to be transmitted to obtain a semantic sampling result; the sending end inputs the semantic sampling result into an encoder to obtain an image feature vector needing to be transmitted; a receiving end receives the image feature vector influenced by the physical channel noise and preprocesses the image feature vector; the receiving end inputs the preprocessing result into a decoder to obtain an image feature vector needing semantic reconstruction; and the receiving end performs semantic reconstruction according to the image feature vector needing semantic reconstruction to obtain a reconstructed image. According to the method provided by the invention, the characteristics of an information source and a channel are combined, image transmission in a variable signal-to-noise ratio environment can be supported while semantic sampling and reconstruction are considered, and the reconstruction quality of the image is ensured.

Description

Technical Field

[0001] The present application relates to the field of semantic communication technology, and in particular to a sampling-semantics-channel coding joint optimization method and system adaptive to channel environment. Background Art

[0002] As communication needs increase, existing communication transmission methods need to meet the high requirements of low sensing cost and storage and computing overhead and adapt to dynamic transmission environments.

[0003] However, the cliff effect of the proposed independent transmission scheme under low signal-to-noise ratio causes the performance to deteriorate sharply, resulting in a sudden and significant decrease in the performance and quality of the transmitted image at the receiving end. In addition, the existing semantic communication system and sensor system are relatively independent in design, and the integrated joint design of sampling and communication at the semantic level is not considered.

[0004] In addition, the semantic communication joint encoding and decoding architecture based on single signal-to-noise ratio training cannot overcome the performance loss caused by the mismatch between the training and test channel environments, and it is difficult to support image transmission in an environment with changing signal-to-noise ratios.

[0005] Based on this, a new method integrating sampling and communication is urgently needed. Summary of the invention

[0006] The present application provides a channel environment adaptive sampling-semantics-channel coding joint optimization method and system to solve the problem that the existing transmission method does not consider the sampling-communication integration at the semantic level and is difficult to support image transmission in an environment with variable signal-to-noise ratio.

[0007] In a first aspect of an embodiment of the present application, a sampling-semantics-channel coding joint optimization method for channel environment adaptation is proposed, the method comprising: The sending end performs semantic sampling on the image to be transmitted to obtain a semantic sampling result; The transmitting end inputs the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is the signal-to-noise ratio and the intermediate feature output by the previous residual convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; The receiving end receives the image feature vector affected by the physical channel noise and performs preprocessing; The receiving end inputs the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction. The decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded. The input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment. The receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

[0008] Optionally, the sender performs semantic sampling on the image to be transmitted to obtain a semantic sampling result, including: The sending end normalizes the image to be transmitted and then divides it into blocks to obtain multiple image blocks; The sending end inputs the image to be transmitted and the signal-to-noise ratio into the semantic scanning network to obtain a semantic information distribution map; The sending end adopts a block sampling ratio aggregation correction algorithm to obtain a semantic sampling ratio distribution map according to the semantic information distribution map and the signal-to-noise ratio; The sending end performs singular value decomposition on the training image set to obtain the initial matrix; For an i-th image block among the multiple image blocks, based on a given overall sampling rate and the semantic information distribution map, obtaining a semantic sampling ratio of the i-th image block; Based on the semantic sampling ratio of the i-th image block and the initial matrix, obtaining a semantic sampling matrix of the i-th image block; Based on the i-th image block and the semantic sampling matrix of the i-th image block, semantic sampling is performed to obtain an i-th initial semantic sampling result; Based on the i-th initial semantic sampling result and the transposed semantic sampling matrix of the i-th image block, block initialization sampling rate alignment is performed to obtain a final semantic sampling result.

[0009] Optionally, the semantic scanning network is composed of a first convolutional layer, a first channel environment adaptation module, multiple simple residual blocks, a second channel environment adaptation module and a sequential cascade of convolutional layers; the semantic scanning network is used to evaluate the significance of semantic information at each position on the image to be transmitted.

[0010] Optionally, the sending end adopts a block sampling ratio aggregation correction algorithm to obtain a semantic sampling ratio distribution map according to the semantic information distribution map and the signal-to-noise ratio, including: According to the order from left to right and from top to bottom, the semantic information distribution map is divided into B × B of l image blocks; right lThe image blocks are aggregated to obtain the semantic sampling ratio distribution map of the block sampling rate including the distribution of all image blocks. 。

[0011] Optionally, the receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image, including: The receiving end expands the image feature vector that needs semantic reconstruction into l image blocks, perform random transformation enhancement on each image block, and then perform block gradient descent to obtain the intermediate result of each image block, and summarize the intermediate results of all image blocks; Receiving a semantic sampling ratio distribution map sent by a transmitting end, and using corresponding sampling ratios in the semantic sampling ratio distribution map as filling elements to obtain an extended semantic sampling ratio distribution map; By means of a semantic extraction network oriented to the semantic sampling ratio distribution map and the signal-to-noise ratio, the features of the extended semantic sampling ratio distribution map are embedded into the feature space, and a feature map based on semantics and signal-to-noise ratio extraction is output; Connecting the intermediate result of each image block with the feature map extracted based on semantics and signal-to-noise ratio, and processing the connection result based on the proximal mapping network of deep learning; The processing result is added to the intermediate result of each image block to obtain the reconstructed image data, and then denormalized to obtain the reconstructed image.

[0012] Optionally, the channel environment adaptation module includes an average pooling layer, a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence; the channel environment adaptation module reprocesses the input features based on a known signal-to-noise ratio, and the specific process is as follows: Performing an average pooling operation on the intermediate features output by the previous module connected to the channel environment adaptation module to obtain an average pooling result; Splicing the average pooling result with the signal-to-noise ratio to obtain a splicing feature; The concatenated features are processed by a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer to obtain a channel environment adaptive scaling factor; According to the channel environment adaptive scaling factor and the intermediate feature output by the previous module connected to the channel environment adaptive module, the intermediate feature for adaptively adjusting the signal-to-noise ratio of the current channel environment is obtained.

[0013] Optionally, the channel environment adaptive sampling-semantics-channel coding joint optimization method is implemented by a channel environment adaptive sampling-semantics-channel coding joint model, and the training process includes: Calculating a loss between the reconstructed image and a corresponding image in the training image set; Based on the loss, the parameters in the semantic sampling module, encoder, decoder and semantic reconstruction module are optimized to obtain the sampling-semantic-channel coding joint model adaptive to the channel environment.

[0014] In a second aspect of an embodiment of the present application, a channel environment adaptive sampling-semantics-channel coding joint optimization system is proposed, the system comprising: A semantic sampling module is used to perform semantic sampling on the image to be transmitted to obtain a semantic sampling result; An encoding module, used for inputting the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is a signal-to-noise ratio and an intermediate feature output by a previous residual convolution module, and the output is an intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; A preprocessing module, used for receiving and preprocessing image feature vectors affected by physical channel noise; A decoding module, used for inputting the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction, wherein the decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; The semantic reconstruction module is used to perform semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

[0015] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the channel environment adaptive sampling-semantics-channel coding joint optimization method described in any one of the first aspects above.

[0016] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the sampling-semantics-channel coding joint optimization method for channel environment adaptation described in any one of the first aspects above is implemented.

[0017] The present application has the following advantages: The present application provides a sampling-semantics-channel coding joint optimization method and system for channel environment adaptation, wherein a transmitting end performs semantic sampling on an image to be transmitted to obtain a semantic sampling result; the transmitting end inputs the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is a signal-to-noise ratio and an intermediate feature output by a previous residual convolution module, and the output is an intermediate feature for adapting to the signal-to-noise ratio of the current channel environment; the receiving end receives the image feature vector affected by physical channel noise and performs preprocessing; the receiving end inputs the preprocessing result into a decoder to obtain an image feature vector for semantic reconstruction, wherein the decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the decoder is a signal-to-noise ratio and an intermediate feature output by a previous residual transposed convolution module, and the output is an intermediate feature for adapting to the signal-to-noise ratio of the current channel environment; the receiving end performs semantic reconstruction according to the image feature vector for semantic reconstruction to obtain a reconstructed image. In the method proposed in this application, a channel environment adaptation module based on the attention mechanism is added to the end-to-end joint design of semantic-based sampling and reconstruction, encoding and decoding, which further combines the characteristics of the source and the channel. While considering semantic sampling and reconstruction, it can support image transmission in an environment with varying signal-to-noise ratio and ensure the reconstruction quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a schematic diagram of the steps of a sampling-semantics-channel coding joint optimization method for channel environment adaptation provided by an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a sampling-semantics-channel coding joint optimization method for channel environment adaptation proposed in an embodiment of the present application; Figure 3 It is a schematic diagram of a flow chart of semantic sampling of an image by a semantic sampling module proposed in an embodiment of the present application; Figure 4 It is a schematic diagram of a flow chart of encoding and decoding by a semantic-channel-encoder-decoder proposed in an embodiment of the present application; Figure 5It is a schematic diagram of a flow chart of semantic reconstruction of an image by a semantic reconstruction module provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the architecture of a channel environment adaptation module provided in an embodiment of the present application; Figure 7 It is an architecture diagram of a sampling-semantics-channel coding joint optimization system for channel environment adaptation proposed in an embodiment of the present application; Figure 8 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] As communication needs increase, existing communication transmission methods need to meet the high requirements of low sensing cost and storage and computing overhead and adapt to dynamic transmission environments.

[0022] However, the cliff effect of the proposed independent transmission scheme under low signal-to-noise ratio causes the performance to deteriorate sharply. That is, when the signal-to-noise ratio is lower than a certain level, the number of error codewords in the transmitted coded image greatly exceeds the error correction capability of the error control codec, and the performance and quality of the transmitted image at the receiving end will suddenly drop significantly. In addition, the existing semantic communication system and sensor system are relatively independent in design, and the integrated joint design of sampling and communication at the semantic level is not considered.

[0023] In addition, the semantic communication joint encoding and decoding architecture based on single signal-to-noise ratio training cannot overcome the performance loss caused by the mismatch between the training and test channel environments, and it is difficult to support image transmission in an environment with changing signal-to-noise ratios.

[0024] Based on this, the present application proposes a transmission method that adapts to the channel environment and integrates sampling and communication.

[0025] In a first aspect of an embodiment of the present application, a sampling-semantics-channel coding joint optimization method for channel environment adaptation is provided. Figure 1 , Figure 1 1 is a schematic diagram of the steps of a sampling-semantics-channel coding joint optimization method for channel environment adaptation provided by an embodiment of the present application, the method comprising the following steps: Step 101: The sending end performs semantic sampling on the image to be transmitted to obtain a semantic sampling result; Step 102: The transmitter inputs the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is the signal-to-noise ratio and the intermediate feature output by the previous residual convolution module, and the output is the intermediate feature adapted to the signal-to-noise ratio of the current channel environment; Step 103: The receiving end receives the image feature vector affected by the physical channel noise and performs preprocessing; Step 104: The receiving end inputs the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction. The decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded. The input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment. Step 105: The receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

[0026] In order to clearly describe the channel environment adaptive sampling-semantics-channel coding joint optimization method proposed in this application, combined with Figure 2 The method is described in detail. Figure 2 It is a flow chart of a channel environment adaptive sampling-semantics-channel coding joint optimization method proposed in an embodiment of the present application.

[0027] When the step 101 is specifically implemented, the above method is applied to an end-to-end image transmission system, that is, from the transmitting end to the receiving end, the transmitting end can be the source end, and the receiving end can be the destination end. Considering that there is no semantic sampling and semantic reconstruction module in the semantic communication system of the joint source channel coding and decoding currently proposed, it is very important to perform downsampling processing on the objects to be transmitted based on a certain strategy in some scenarios where it is necessary to reduce the number of sensing times and storage and computing overhead, such as multiple medical imaging is harmful to the health of patients, and some sensors such as satellite imaging are expensive when working. Excessive amounts of data and models will increase network storage and computing burdens. In order to avoid the above situation, the present application sets a semantic sampling module based on downsampling at the transmitting end, performs semantic sampling on the images to be transmitted, and obtains semantic sampling results; semantic-based downsampling processing is conducive to ensuring that the image content with high semantic importance or rich semantic information is accurately extracted and encoded in the subsequent processing, and other redundant parts are reduced or even ignored, thereby reducing the source level and further reducing the transmission overhead of communication.

[0028] In an optional embodiment of the present application, see Figure 3 , Figure 3It is a flow chart of semantic sampling of an image by a semantic sampling module proposed in an embodiment of the present application. When semantic sampling is performed on an image, when the overall average sampling rate of the entire image is determined, the image is divided into blocks to obtain multiple image blocks. The sampling rate allocation for each image block in each image is completed by semantic information and signal-to-noise ratio, that is, the sampling number of each block is based on a given overall average sampling number, and is determined jointly by semantic information scanning and block sampling rate aggregation correction.

[0029] In an optional embodiment of the present application, the semantic scanning network is composed of a first convolutional layer, a first channel environment adaptation module, a plurality of simple residual blocks, a second channel environment adaptation module and a sequential cascade of convolutional layers; the semantic scanning network is used to evaluate the significance of the semantic information of each position on the image to be transmitted, and highlight the importance of different areas in the image. Due to the attention focus of the channel environment adaptation module on the channel signal-to-noise ratio, the semantic scanning network comprehensively considers the channel conditions, can adapt to the signal-to-noise ratio of the current channel environment, and achieves more accurate allocation in conjunction with the characteristics of the source channel. Furthermore, in order to achieve accurate sampling ratio allocation for each block, the present application proposes to process the semantic information distribution map and the signal-to-noise ratio through a block sampling ratio aggregation correction algorithm to obtain a semantic sampling ratio allocation map, and share the semantic sampling ratio allocation map with the reconstruction end. At the same time, based on the given overall sampling rate, the number of sampling times for each block after image segmentation is adjusted and allocated.

[0030] In an optional embodiment of the present application, the sending end performs semantic sampling on the image to be transmitted to obtain a semantic sampling result, which specifically includes the following process: first, the sending end normalizes the image to be transmitted, and then divides the normalized image S into blocks to obtain multiple image blocks. The image to be transmitted is , n is the image size, is a set of real numbers, H is the image height, W is the image width, and C is the number of image channels, that is, n=H×W×C. The set of all image blocks can be expressed as , l is the number of image blocks, which is an integer greater than or equal to 2. Then the sender sends the above image to be transmitted and the signal-to-noise ratio μ Input to the semantic scanning network D In the above example, we get the semantic information distribution map M The sending end adopts the block sampling ratio aggregation correction algorithm according to the semantic information distribution map M and the signal-to-noise ratio μ , get the semantic sampling ratio distribution map R ; The sending end is training image set Perform singular value decomposition (SVD) ( is the number of images in the training image set), and the initial matrix A is obtained. The size of the initial matrix A is N×N, where N is the dimension of its feature basis vector. The features of the initial matrix A are based on the decreasing importance from the first row to the last row, that is, the generated matrix can be obtained by purposefully cropping the sampling matrix of any sampling rate, and can be learned from the data. For example, if image block A has more details than image block B, then the importance of the image block is greater, and the subsequent semantic sampling matrix retains more rows in the initial matrix A, that is, more information is collected.

[0031] Furthermore, based on the given overall sampling rate r and the above semantic information distribution map M, the semantic sampling ratio of the i-th image block is obtained: ; Semantic sampling ratio based on the i-th image block And the above initial matrix A, get the semantic sampling matrix of the i-th image block ; The sending end is based on the semantic sampling matrix of the i-th image block and the i-th image block , perform semantic sampling and obtain the i-th initial semantic sampling result , the initial semantic sampling result set of each image block can be expressed as ; Based on the i-th initial semantic sampling result and the semantic sampling matrix of the i-th image block after transposition , perform block initialization sampling rate alignment to obtain the final semantic sampling result , the semantic sampling result set of each image block can be expressed as .

[0032] In the process of sampling the image through the semantic sampling module, the parameter of the semantic sampling module is ξ, and the process of semantic sampling can be expressed as: in, (·) is the semantic sampling module, s is the image to be sampled, μ is the signal-to-noise ratio, and r is the given overall sampling rate.

[0033] In an optional embodiment of the present application, the sending end adopts a block sampling ratio aggregation correction algorithm, according to the semantic information distribution map M and the signal-to-noise ratio μ , and obtain the semantic sampling ratio distribution map R, which specifically includes the following method: in order from left to right and from top to bottom, divide the above semantic information distribution map M into B × B of l image blocks, where the number of image blocks l is the same as the number of image blocks l obtained by dividing the normalized image S into blocks; l The image blocks are aggregated to obtain the block sampling rate including all block allocation conditions. The above semantic sampling ratio distribution graph R.

[0034] In an optional embodiment of the present application, in the actual semantic sampling process, due to the limited size of the semantic sampling matrix, the available sampling rate of each block can only be Select from q i Expressed as i The block sampling size is the exact sampling size of the block, and N is the dimension of the eigenvalue vector of the initial matrix A. The block sampling ratio aggregation correction algorithm can be summarized as softmax normalization, sumpooling aggregation and error correction. The semantic sampling ratio distribution graph obtained by the block sampling ratio aggregation correction algorithm is as follows: First, softmax normalization is applied to the semantic information distribution graph M, and B × B Pooling obtains an aggregated weight map, which is combined with the target measurement size and ql ( ql represents the sampling size of l image blocks) and multiply them to get the measured size mapping Q , then Q Perform iterative verification and correction. In each correction iteration, Q Cut and discretize it so that all its elements are [0, N ] specifies the upper limit of the image block measurement size, and finally obtains the semantic sampling ratio distribution map R.

[0035] When step 102 is specifically implemented, the semantic sampling result obtained by the semantic sampling module is input into the encoder in the semantic-channel encoder-decoder to obtain the image feature vector to be transmitted. The process can be referred to in Figure 4 , Figure 4 1 is a flow chart of a semantic-channel-encoder-decoder for encoding and decoding proposed in an embodiment of the present application. The parameter of the semantic channel-encoder is θ, and the encoding process can be expressed as: in, (·) is the semantic channel encoder, and x is the semantic sampling result.

[0036] After the semantic sampling result x is encoded by the encoder, it is reshaped to generate a vector of size 1×k. Before the obtained image feature vector to be transmitted is output, in order to meet the average power constraint, that is, the following conditions are met:

[0037] The power normalization formula is used to normalize the vector to obtain the coding symbol z (corresponding to the image feature vector). The calculation formula is as follows: , in, for The complex conjugate of The output of the last submodule of the encoder before power normalization. This application sets the normalized power = 1. The coded symbol z is then input into the physical channel for transmission.

[0038] In an optional embodiment of the present application, the encoder is composed of a plurality of residual convolution modules (RCM) and a plurality of channel environment adaptation modules (ACAM) alternately cascaded, and the input of each channel environment adaptation module (ACAM) in the encoder is a signal-to-noise ratio μ (corresponding to the SNR in the figure) and the intermediate features output by the previous residual convolution module RCM, the output is the intermediate features that are adaptive to the signal-to-noise ratio of the current channel environment. In an optional embodiment of the present application, by changing the output channel size of the last layer of convolution of the last residual convolution module RCM in the encoder k , models with different bandwidth ratios CBR can be obtained so that the encoder can better adapt to changes in the channel environment.

[0039] When step 103 is specifically implemented, the encoding symbol z is obtained by the function (·) represents the transmission of a noisy physical channel. Let the channel gain be 1. The channel type is an AWGN channel or a Rayleigh fading channel. After being affected by noise, the channel output is expressed as , the AWGN channel transmission model can be expressed as: Among them, n For noise.

[0040] That is, the image feature vector received by the receiving end is an image feature vector affected by physical channel noise, and the received image feature vector needs to be preprocessed before being transmitted to the decoder for decoding. The preprocessing includes padding and reshaping.

[0041] When step 104 is specifically implemented, the transmitter inputs the preprocessing result into the decoder, and decodes it in combination with the signal-to-noise ratio to obtain the image feature vector that needs semantic reconstruction. The decoder parameters are , the decoding process can be expressed as: in, (·) is the semantic channel-decoder.

[0042] See also Figure 4The above decoder is composed of multiple residual transposed convolutional modules RTCM (Residual Transposed Convolutional Module) and multiple channel environment adaptation modules ACAM alternately cascaded. The input of each channel environment adaptation module ACAM in the decoder is the signal-to-noise ratio μ The intermediate features output by the previous residual transposed convolution module RTCM are output as intermediate features that are adaptive to the signal-to-noise ratio of the current channel environment. Figure 4 The ↓↑ symbols on each residual convolution module or residual transposed convolution module in represent upsampling and downsampling operations, respectively, and the | symbol represents the interval. The corresponding numbers on the module are explained by taking the numbers on the first residual convolution module RCM in the encoder as an example. "9×9×256" represents the width W, height H, and number of output channels C of the convolution kernel used by the module.

[0043] In an optional embodiment of the present application, the encoder (Encoder) in the above-mentioned semantic-channel-encoder-decoder is composed of 5 residual convolution modules RCM and 5 channel environment adaptation modules ACAM alternately cascaded, and the decoder (Decoder) is composed of 5 residual transposed convolution modules RTCM and 5 channel environment adaptation modules ACAM alternately cascaded.

[0044] When the step 105 is specifically implemented, the receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image. In the embodiment of the present application, the semantic reconstruction is implemented by a semantic reconstruction module, which performs semantic reconstruction based on the decoding result, signal-to-noise ratio, and semantic sampling ratio distribution map R shared by the transceiver, and then performs denormalization to obtain a reconstructed image. The above semantic reconstruction process can be expressed as: in, (·) is the semantic reconstruction module, and the parameter of the semantic reconstruction module is ζ.

[0045] The semantic reconstruction process of the semantic reconstruction module can be found in Figure 5 , Figure 5 The flowchart of a semantic reconstruction module provided in the embodiment of the present application for semantic reconstruction of an image is shown in FIG. Specifically, the receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image, including the following process: First, the receiving end converts the image feature vector that needs semantic reconstruction into the image feature vector. Expand to l image blocks, the set of all expanded image blocks can be expressed as , It means that the processed image block is the result of the previous iteration round. Each image block is randomly transformed and enhanced, and then the block gradient is descended to obtain the intermediate result of each image block, which can be expressed as , and summarize the intermediate results of all image blocks, which can be expressed as , the intermediate results of all image blocks are sent to the next stage for joint recovery. Among them, random transformation enhancement for each image block refers to: random upside down flipping, left-right flipping, scaling, random brightness, contrast, hue, multi-image splicing and other enhancements according to proportion. The random transformation enhancement method can better solve the bottleneck between each two adjacent iteration rounds. The block gradient descent processing of the image block after random transformation enhancement is carried out using the following formula:

[0046] Among them, k represents the round of the current iteration, and the total number of iteration rounds is N p , is the iteration step length of the kth round, is the semantic sampling matrix of the i-th image block, T represents the transposition of the semantic sampling matrix of the i-th image block, is the i-th initial semantic sampling result , is the i-th image block expanded in the k-1th round.

[0047] The intermediate result obtained by folding the i-th image block , the i-th image block expanded in the k-th round is obtained according to the following formula: : .

[0048] This formula is the process of approximate point mapping, where is the semantic sampling result of the i-th image block, is the regular term and its proportional factor, argmin represents the parameter value at which a function achieves the minimum value in its domain.

[0049] Then, the semantic sampling ratio distribution map R sent by the sender is received, and the corresponding sampling ratio in the semantic sampling ratio distribution map R is used as a filling element to obtain the extended semantic sampling ratio distribution map ; Furthermore, through the semantic extraction network oriented to the semantic sampling ratio distribution map and the signal-to-noise ratio (i.e., the SNR of the input in the map) , the extended semantic sampling ratio distribution map The features of are embedded into the feature space, and the feature map extracted based on semantics and signal-to-noise ratio is output. ; The intermediate results of each image block And the feature map extracted based on semantics and signal-to-noise ratio Connection and proximal mapping network based on deep learning Process the connection result; finally, combine the processing result with the intermediate result of each image block The reconstructed image data is obtained by adding them together, and then denormalized to obtain the reconstructed image.

[0050] The above-mentioned proximal mapping network based on deep learning It is used to guide the semantics and signal-to-noise ratio of the reconstruction process of the approximate point mapping by taking the connection result as input and giving the recovered residual content. Specifically, it is achieved through the following formula:

[0051] in, is the image data reconstructed in the kth round, concat is the connection process, and the meanings of the remaining letters are the same as those in the above description. Np The iteration step length is ρ , and get the optimized result.

[0052] In an optional embodiment of the present application, the semantic extraction network It consists of a convolutional layer, a channel environment adaptation module ACAM, three residual blocks, another channel environment adaptation module ACAM and a 1×1 kernel convolutional layer cascaded in sequence.

[0053] In an optional embodiment of the present application, the architecture of the channel environment adaptation module ACAM used in the above-mentioned semantic sampling module, encoder, decoder and semantic reconstruction module can be referred to. Figure 6 , Figure 6 It is a schematic diagram of the architecture of a channel environment adaptation module provided by an embodiment of the present application. The above-mentioned channel environment adaptation module ACAM includes an average pooling layer, a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence; the function of the above-mentioned channel environment adaptation module ACAM is to continue to reprocess the input features based on the signal-to-noise ratio in a known environment, and the specific process is as follows: first, the intermediate features output by the previous module connected to the above-mentioned channel environment adaptation module ACAM are averaged and pooled to obtain an average pooling result; then, the above-mentioned average pooling result is spliced ​​with the signal-to-noise ratio to obtain a spliced ​​feature; the above-mentioned spliced ​​feature is processed by the first fully connected layer, the PReLU layer, the second fully connected layer, and the sigmoid layer to obtain a channel environment adaptive scaling factor; according to the above-mentioned channel environment adaptive scaling factor and the intermediate features output by the previous module connected to the channel environment adaptive module ACAM, an intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment is obtained.

[0054] Exemplarily, the intermediate feature output by the previous module connected to the channel environment adaptation module ACAM is , the size of each feature is h×w, c is the number of features, and the intermediate feature is used as the input feature for average pooling operation to obtain the average pooling result , which is expressed as follows: , Among them, gloalAvergePooling is the average pooling operation. is the intermediate feature of the input In position [ j , k ] on the elements, is the set of real numbers.

[0055] The average pooling result SNR μ Perform splicing to obtain splicing features I , where the splicing operation Represents the concatenation of two features along a specific dimension, which can be expressed as:

[0056] Then, a nonlinear network consisting of a fully connected layer and an activation function is used to predict the scaling parameter S for each feature. The first fully connected layer FC The activation function after 1 is the PReLU layer, the second fully connected layer FC The activation function after 2 is the sigmoid layer, which is used to map the output to the range of [0, 1] and play the role of allocating proportions, as shown below:

[0057] Finally, perform element-to-element multiplication to obtain the output features of the aggregated input features and the signal-to-noise ratio attention. , , whose size is similar to the input feature The same is expressed as follows: .

[0058] In an optional embodiment of the present application, the channel environment adaptive sampling-semantics-channel coding joint optimization method proposed in the present application is implemented by a channel environment adaptive sampling-semantics-channel coding joint model, and the training process is as follows: calculate the loss between the above-mentioned reconstructed image and the corresponding image in the above-mentioned training image set; optimize the parameters in the semantic sampling module, encoder, decoder and semantic reconstruction module based on the above-mentioned loss to obtain the channel environment adaptive sampling-semantics-channel coding joint model. Among them, the set of all end-to-end trainable parameters is used Representation, including the semantic scanning network in the semantic sampling module , generate matrix A, encoder Parameters , decoder Parameters , the gradient descent step size in the semantic reconstruction module , semantic extraction network , proximal mapping network , expressed as:

[0059] In an optional embodiment of the present application, for N b Size The training set of pictures , the average size of a given block sample q i (i.e. overall sampling ratio ), in each round of training, the channel signal-to-noise ratio μ ∈[0,20]dB and uniformly distributed, that is, μ from The medium probability of random selection is q i from A medium probability random selection, where N = B × B is the block size, l is the number of blocks, and the loss based on mean square error (mean square error is used to measure the distortion between the input image and the reconstructed image) can obtain stable convergence and higher reconstruction accuracy. (·) indicates the end-to-end processing process, represents the reconstructed image after transmission through the system, then the loss function of end-to-end training can be expressed as follows:

[0060] The embodiment of the present application proposes a channel environment adaptive sampling-semantic-channel coding joint optimization method, wherein a transmitting end performs semantic sampling on an image to be transmitted to obtain a semantic sampling result; the transmitting end inputs the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptive modules alternately cascaded, wherein the input of each channel environment adaptive module in the encoder is a signal-to-noise ratio and an intermediate feature output by a previous residual convolution module, and the output is an intermediate feature adaptive to the signal-to-noise ratio of the current channel environment; the receiving end receives the image feature vector affected by physical channel noise and performs preprocessing; the receiving end inputs the preprocessing result into a decoder to obtain an image feature vector to be semantically reconstructed, wherein the decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptive modules alternately cascaded, wherein the input of each channel environment adaptive module in the decoder is a signal-to-noise ratio and an intermediate feature output by a previous residual transposed convolution module, and the output is an intermediate feature adaptive to the signal-to-noise ratio of the current channel environment; the receiving end performs semantic reconstruction according to the image feature vector to be semantically reconstructed to obtain a reconstructed image. In the method proposed in this application, a channel environment adaptation module based on the attention mechanism is added to the end-to-end joint design of semantic-based sampling and reconstruction, encoding and decoding, which further combines the characteristics of the source and the channel. While considering semantic sampling and reconstruction, it can support image transmission in an environment with varying signal-to-noise ratio and ensure the reconstruction quality of the image.

[0061] In a second aspect of the embodiment of the present application, a sampling-semantics-channel coding joint optimization system for channel environment adaptation is proposed, see Figure 7 , Figure 7 : is an architecture diagram of a channel environment adaptive sampling-semantics-channel coding joint optimization system proposed in an embodiment of the present application, and the system includes: The semantic sampling module 701 is used to perform semantic sampling on the image to be transmitted to obtain a semantic sampling result; The encoding module 702 is used to input the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is the signal-to-noise ratio and the intermediate feature output by the previous residual convolution module, and the output is the intermediate feature adapted to the signal-to-noise ratio of the current channel environment; A preprocessing module 703, used for receiving and preprocessing image feature vectors affected by physical channel noise; A decoding module 704 is used to input the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction. The decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded. The input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment. The semantic reconstruction module 705 is used to perform semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

[0062] Wherein, the semantic sampling module includes: A block division submodule, used for normalizing the image to be transmitted and then dividing it into blocks to obtain a plurality of image blocks; A semantic scanning submodule, used for inputting the image to be transmitted and the signal-to-noise ratio into a semantic scanning network to obtain a semantic information distribution map; An aggregation correction submodule, used for obtaining a semantic sampling ratio distribution map according to the semantic information distribution map and the signal-to-noise ratio by adopting a block sampling ratio aggregation correction algorithm; The decomposition submodule is used to perform singular value decomposition on the training image set to obtain an initial matrix; A semantic sampling ratio acquisition submodule, for obtaining a semantic sampling ratio of an i-th image block among the multiple image blocks based on a given overall sampling rate and the semantic information distribution map; A semantic sampling matrix acquisition submodule, used to obtain a semantic sampling matrix of the i-th image block based on the semantic sampling ratio of the i-th image block and the initial matrix; A semantic sampling result acquisition submodule, used for performing semantic sampling based on the i-th image block and the semantic sampling matrix of the i-th image block to obtain the i-th initial semantic sampling result; The sampling rate alignment submodule is used to perform block initialization sampling rate alignment based on the i-th initial semantic sampling result and the transposed semantic sampling matrix of the i-th image block to obtain a final semantic sampling result.

[0063] Among them, the semantic scanning network in the semantic scanning submodule is composed of a first convolutional layer, a first channel environment adaptation module, multiple simple residual blocks, a second channel environment adaptation module and a sequential cascade of convolutional layers; the semantic scanning network is used to evaluate the significance of the semantic information at each position on the image to be transmitted.

[0064] Wherein, the aggregation correction submodule includes: A division unit is used to divide the semantic information distribution map into blocks of size B ×B of l image blocks; Block aggregation unit for l The image blocks are aggregated to obtain the semantic sampling ratio distribution map of the block sampling rate including the distribution of all image blocks. 。

[0065] Wherein, the semantic reconstruction module includes: The expansion submodule is used to expand the image feature vector that needs semantic reconstruction into l image blocks, perform random transformation enhancement on each image block, and then perform block gradient descent to obtain the intermediate result of each image block, and summarize the intermediate results of all image blocks; A filling and extension submodule is used to receive the semantic sampling ratio distribution map sent by the sending end, and use the corresponding sampling ratio in the semantic sampling ratio distribution map as a filling element to obtain an extended semantic sampling ratio distribution map; A feature embedding submodule, used for embedding the features of the extended semantic sampling ratio distribution map into a feature space through a semantic extraction network oriented to the semantic sampling ratio distribution map and the signal-to-noise ratio, and outputting a feature map extracted based on semantics and the signal-to-noise ratio; A connection submodule, used to connect the intermediate result of each image block with the feature map extracted based on semantics and signal-to-noise ratio, and process the connection result based on the proximal mapping network of deep learning; The reconstruction submodule is used to add the processing result to the intermediate result of each image block to obtain the reconstructed image data, and perform denormalization to obtain the reconstructed image.

[0066] The channel environment adaptation module in the channel environment adaptive sampling-semantic-channel coding joint optimization system includes an average pooling layer, a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence; the channel environment adaptation module reprocesses the input features based on a known signal-to-noise ratio, and the channel environment adaptation module includes: An average pooling submodule, used to perform an average pooling operation on the intermediate features output by the previous module connected to the channel environment adaptation module to obtain an average pooling result; A splicing submodule, used for splicing the average pooling result with the signal-to-noise ratio to obtain a splicing feature; A feature prediction scaling submodule, for processing the concatenated features through a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer to obtain a channel environment adaptive scaling factor; The feature convergence submodule is used to obtain intermediate features that are adaptive to the signal-to-noise ratio of the current channel environment based on the channel environment adaptive scaling factor and the intermediate features output by the previous module connected to the channel environment adaptive module.

[0067] Wherein, the system further includes a training module, and the training module includes: A loss calculation submodule, used to calculate the loss between the reconstructed image and the corresponding image in the training image set; The optimization submodule is used to optimize the parameters in the semantic sampling module, encoder, decoder and semantic reconstruction module based on the loss to obtain the sampling-semantic-channel coding joint model adaptive to the channel environment.

[0068] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Figure 8 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Figure 8 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected via a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement a channel environment adaptive sampling-semantics-channel coding joint optimization method disclosed in an embodiment of the present application.

[0069] Based on the same application concept, an embodiment of the present application discloses in a fourth aspect a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, a channel environment adaptive sampling-semantics-channel coding joint optimization method disclosed in an embodiment of the present application is implemented.

[0070] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0071] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0075] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0076] The above is a detailed introduction to a channel environment adaptive sampling-semantics-channel coding joint optimization method and system provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A sampling-semantics-channel coding joint optimization method for channel environment adaptation, characterized in that: The method comprises: The sending end performs semantic sampling on the image to be transmitted to obtain a semantic sampling result; The transmitting end inputs the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is the signal-to-noise ratio and the intermediate feature output by the previous residual convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; The receiving end receives the image feature vector affected by the physical channel noise and performs preprocessing; The receiving end inputs the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction. The decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded. The input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment. The receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

2. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 1, characterized in that: The sender performs semantic sampling on the image to be transmitted and obtains the semantic sampling results, including: The sending end normalizes the image to be transmitted and then divides it into blocks to obtain multiple image blocks; The sending end inputs the image to be transmitted and the signal-to-noise ratio into the semantic scanning network to obtain a semantic information distribution map; The sending end adopts a block sampling ratio aggregation correction algorithm to obtain a semantic sampling ratio distribution map according to the semantic information distribution map and the signal-to-noise ratio; The sending end performs singular value decomposition on the training image set to obtain the initial matrix; For an i-th image block among the multiple image blocks, based on a given overall sampling rate and the semantic information distribution map, obtaining a semantic sampling ratio of the i-th image block; Based on the semantic sampling ratio of the i-th image block and the initial matrix, obtaining a semantic sampling matrix of the i-th image block; Based on the i-th image block and the semantic sampling matrix of the i-th image block, semantic sampling is performed to obtain an i-th initial semantic sampling result; Based on the i-th initial semantic sampling result and the transposed semantic sampling matrix of the i-th image block, block initialization sampling rate alignment is performed to obtain a final semantic sampling result.

3. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 2, characterized in that: The semantic scanning network is composed of a first convolutional layer, a first channel environment adaptation module, multiple simple residual blocks, a second channel environment adaptation module and a sequential cascade of convolutional layers; the semantic scanning network is used to evaluate the significance of semantic information at each position on the image to be transmitted.

4. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 2, characterized in that: The sending end adopts a block sampling ratio aggregation correction algorithm to obtain a semantic sampling ratio distribution map according to the semantic information distribution map and the signal-to-noise ratio, including: According to the order from left to right and from top to bottom, the semantic information distribution map is divided into B × B of l image blocks; right l The image blocks are aggregated to obtain the semantic sampling ratio distribution map including the block sampling rates of all image block distribution conditions.

5. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 1, characterized in that: The receiving end performs semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image, including: The receiving end expands the image feature vector that needs semantic reconstruction into l image blocks, perform random transformation enhancement on each image block, and then perform block gradient descent to obtain the intermediate result of each image block, and summarize the intermediate results of all image blocks; Receiving a semantic sampling ratio distribution map sent by a transmitting end, and using corresponding sampling ratios in the semantic sampling ratio distribution map as filling elements to obtain an extended semantic sampling ratio distribution map; By means of a semantic extraction network oriented to the semantic sampling ratio distribution map and the signal-to-noise ratio, the features of the extended semantic sampling ratio distribution map are embedded into the feature space, and a feature map based on semantics and signal-to-noise ratio extraction is output; Connecting the intermediate result of each image block with the feature map extracted based on semantics and signal-to-noise ratio, and processing the connection result based on the proximal mapping network of deep learning; The processing result is added to the intermediate result of each image block to obtain the reconstructed image data, and then denormalized to obtain the reconstructed image.

6. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 1, characterized in that: The channel environment adaptation module includes an average pooling layer, a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence; the channel environment adaptation module reprocesses the input features based on a known signal-to-noise ratio, and the specific process is as follows: Performing an average pooling operation on the intermediate features output by the previous module connected to the channel environment adaptation module to obtain an average pooling result; Splicing the average pooling result with the signal-to-noise ratio to obtain a splicing feature; The concatenated features are processed by a first fully connected layer, a PReLU layer, a second fully connected layer, and a sigmoid layer to obtain a channel environment adaptive scaling factor; According to the channel environment adaptive scaling factor and the intermediate feature output by the previous module connected to the channel environment adaptive module, the intermediate feature for adaptively adjusting the signal-to-noise ratio of the current channel environment is obtained.

7. The sampling-semantics-channel coding joint optimization method for channel environment adaptation according to claim 2, characterized in that: The channel environment adaptive sampling-semantics-channel coding joint optimization method is implemented through the channel environment adaptive sampling-semantics-channel coding joint model. The training process includes: Calculating a loss between the reconstructed image and a corresponding image in the training image set; Based on the loss, the parameters in the semantic sampling module, encoder, decoder and semantic reconstruction module are optimized to obtain the sampling-semantic-channel coding joint model adaptive to the channel environment.

8. A channel environment adaptive sampling-semantics-channel coding joint optimization system, characterized in that: The system comprises: A semantic sampling module is used to perform semantic sampling on the image to be transmitted to obtain a semantic sampling result; An encoding module, used for inputting the semantic sampling result into an encoder to obtain an image feature vector to be transmitted, wherein the encoder is composed of a plurality of residual convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the encoder is a signal-to-noise ratio and an intermediate feature output by a previous residual convolution module, and the output is an intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; A preprocessing module, used for receiving and preprocessing image feature vectors affected by physical channel noise; A decoding module, used for inputting the preprocessing result into a decoder to obtain an image feature vector that needs semantic reconstruction, wherein the decoder is composed of a plurality of residual transposed convolution modules and a plurality of channel environment adaptation modules alternately cascaded, wherein the input of each channel environment adaptation module in the decoder is the signal-to-noise ratio and the intermediate feature output by the previous residual transposed convolution module, and the output is the intermediate feature that is adaptive to the signal-to-noise ratio of the current channel environment; The semantic reconstruction module is used to perform semantic reconstruction according to the image feature vector that needs semantic reconstruction to obtain a reconstructed image.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the channel environment adaptive sampling-semantics-channel coding joint optimization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the sampling-semantics-channel coding joint optimization method for channel environment adaptation described in any one of claims 1-7 is implemented.

Images