Adaptive semantic communication transmission method and electronic device

Abstract

The present disclosure provides a kind of adaptive semantic communication transmission method and electronic equipment, the method first determines the first model parameter corresponding to transmitter, the second model parameter corresponding to receiver, at least one in transmission content, obtains the transmission model after training;With the source data to be transmitted as input, through the transmission model after training, at the receiver side, obtain the reconstructed target data suitable for source data;And / or, determine the channel state information and adjusted transmission model;Channel state information is used to characterize the state of the channel between transmitter and receiver;Through the adjusted transmission model, with channel state information as input, respectively participate in encoding modulation and decoding modulation, obtain the reconstructed target data suitable for channel state information.The method can significantly improve the rate-distortion gain of semantic communication.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of semantic communication, in particular to an adaptive semantic communication transmission method and an electronic device. BACKGROUND

[0002] Semantic communication is a new architecture that can integrate user's demand and information meaning into the communication process, which is expected to become a new basic paradigm of future Internet of Everything, and fundamentally solve the problems of cross-system, cross-protocol, cross-network, cross-human-machine incompatibility and difficult interconnection in traditional communication protocols based on data, thereby promoting the deep integration of information communication technology progress and artificial intelligence innovation.

[0003] In terms of image data, the current semantic-based source channel joint coding technology has achieved excellent performance. Firstly, it maps the picture into a semantic representation space, then introduces an entropy model to estimate the information amount of each representation, and according to the requirements of transmission effectiveness and reconstruction quality, it allocates reasonable transmission bandwidth for each representation, finally it uses deep learning-based source channel joint coding to map the semantic representation to continuous channel symbols for transmission. Compared with deep learning-based source channel joint coding that directly extracts source features, it greatly improves the effectiveness of coding.

[0004] However, the current methods only focus on the average performance on the training set, and fail to reflect the differentiated characteristics of different samples. For each specific test sample, it cannot achieve the optimal performance, especially when the distribution of the test data set is quite different from that of the training data set, the suboptimal problem is more serious. SUMMARY

[0005] Therefore, the purpose of the present disclosure is to provide an adaptive semantic communication transmission method and an electronic device.

[0006] To achieve the above object, in a first aspect, the disclosure provides a self-adaptive semantic communication transmission method, wherein the method is performed based on a transmission model supporting semantic communication, the transmission model comprising a transmitter and a receiver; the method comprises: determining at least one of a first model parameter corresponding to the transmitter, a second model parameter corresponding to the receiver, and transmission content through training, to obtain a trained transmission model; taking source data to be transmitted as input, obtaining reconstructed target data corresponding to the source data on the receiver side through the trained transmission model; and / or determining channel state information and an adjusted transmission model; the channel state information is used to represent the state of the channel between the transmitter and the receiver; the adjusted transmission model is used to participate in encoding and decoding modulation respectively by taking the channel state information as input, to obtain reconstructed target data corresponding to the channel state information.

[0007] As an implementable manner, the transmitter comprises an encoder and a nonlinear transformation module; the first model parameter comprises an encoder parameter and a nonlinear transformation parameter; before the determining the first model parameter corresponding to the transmitter through training, the method further comprises: determining source data to be transmitted, and establishing a first test data set corresponding to the source data; the determining at least one of the first model parameter corresponding to the transmitter, the second model parameter corresponding to the receiver, and the transmission content through training comprises: based on the first test data set, using gradient descent method to optimize the encoder parameter corresponding to the encoder in the transmitter and the nonlinear transformation parameter corresponding to the nonlinear transformation module.

[0008] As an implementable manner, the receiver comprises a decoder and a nonlinear inverse transformation module; the second model parameter comprises a decoder parameter and a nonlinear inverse transformation parameter; the determining at least one of the first model parameter corresponding to the transmitter, the second model parameter corresponding to the receiver, and the transmission content through training comprises: based on the first test data set, using gradient descent method to optimize the encoder parameter corresponding to the encoder in the transmitter and the nonlinear transformation parameter corresponding to the nonlinear transformation module; and optimizing the decoder parameter corresponding to the decoder in the receiver and the nonlinear inverse transformation parameter corresponding to the nonlinear inverse transformation module.

[0009] As an implementable manner, the optimization of the encoder parameters corresponding to the encoder in the transmitter and the nonlinear transformation parameters corresponding to the nonlinear transformation module, and the optimization of the decoder parameters corresponding to the decoder in the receiver and the nonlinear inverse transformation parameters corresponding to the nonlinear inverse transformation module based on the first test data set include: sending a sampling frame in the first test data set into the transmission model, calculating an RD loss function value corresponding to the transmission model; calculating the residual error of the parameters of the decoder and the nonlinear inverse transformation after the update and before the update, modeling the residual error by using a probability distribution model to obtain the probability distribution of the residual error; obtaining an RDM loss function of the model based on the RD loss function value and the probability distribution of the residual error; calculating the gradient of the nonlinear transformation, the encoder, the decoder and the nonlinear inverse transformation respectively by using the RDM loss function, and optimizing the nonlinear transformation parameters, the encoder parameters, the decoder parameters and the nonlinear inverse transformation parameters respectively by using the gradient descent method.

[0010] As an implementable manner, the transmission content includes semantic representation algorithm parameters and channel symbols; before the at least one of the first model parameters corresponding to the transmitter, the second model parameters corresponding to the receiver and the transmission content is determined by training, the method includes: determining source data to be transmitted, and establishing a first test data set corresponding to the source data; the at least one of the first model parameters corresponding to the transmitter, the second model parameters corresponding to the receiver and the transmission content is determined by training, including: based on the first test data set, the semantic representation algorithm parameters in the transmitter are optimized by using the gradient descent method, and an updated semantic vector is obtained; the semantic vector is encoded by the encoder to obtain the channel symbols; the channel symbols are optimized by using the gradient descent method.

[0011] As an implementable manner, the encoder in the transmitter includes a first auxiliary codec module; the decoder in the receiver includes a second auxiliary codec module; the determination of the adjusted transmission model includes: adding a pair of channel modulation networks in the transmission model; the channel modulation network includes a plurality of channel modulation modules and a plurality of first nonlinear operation layers; the channel modulation module includes at least one second nonlinear operation layer and a corresponding activation function; wherein, after the first auxiliary codec module in the encoder in the transmitter, the first channel modulation network is added; before the second auxiliary codec module in the decoder in the receiver, the second channel modulation network corresponding to the first channel modulation network is added.

[0012] As an implementable manner, the participating in encoding modulation and decoding modulation by the adjusted transmission model and taking the channel state information as input comprises: in the encoding modulation, the output of the semantic representation vector to be sent in the transmitter after the first auxiliary codec module in the encoder, as input, is sent to the first nonlinear operation layer in the plurality of first nonlinear operation layers; the channel state information and the output of the first nonlinear operation layer are sent to the first channel modulation module; the first channel modulation module is located at the first position in the plurality of channel modulation modules and is connected with the output end of the first nonlinear operation layer; in the channel modulation module, the output of the channel state information after the operation of at least one second nonlinear operation layer in the channel modulation module is multiplied with the output of the first nonlinear operation layer to obtain the output corresponding to the first channel modulation module; the output end corresponding to the first channel modulation module is connected with the second nonlinear operation layer in the plurality of first nonlinear operation layers; the output end of the second nonlinear operation layer is connected with the second channel modulation module in the plurality of channel modulation modules; in the decoding modulation, the semantic representation vector obtained by the receiver is sent to the first nonlinear operation layer in the plurality of first nonlinear operation layers as input before being input to the second auxiliary codec module in the decoder.

[0013] As an implementable manner, the transmission model is an adaptive semantic communication model.

[0014] In a second aspect, the disclosure also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the above first aspect when executing the program.

[0015] In a third aspect, the disclosure also provides a non-transitory computer readable storage medium, which stores computer instructions for causing a computer to execute the method of any one of the above first aspect.

[0016] As can be seen from the above, the adaptive semantic communication transmission method and the electronic device provided by the disclosure perform corresponding overfitting processing on the source data and the channel state information respectively, establish a test data set corresponding to the source data according to the input source data and / or channel state information, adjust the parameters or transmission content in the transmission model, or improve the network structure of the transmission model, so that the channel state information participates in the calculation in the process of encoding modulation and decoding modulation, so that the extracted features carry the channel state characteristics, and the influence of the channel state on the reconstruction result is reduced. In this way, the overfitting method based on deep learning is successfully applied to the semantic communication system, and compared with the semantic communication system in the related art, the rate distortion performance is better. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the disclosure or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The application scenario of the embodiment of the disclosure is shown in the figure.

[0019] Figure 2 The data processing flowchart of the scheme of adjusting the transmitter parameters in the source data adaptive scheme of the embodiment of the disclosure is shown in the figure.

[0020] Figure 3 The data processing flowchart of the scheme of adjusting the channel symbols in the source data adaptive scheme of the embodiment of the disclosure is shown in the figure.

[0021] Figure 4 The data processing flowchart of the scheme of adjusting the transmitter and receiver parameters in the source data adaptive scheme of the embodiment of the disclosure is shown in the figure.

[0022] Figure 5 The overall architecture of the transmission model inserting the channel modulation network Channel ModNet and the local enlarged view of the SM module of the embodiment of the disclosure are shown in the figure.

[0023] Fig. 6(a) and Fig. 6(b) are PSNR performance diagrams of different channel bandwidth ratios under the AWGN channel with a signal-to-noise ratio of 0 dB and 10 dB in the embodiment of the disclosure.

[0024] Figure 7Fig. 4 is a schematic diagram of PSNR performance of different SNRs in a Cost2100 (5.3 GHz) channel with the same channel bandwidth ratio according to an embodiment of the present disclosure;

[0025] Figure 8 Fig. 5 is a schematic diagram of PSNR performance of different SNRs in an AWGN channel with the same channel bandwidth ratio according to an embodiment of the present disclosure;

[0026] Figure 9 Fig. 6 is a schematic diagram of a structural framework of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to specific embodiments and drawings.

[0028] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present disclosure should be understood as their common meanings to those skilled in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in the embodiments of the present disclosure do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "front", "back", and the like only represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0029] In the field of semantic communication, the design idea of traditional communication is how to make the source information pass through the communication channel with interference without error. The goal of semantic communication is to use human perception indicators and machine task accuracy as distortion metrics to more efficiently extract the most valuable semantic information.

[0030] In learning based on a semantic communication model, a hypothesis can obtain better fitting on training data than other hypotheses, but cannot well fit the data on a data set outside the training data. At this time, it is considered that the hypothesis has overfitting. The main reason for this phenomenon is that there is noise in the training data or the training data is too small. Overfitting processing needs to be used to deal with the overfitting problem.

[0031] The existing related methods only focus on the average performance on the training set, and cannot achieve the optimal performance for a specific personalized test sample, especially when the distribution of the test data set is quite different from that of the training data set, the sub-optimization problem is more serious, resulting in low data restoration degree of the reconstructed data at the receiving end and reduced transmission reliability.

[0032] An important application field of voice communication is image processing. In traditional image coding, some compression standards use different linear transformations for different compressed data to achieve better compression performance. In the field of deep learning-based image coding, the inventors found that using this overfitting idea, fine-tuning network parameters or modifying hidden variables for different compressed data can solve the sub-optimization problem of neural networks.

[0033] To achieve the above purpose, the present disclosure provides an adaptive semantic communication transmission method based on overfitting. The method is executed based on an adaptive semantic communication (ASC) system. The ASC system structure is a novel end-to-end transmission system using overfitting promotion technology. Compared with the original nonlinear source channel joint coding system, the model can perform overfitting on the source data and channel state information domain through online learning in the deployment stage. The method uses a nonlinear source channel joint coding system as a transmission framework and can overfit different source data and channel states. It solves the sub-optimization problem and has achieved significant rate-distortion gain.

[0034] Reference Figure 1 Fig. 1 is a schematic diagram of an application scenario of an adaptive semantic communication transmission method provided by an embodiment of the present application. The application scenario includes a transmitter 101 and a receiver 102. The transmitter 101 and the receiver 102 can be connected through a wired or wireless communication network, and the communication mode is semantic communication. The transmitter 101 or the receiver 102 can be integrated into a terminal device, which includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA), or other electronic devices that can realize the above functions. The transmitter 101 or the receiver 102 can also be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0035] Transmitter 101 is used to send the content to be transmitted to receiver. The input data to transmitter 101 can be source data, i.e., source data that needs to be transmitted using a model that supports semantic transmission. Transmitter 101 performs semantic representation processing and encoding on the source data to obtain a data format that can be transmitted in the channel. Receiver 102 is used to decode the received data, restore the corresponding semantics, and then reconstruct the target data corresponding to the source data, thus realizing semantic-based data transmission. The transmission model can be understood as an Adaptive Semantic Communication (ASC) model, specifically, it can be various network models that support semantic communication transmission, or it can be obtained based on a network model that supports semantic communication transmission.

[0036] The following is combined Figure 1 The application scenarios described above illustrate the adaptive semantic communication transmission method according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this regard. Rather, the embodiments of this application can be applied to any applicable scenario.

[0037] For example, Figure 1 The transmission model shown can be obtained by improving upon the NTSCC (nonlinear transform source-channel coding) model. The transmitter in this model includes at least an encoder and a nonlinear transform module, and the receiver includes at least a decoder and a nonlinear inverse transform module. The implementation of this transmission method first requires online training and learning during the model deployment phase to optimize the model. Specifically, model optimization can involve adjusting the nonlinear transform g in the transmitter. a encoder f e The semantic features y and channel symbols s can also be used to adjust the decoder f in the receiver. d and nonlinear inverse transform g s In addition, in some embodiments, embodiments of this disclosure may also include encoder f e and decoder f d A pair of Channel Modulation Networks (CMONets) were added, which can adjust the decoder f according to specific channel state information. e and encoder f d The output is adjusted to implement a channel-dependent mechanism.

[0038] The overfitting processing provided in this disclosure includes two aspects: the first aspect is overfitting based on source data, and the second aspect is overfitting based on channel state information.

[0039] The first is the over-fitting method for source data. This method includes various training strategies, which can be summarized as optimizing the first model parameters in the transmitter, the second model parameters in the receiver, and at least one of the transmission content through training. Among them, the first model parameters refer to the optimizable parameters of the corresponding modules on the transmitter side. The second model parameters refer to the optimizable parameters of the corresponding modules on the receiver side. The transmission content can include optimizable parameters in the semantic representation algorithm and channel symbols.

[0040] Exemplarily, three training strategies are listed below.

[0041] The first training strategy is to realize self-adaptation by adjusting the network model parameters in the transmitter. Specifically, the specific test data or the sampling of the test data domain is transmitted into the trained NTSCC model, the rate distortion function is calculated for the gradient of the non-linear transformation and the encoder in the receiver, the model parameters of the non-linear transformation and the encoder are optimized by using the gradient descent method, and then the gradient is recalculated. After a certain number of iterations, the training is completed. The specific steps are as follows:

[0042] Step 201, collect pictures of quality meeting predetermined requirements to form a training image data set, and perform random cropping preprocessing on the training set images. Use the training set to fully train the NTSCC model and initialize the network parameters.

[0043] Step 202, collect short videos of quality meeting predetermined requirements, downsample the frame sequence of the video, and take out all I frames to form a test data set. The test data set can include multiple frames of images, or can be further classified to obtain multiple subsets, each subset containing multiple frames of images of the same category.

[0044] Step 203, send the sampling frame of one frame or one subset in the test set into the NTSCC model, calculate the rate distortion (RD) loss function value of the model (hereinafter referred to as RD loss function), and optimize the non-linear transformation g a and the encoder f e using the gradient of the RD loss function. Iterate this process T max times.

[0045] Step 204, transmit the specific frame or sampling of the same subset in step 203 into the updated NTSCC model to obtain the reconstructed frame

[0046] The data processing flowchart corresponding to steps 203-204 is shown in Figure 2 . Among them, Latent Prior represents the latent prior parameter, g a represents the non-linear transformation, and f edenotes encoding, W denotes semantic encoding, f d denotes decoding, g a denotes nonlinear inverse transform.

[0047] The second training strategy is to realize self-adaptation by adjusting the semantic representation and channel symbol in the transmitter. Specific test data is sent into the trained NTSCC model, the rate distortion function is calculated for the semantic representation, and then the gradient descent method is used to optimize the semantic representation. After a certain number of iterations, the new semantic representation is encoded and transmitted, and is reconstructed at the receiving end.

[0048] The gradient of the rate distortion function on the channel symbol is calculated again, and the gradient descent method is used to optimize the channel symbol. After a certain number of iterations, the receiving end finally recovers and reconstructs using the updated channel symbol.

[0049] It should be noted that this strategy is only applicable to overfitting specific test data and cannot be self-adaptive to the test data domain.

[0050] The specific steps are as follows:

[0051] Step 301, collect pictures of quality meeting predetermined requirements to form a training image data set, and perform random cropping preprocessing on the training set images. The training set is used to fully train the NTSCC model, and the network parameters are initialized.

[0052] Step 302, collect short videos of quality meeting predetermined requirements, downsample the frame sequence of the video, and take out all I frames to form a test data set.

[0053] Step 303, a certain frame in the test set is transmitted into the NTSCC model, the RD loss function value of the model is calculated, the gradient of the loss function is used to optimize the semantic representation y, and the process is iterated Y max times.

[0054] Step 304, the updated semantic representation y * is transmitted into the encoder f e The source channel joint encoding is performed to obtain the channel symbol s. s is transmitted through the channel into the receiver to obtain the recovered frame.

[0055] Step 305, the RD loss function value of step 304 is calculated, the gradient of the loss function is used to optimize the channel symbol s, and the process is iterated S max times.

[0056] Step 306, the updated channel symbol s * is transmitted through the channel into the receiver, the receiver decodes and inversely transforms it to obtain the final reconstructed frame

[0057] Steps 303-306 correspond to the data processing procedure shown in Figure 3 , where each English character represents the same Figure 2 , and will not be repeated here.

[0058] The third strategy is to update the model parameters of the decoder and the nonlinear inverse transform in the receiver simultaneously based on the first strategy. Although this strategy needs to transmit the updated parameters of the receiver through the channel, introducing a small additional overhead, it can obtain greater gain compared to the first strategy.

[0059] Referring to Figure 4 , the specific steps are as follows:

[0060] Step 401, collect pictures of quality meeting predetermined requirements to form a training image dataset, and perform random cropping preprocessing on the training set images. Use the training set to fully train the NTSCC model and initialize the network parameters.

[0061] Step 402, collect short videos of quality meeting predetermined requirements, downsample the frame sequence of the video, and take out all I frames to form a test dataset.

[0062] Step 403, send a frame or a subset of sample frames in the test set into the NTSCC model to calculate the RD loss function value of the model.

[0063] Step 404, simultaneously calculate the decoder f d and the nonlinear inverse transform g s The updated parameters and the residual error δ (initial value is 0) before updating, quantize δ to obtain and use a probability distribution model to model the entropy of δ to obtain the probability distribution of . Exemplarily, the probability distribution model can be a Gaussian Mixed Model (GMM).

[0064] Step 405, combine the results of steps 403 and 404 to obtain the ratedistortion model (RDM) loss function (hereinafter referred to as RDM loss function) of the model, expressed as L RDM = λ * R + D + β * (-η g logp δ (δ)). Where η g is a constant related to the channel coding capacity, and λ and β are weights controlling the allocation between the RD performance and the model transmission rate.

[0065] Step 406, use the RDM loss function to calculate its respective nonlinear transform g a and encoder f e, decoder f d and nonlinear inverse transform g s The gradient of the loss function is calculated, and the network parameters are optimized using a gradient descent optimization algorithm.

[0066] Step 407: Repeat steps 403-406 for T iterations. The specific frame or subset of samples in step 403 is input into the updated NTSCC model. max Step 407: Repeat steps 403-406 for T iterations. The specific frame or subset of samples in step 403 is input into the updated NTSCC model.

[0067] wherein, in one embodiment, the quantized model residual The separated source coding and channel coding are used for transmission, and the decoded residual is added to the parameters of the receiver, and the receiver is used to recover and reconstruct the channel symbols to obtain the final reconstructed frame

[0068] The data processing flowchart corresponding to steps 403-407 is shown in Figure 4 , where Encoder is the encoder, Decoder is the decoder, and Model Prior is the prior model.

[0069] The above are several examples of overfitting schemes for source data.

[0070] Since the channel data used in the above scheme is a random sample from the channel state domain known to the transmitter, only overfitting of the channel domain can be achieved. However, in actual communication, channel estimation techniques enable the transmitter to obtain accurate channel state information, and accurate channel state information is a key factor affecting the performance of source-channel joint coding.

[0071] Therefore, the embodiments of the present disclosure further introduce a channel-dependent mechanism, or a channel adjustment mechanism, so that the entire semantic communication system can adapt to specific channel state information without relying on gradient descent. The channel state information is used to represent the state of the channel between the transmitter and the receiver. The state can include various index features representing communication capabilities, data carrying capacity, signal-to-noise ratio, etc.

[0072] Specifically, the adaptive mechanism can be achieved by improving the network architecture of the transmission model, so that the channel state information participates in the operation process of encoding and decoding modulation. First, the adjusted transmission model needs to be determined, and the current spatial or temporal channel state information needs to be determined.

[0073] Exemplarily, for the improvement of the transmission model, a channel modulation network (Channel ModNet) including a plurality of channel modulation modules and a plurality of nonlinear operation layers can be added to the original encoder and decoder.

[0074] The channel modulation module can be one of a signal noise ratio modulation (SM) module, a channel response modulation (HM) module, or a combination of two or more modules.

[0075] The nonlinear operation layer can be one of a fully connected layer, a convolutional layer, or a combination of two or more network layers.

[0076] In some embodiments, the encoder in the transmitter includes a first auxiliary codec module, and the decoder in the receiver includes a second auxiliary codec module.

[0077] Exemplarily, in some embodiments, the first or second auxiliary codec module can be a Transoformer model. In other embodiments, the first or second auxiliary codec module can include a plurality of convolutional layers or other nonlinear operation layers, for example, the auxiliary decoding module can be a convolutional network model composed of a plurality of convolutional layers, or can be a neural network model composed of a plurality of other nonlinear operation layers.

[0078] Hereinafter, the channel modulation module is exemplarily taken as an SM module, the nonlinear operation layer is exemplarily taken as a fully connected layer, and the auxiliary codec module is exemplarily taken as a Transoformer.

[0079] Specifically, the channel modulation network is composed of a plurality of SM modules and a plurality of first fully connected layers (FC, Full Connection), wherein the SM module is composed of at least one second fully connected layer and a corresponding activation function. In order to describe clearly, the fully connected layer inside the SM module is defined as the second fully connected layer, and the remaining fully connected layers in the Channel ModNet are defined as the first fully connected layer.

[0080] Referring to FIG. 1, the encoder in the transmitter includes a Channel ModNet composed of a plurality of SM modules and a plurality of first fully connected layers. Figure 5 In one embodiment, the SM module can include three second fully connected layers. Specifically, in the encoder, the semantic representation vector {y i , i = 1, 2,..., l} after the Transoformer module in the encoder f e , i = 1, 2,..., l} after the Transoformer module in the encoder f i, i = 1, 2, …, l} as input into the first first fully connected layer, and then the specific channel state information SNR i and the output of the fully connected (FC) layer are sent into the first SM module 502, SNR i will pass through three second fully connected layers inside the SM module, and then the output results will be multiplied with the output results of the first first fully connected layer in correspondence, so as to realize the modulation of the specific channel state information h on y′ i .

[0081] Correspondingly, in the receiver, the received channel symbols first pass through the FC network in the encoder f d , and then the FC output results and SNR i are sent into the Channel ModNet network, and the subsequent operations are the same as the calculation process in the encoder, which will not be described again.

[0082] In Figure 5 , Tokens represent features, and Rate Tokens represent rate features. Auxiliary FC Headers represent auxiliary FC headers. CQI (Channel Quality Indicator) represents channel quality indicator, and CSI (Channel State Information) represents channel state information. SM module is an SM module, input token is an input feature, and output token is an output feature.

[0083] Patch+Rate Tokens, that is, Patch Token+Rate Token, where Patch Token represents patch features. Channel ModNet is a channel modulation network. Shared transformer block is a shared transformation block.

[0084] It should be noted that the determination of the channel state information can be performed in the following manner:

[0085] The receiver can obtain accurate channel state information, for example, SNR i in the receiver is the average value along the channel dimension i of the channel symbols s .

[0086] The channel state information SNR iThe ASC can be calculated according to a channel quality indicator (hereinafter referred to as CQI) provided by the feedback link. The CQI is the average signal-to-noise ratio of all transmitted channel symbols s, so in the transmitter

[0087] It should be noted that in the embodiments of the present disclosure, the overfitting mode can be for the domain or for the individual case:

[0088] For a given source data and channel state, the ASC not only can overfit each specific source data and channel state information, but also can overfit the source data domain and the channel state information domain. For example, a series of I frames from the same video from the same scene are highly similar, and the whole can be regarded as a source data domain. Correspondingly, the method of the ASC can not only overfit a specific frame, but also can overfit a frame set of the same scene, or even can overfit frames of different scenes.

[0089] Transmission of content stream and model stream: in the proposed ASC system, not only the transmission content can be updated, but also the model parameters required to be updated by the receiver can be transmitted. After the transmitter completes the adaptation, the transmitter transmits the overfitting output result to the receiver through the channel, and the transmitted symbol is called a content stream. In addition, in order to realize the adaptation of the receiver, the transmitter also transmits the model parameters required to be updated to the receiver in the form of residual, and here the transmitted residual is called a model stream, so as to obtain further gain.

[0090] Next, taking video data as an example of source data, a specific embodiment is listed.

[0091] The embodiment can be based on Figure 5 The transmission model is shown in the figure. The transmission model includes a transmitter and a receiver, the transmitter at least includes an encoder, and the receiver at least includes a decoder; wherein the encoder is a deep learning-based joint source-channel coding encoder Deep JSCC (Joint Source-Channel Coding) Encoder, and the symbol f e is represented. The decoder is a deep learning-based joint source-channel coding decoder Deep JSCC (Joint Source-Channel Coding) Decoder.

[0092] Firstly, determine the data set. The embodiments of the present disclosure use the VSD4K video data set to verify the performance of the ASC model.

[0093] Among them, the VSD4K video dataset is described as follows: public video super-resolution datasets such as Vimeo-90K and REDS only contain adjacent frame sequences (often too short) and are not suitable for video transmission tasks, therefore, multiple 4K videos are collected to simulate actual video transmission scenarios, and a standard bicubic interpolation is used to generate low-resolution videos. Six popular video categories are selected to build VSD4K, including: games, vlogs, interviews, sports, dances, and cityscapes. Each category can be composed of different video lengths, including: 15 seconds, 30 seconds, 45 seconds, 1 minute, 2 minutes, 5 minutes, and the like. For details of the VSD4K dataset, please refer to the related art, and the embodiments of the present disclosure will not be repeated here.

[0094] In the embodiments of the present disclosure, the VSD4K video dataset includes six categories of scenes, including electronic games, vlogs, interviews, sports, dances, and cityscapes, of which the first three categories are single-scene multi-view 4K 30fps (4K resolution, 30 frames per second) videos, and the last three categories are 4K 30fps videos of the same scene but multiple scenes and with large motion.

[0095] Exemplarily, the most representative 45s segment is cut from each category, and then the frame rate is down-sampled by 1 / 4, while an interpolation algorithm is used to produce a 480p video, to obtain an I-frame (intra picture) test set.

[0096] Then, the pre-training set is determined. The embodiments of the present disclosure use the DIV2K public dataset to pre-train the NTSCC model. The DIV2K dataset generally has 1000 high-definition pictures (2K resolution), of which 800 are used for training, 100 are used for verification, and 100 are used for testing.

[0097] In the embodiments of the present disclosure, the DIV2K dataset includes 800 2K resolution images. The training set images are randomly cropped to a resolution of 256'256, and trained using an Adam (Adaptive Moment Estimation) optimizer with a learning rate of 1x10 -4 The training batch size is 8, and a total of 6000 rounds of training are performed. Exemplarily, a total of 4 models with different network parameters are trained, in which λ in the RD function is 256, 64, 16, and 4, respectively. The RD loss function is:

[0098]

[0099] Where y is the semantic representation, z and are side information and quantized side information, respectively, x and are the original image and the reconstructed image, respectively, d is the distortion measure function, and LRD Loss value representing the RD loss function. In embodiments of the present disclosure, not only the traditional MSE (mean-square error) is used as the distortion measure function to train the model, but also a novel perceptual metric LPIPS (structural similarity, SSIM, SSIM (Structural Similarity)) is used as the distortion measure function to train the model.

[0100] The channel used by the pre-trained model is a Gaussian channel with SNR = 10 dB and SNR = 0 dB, and the corresponding scaling factors η y = 0.2 and η y = 0.4.

[0101] Next, for the adaptive scheme of video data, the corresponding scheme can be adjusted according to the training strategy listed above, which can be divided into three ways: adjusting the transmitter parameters, adjusting the transmitter encoding, and adjusting the transmitter and receiver parameters. The specific implementation is described in detail as follows:

[0102] Method one: adjusting the model parameters corresponding to the transmitter (first model parameters).

[0103] Step 1-1, input the video data x into the pre-trained NTSCC model, x can be a certain specific frame, or a random sample from the same picture in a certain video scene, or a random sample of multiple pictures in a certain video scene, which is set according to the range of overfitting desired.

[0104] Step 1-2, calculate the RD loss function corresponding to x, and use the RD loss function to perform gradient descent optimization on the parameters of the nonlinear transformation Φ g and the parameters of the encoder Φ f , the learning rate γ = 1 × 10 -4 , and the iteration T max = 100 times, which can be represented as:

[0105]

[0106]

[0107] Step 1-3, input the video data x into the overfitted NTSCC model, and recover the reconstructed video data

[0108] The following lists an example of the training process corresponding to method one:

[0109]

[0110]

[0111] Method two: adjust the encoding of the transmitter side (semantic representation and channel symbol encoding).

[0112] Step 2-1, input the video data x into the pre-trained NTSCC model to obtain the semantic representation y. In this embodiment, x is a specific frame, which can be specified in advance.

[0113] Step 2-2, calculate the RD loss function corresponding to y, and use the RD loss function to perform gradient descent optimization on the semantic representation y. Specifically, optimize the parameters in the semantic representation algorithm that can be optimized, and the learning rate γ = 1 × 10 -3 , and iterate Y max = 50 times, which can be expressed as:

[0114]

[0115] Step 2-3, input the updated semantic representation y * into the encoder f e to obtain the channel symbol s.

[0116] Step 2-4, calculate the RD loss function corresponding to the channel symbol s, and use the RD loss function to perform gradient descent optimization on the channel symbol s, γ = 1 × 10 -3 , and iterate S max = 50 times.

[0117]

[0118] The following lists an example of the training process corresponding to method two:

[0119]

[0120]

[0121] Method three: adjust the transmitter parameters (first model parameters) and receiver parameters (second model parameters).

[0122] Step 3-1, input the video data into the pre-trained NTSCC model, x can be a specific frame, a random sample from the same picture in a video scenario, or a random sample of multiple pictures in a video scenario, which is set according to the range of overfitting desired.

[0123] Step 3-2, calculate the residual of the parameters that need to be updated in the receiver network, i.e. δ = (δ g , δ f ). Wherein and These are the updated model parameters and their initial values.

[0124] Step 3-3: During the testing phase, δ is transmitted using separate source coding and channel coding; therefore, δ needs to be quantized. Due to the model training process The gradient is almost zero everywhere, therefore δ is used. i The gradient as The estimated value of the gradient.

[0125] Steps 3-4, Calculation The transmission rate is estimated here by adding uniform noise. The cumulative distribution function. We have... The quantization interval Δ = 0.005. Therefore, we have... In this embodiment, a Gaussian mixture model is used as the probability model, wherein... Where q slab (δ i )=N(δ i |0,σ 2 ), σ=0.05, α = 1000.

[0126] Steps 3-5, using After updating the receiver's network parameters θ, calculate the system's RD function, and combine it with... The transmission rate is then used to obtain the RDM loss function. The RDM loss function is: Where η δ (λ, β) is a constant related to the channel coding capacity, controlling the weights allocated between RD performance and model transmission rate. To make the trained model close to the pre-trained model in terms of code rate, four different (λ, β) values ​​were set: (256, 4), (32, 4), (4, 1), and (2, 1).

[0127] Steps 3-6: Calculate the RDM function for the nonlinear transformation g. a encoder f e decoder f d and nonlinear inverse transform g s The gradient is used to optimize the network parameters using the gradient descent optimization algorithm, and the learning rate γ of the transmitter parameters is... φ =1×10 -5 The learning rate γ of the receiver parameters θ =1×10 -5 Specifically, it is expressed as:

[0128]

[0129]

[0130]

[0131]

[0132] Repeat steps 3-2 to 3-6, iteration T max = 10000 times. Then the specific frame or the sample of the same data field in step 3-1 is transmitted into the updated NTSCC model, and the residual of the model update parameter is calculated, The quantized δ is transmitted using entropy coding and LDPC code, and the receiver uses After updating the network parameters, the channel symbol is decoded and inverse transformed to obtain the reconstructed frame

[0133] The following lists an example of the training process corresponding to the third mode:

[0134]

[0135]

[0136] The above-mentioned mode one to mode three are exemplary descriptions of overfitting processing based on source data. It should be noted that in actual application, the overfitting processing based on source data can be implemented in one of the above-mentioned mode one to mode three, or a combination of two or more. For example, the combination of mode one and mode two can obtain another implementation mode, and the combination of mode two and mode three can obtain another implementation mode. Moreover, the specific implementation mode is not limited to the above-mentioned mode one to mode three, and for one or a combination of two or more of the transmitter first model parameter, the receiver second model parameter and the transmission content as the optimization target, other more implementation modes can also be obtained.

[0137] The following describes how to perform adaptive semantic communication processing corresponding to the channel state information in this embodiment.

[0138] Different channel states may have different effects on the ability and performance of semantic communication. Performing the same semantic transmission strategy for different channel states can cause suboptimal problems, so that the target data reconstructed on the receiver side may not meet the expected requirements.

[0139] In order to enable the semantic communication system to flexibly adapt to each channel state, this disclosure embodiment designs a channel dependency mechanism, introducing an insertable module into the Joint Source-Channel Coding (JSCC) or decoder. In the transmitter, it is located after the Transoffmer module in the encoder; in the receiver, it is located before the Transoffmer module in the decoder, as shown below. Figure 5 As shown. For clarity, the module added to the transmitter encoder is defined as the first channel modulation network, and the module added to the receiver decoder is defined as the second channel modulation network.

[0140] The specific modulation process is as follows:

[0141] Step 4-1, from the nonlinear transformation g a N-dimensional semantic representation vector {y i |i=1,2...,l} are encoded by f e After using the Transformer module, we get {y′ i |i=1,2...,l}.

[0142] Step 4-2, set {y′ i |i=1,2...,l} are fed into the first channel modulation network (due to limited view space) Figure 5 (The first channel modulation network is not shown in the image). For example, in this embodiment, the first channel modulation network consists of 8 FC layers (first fully connected layers) and 7 channel modulation modules (SMs). First {y′ i |i=1,2...,l} passes through the first FC layer out of 8 FC layers, and {y′ i The transformation of |i=1,2...,l} into an M-dimensional vector only involves the transformation along the channel dimension.

[0143] Step 4-3: Determine the channel state information (SNR). i Input into all SM modules.

[0144] It should be noted that in the encoder, the encoder can only obtain the average signal-to-noise ratio of all transmitted symbols through the feedback link. Therefore, the way the transmitter obtains channel state information is by calculating the average signal-to-noise ratio, i.e. The receiver can obtain the precise signal-to-noise ratio at each location by means of channel estimation.

[0145] Step 4-4, Channel State Information (SNR) i It passes through the three FC layers (i.e., the three second fully connected layers) inside the SM module in sequence.

[0146]

[0147]

[0148] sm i =Sigmoid(W (3) ·sm i (2) +b (3) )

[0149] Step 4-5, in the SM module, multiply the M-dimensional vector output by the previous FC layer with the corresponding sm i element

[0150] output=input⊙sm i

[0151] Step 4-6, in the first channel modulation network, input the output of the SM module output into the next FC layer (the first fully connected layer, i.e. one of the above 8 FC layers), repeat step 4-5 until all layers and modules in the first channel modulation network are traversed.

[0152] Step 4-7, in the receiver, the decoder f d decodes the channel symbol sequence to obtain and then inputs it into the FC layer (the original fully connected layer of the transmission model) to obtain an N-dimensional vector

[0153] Step 4-8, correspondingly, the second channel modulation network is inserted in the receiver. The N-dimensional vector is sent into the second channel modulation network in the receiver, and the calculation process is similar to that in the transmitter, and finally the recovered semantic representation vector

[0154] The following lists the verification data results of the embodiments of the present disclosure.

[0155] The embodiments of the present disclosure have been trained in multiple configurations, verified multiple times on the verification set, and compared with existing mainstream image transmission schemes.

[0156] Analysis of the verification results:

[0157] Referring to FIGS. 6(a) and 6(b), FIGS. 6(a) and 6(b) respectively show comparison charts of seven picture transmission schemes under the condition of different channel bandwidth ratios in the AWGN (Additive White Gaussian Noise) channel with a signal-to-noise ratio of 0 dB and 10 dB, wherein the seven picture transmission schemes on each chart are as follows:

[0158] NTSCC(TXRX-adapt): TXRX-NTSCC scheme, i.e., the optimized transmitter and receiver parameter scheme performed based on the NTSCC model;

[0159] NTSCC(TX-adapt): TX-NTSCC scheme, i.e., the optimized transmitter parameter scheme;

[0160] NTSCC: the conventional NTSCC scheme without channel data overfitting processing and channel state information overfitting processing;

[0161] Deep JSCC: the conventional fixed-length deep JSCC scheme;

[0162] BPG+5G LDPC: BPG (Better Portable Graphics) image encoding + LDPC (Low Density Parity Check Code) channel coding;

[0163] VTM+5G LDPC: VTM (Virtual Tape Machines) image encoding + LDPC channel coding;

[0164] NTC+5G LDPC: NTC (national telecommunications commission) image encoding + LDPC channel coding.

[0165] Wherein, game represents a game scene, and AWGN channel 0dB is an AWGN channel with a signal-to-noise ratio of 0dB.

[0166] As can be seen from FIGS. 6(a) and 6(b), the TX-NTSCC scheme is overall superior to the ordinary NTSCC scheme in the range of all channel bandwidth ratios shown in the figures, and the TXRX-NTSCC scheme further improves the gain on the basis thereof, and is close to or slightly better than the currently best transmission scheme VTM image encoding + LDPC channel coding. It can be seen that the adaptive scheme proposed in the embodiments of the present disclosure achieves excellent RD performance gain.

[0167] Figure 7 and Figure 8 The effectiveness of the channel modulation network verified in the present embodiment is verified in the AWGN (Additive White Gaussian Noise) channel and the wireless fading channel COST2100, respectively. In Figure 8It can be seen that when the test signal-to-noise ratio is different from the training signal-to-noise ratio, the performance loss of the NTSCC added with the Channel ModNet is reduced compared with different NTSCCs, and the designed scheme outperforms the BPG image encoding + LDPC channel encoding in performance.

[0168] In Figure 8 In the TXRX-NTSCC scheme, it can also be seen that the performance of the NTSCC added with the Channel ModNet is obviously improved, and the performance of the TXRX-NTSCC scheme after adding the Channel ModNet is much higher than that of the BPG image encoding + LDPC channel encoding, even though the latter uses multiple CC-HARQ retransmissions and consumes additional time overhead. It can be seen from the above that the Channel ModNet proposed in the embodiments of the present disclosure can effectively adapt the model to different channel conditions.

[0169] Compared with the related art, the method provided by the embodiments of the present disclosure has the following technical effects:

[0170] First, the neural network can learn based on a large amount of actual source channel data, and has stronger semantic and content adaptive ability compared with the hand-designed in the traditional encoding and transmission standard;

[0171] Second, the neural network can well exhibit and utilize the characteristics of the source channel, which enables the semantic communication system to be optimized for human visual perception and machine vision tasks. However, the existing source channel joint coding standard only pursues high performance in objective evaluation indicators;

[0172] Third, the method of the present disclosure obtains a large gain in RD performance compared with ordinary semantic communication systems, and provides a solution for improving the performance of all deep learning-based semantic communication systems;

[0173] Fourth, the method provided by the embodiments of the present disclosure can utilize the overfitting characteristics to realize any encoding mode, while the traditional source channel coding can only use a limited number of predefined encoding modes. Compared with the traditional encoding, the method of the present disclosure is more flexible, can be customized for specific source and channel instances, and has lower adaptation complexity compared with the mode selection in the traditional coding.

[0174] In summary, the adaptive semantic transmission scheme proposed in the embodiments of the present disclosure has significant innovation and excellent RD performance for high-dimensional source transmission such as images and videos, promotes the development of semantic communication systems, and has good practical prospects.

[0175] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server, etc. The method of the embodiments can also be applied to a distributed scenario, and be completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present disclosure, and the multiple devices can interact with each other to complete the method.

[0176] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0177] Based on the same inventive concept, the present disclosure also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the adaptive semantic communication transmission method of any of the embodiments.

[0178] Figure 9 A more specific hardware structure schematic diagram of an electronic device provided by the embodiments is shown, which can include a processor 910, a memory 920, an input / output interface 930, a communication interface 940, and a bus 950. The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 for internal communication within the device.

[0179] The processor 910 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0180] The memory 920 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 920 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910.

[0181] The input / output interface 930 is configured to connect an input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0182] The communication interface 940 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB, a network cable, etc.) or through a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0183] The bus 950 includes a channel for transmitting information between various components (such as the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.

[0184] It should be noted that although the above device only shows the processor 910, the memory 920, the input / output interface 930, the communication interface 940, and the bus 950, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for implementing the embodiments of the present specification, and does not have to include all the components shown in the figure.

[0185] The electronic device of the above embodiments is used to implement the corresponding adaptive semantic communication transmission method in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0186] Based on the same inventive concept, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the adaptive semantic communication transmission method according to any of the above embodiments.

[0187] The computer readable media of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0188] The storage medium of the above embodiments stores computer instructions for causing the computer to perform the adaptive semantic communication transmission method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0189] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present disclosure, the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present disclosure as described above. In order to be brief, they are not provided in detail.

[0190] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present disclosure (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe an exemplary embodiment of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations on these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0191] While the present disclosure has been described in connection with certain embodiments thereof, many modifications, substitutions, and variations will be apparent to those of ordinary skill in the art from the foregoing description. For instance, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0192] Embodiments of the disclosure are intended to cover all such alternatives, modifications, and variations as falling within the broad scope of the appended claims. Accordingly, any one or more of the above-described embodiments can be combined with any one or more of the other embodiments, and the application should not be construed as limited to only those embodiments described and shown in the attached drawings and following detailed description.

Claims

1. A method of adaptive semantic communication transmission, wherein, The method is executed based on a transmission model supporting semantic communication, the transmission model comprising a transmitter and a receiver; an encoder in the transmitter comprises a first auxiliary codec module; a decoder in the receiver comprises a second auxiliary codec module; the method comprises: determining at least one of a first model parameter corresponding to the transmitter, a second model parameter corresponding to the receiver, and transmission content through training, to obtain a trained transmission model; taking source data to be transmitted as input, the reconstructed target data corresponding to the source data is obtained at the receiver side through the trained transmission model; determining source data to be transmitted, and establishing a first test data set corresponding to the source data; The method further comprises: based on the first test data set, the semantic representation algorithm parameters in the transmitter are optimized by using the gradient descent method to obtain updated semantic vectors; the semantic vectors are encoded by the encoder to obtain channel symbols; the gradient descent method is used to optimize the channel symbols; and / or, determining channel state information and an adjusted transmission model; wherein the adjusted transmission model comprises: adding a pair of channel modulation networks in the transmission model; the channel modulation network comprises a plurality of channel modulation modules and a plurality of first nonlinear operation layers; the channel modulation module comprises at least one second nonlinear operation layer and a corresponding activation function; wherein a first channel modulation network is added after the first auxiliary codec module in the encoder in the transmitter; a second channel modulation network corresponding to the first channel modulation network is added before the second auxiliary codec module in the decoder in the receiver; the channel state information is used to represent the state of the channel between the transmitter and the receiver; through the adjusted transmission model, the channel state information is used as input to participate in encoding modulation and decoding modulation respectively, to obtain reconstructed target data corresponding to the channel state information.

2. The method of claim 1, wherein, The transmitter comprises an encoder and a nonlinear transformation module; the first model parameter comprises an encoder parameter and a nonlinear transformation parameter; Before determining the first model parameter corresponding to the transmitter through training, the method further comprises: determining source data to be transmitted, and establishing a first test data set corresponding to the source data; The method further comprises: based on the first test data set, the encoder parameters corresponding to the encoder in the transmitter and the nonlinear transformation parameters corresponding to the nonlinear transformation module are optimized by using the gradient descent method.

3. The method of claim 2, wherein, The receiver comprises a decoder and a nonlinear inverse transformation module; the second model parameter comprises a decoder parameter and a nonlinear inverse transformation parameter; The at least one of the first model parameter corresponding to the transmitter, the second model parameter corresponding to the receiver, and the transmission content is determined through training, and the method comprises: Based on the first test data set, the gradient descent method is used to optimize the encoder parameters corresponding to the encoder in the transmitter and the nonlinear transformation parameters corresponding to the nonlinear transformation module; And, the decoder parameters corresponding to the decoder in the receiver and the nonlinear inverse transformation parameters corresponding to the nonlinear inverse transformation module are optimized.

4. The method of claim 3, wherein, Based on the first test data set, the gradient descent method is used to optimize the encoder parameters corresponding to the encoder in the transmitter and the nonlinear transformation parameters corresponding to the nonlinear transformation module; And, the decoder parameters corresponding to the decoder in the receiver and the nonlinear inverse transformation parameters corresponding to the nonlinear inverse transformation module are optimized, comprising: The sampled frames in the first test data set are sent into the transmission model, and the RD loss function value corresponding to the transmission model is calculated; The residual error of the updated parameters of the decoder and the nonlinear inverse transformation after updating and before updating is calculated, the residual error is entropy modeled by using a probability distribution model to obtain the probability distribution of the residual error; Based on the RD loss function value and the probability distribution of the residual error, the RDM loss function of the model is obtained; The gradients of the nonlinear transformation, the encoder, the decoder, and the nonlinear inverse transformation are calculated by using the RDM loss function, and the nonlinear transformation parameters, the encoder parameters, the decoder parameters, and the nonlinear inverse transformation parameters are optimized by using the gradient descent method.

5. The method of claim 1, wherein, The adjusted transmission model is used to participate in encoding modulation and decoding modulation respectively by taking the channel state information as input, comprising: In the encoding modulation, the output of the semantic representation vector to be sent in the transmitter after the first auxiliary codec module in the encoder is taken as input and sent into the nonlinear operation layer located at the first in the plurality of first nonlinear operation layers; the channel state information and the output of the nonlinear operation layer located at the first are sent into the first channel modulation module together; the first channel modulation module is located at the first in the plurality of channel modulation modules and is connected with the output end of the nonlinear operation layer located at the first. In the channel modulation module, the output of the channel state information after the operation of at least one second nonlinear operation layer inside the channel modulation module is multiplied with the output of the nonlinear operation layer located at the first to obtain the output corresponding to the first channel modulation module. The output end of the first channel modulation module is connected with the nonlinear operation layer located at the second in the plurality of first nonlinear operation layers; the output end of the nonlinear operation layer located at the second is connected with the channel modulation module located at the second in the plurality of channel modulation modules. In the decoding modulation, the semantic representation vector obtained by the receiver is taken as input and sent into the nonlinear operation layer located at the first in the plurality of first nonlinear operation layers before being input into the second auxiliary codec module in the decoder.

6. The method of any one of claims 1-5, wherein, the transmission model is an adaptive semantic communication (ASC) model.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1-5 when executing the program.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1-5.

Images