A semantic communication method based on multi-data adaptive technology

Abstract

The application discloses a semantic communication method based on multi-data adaptive technology, comprising the following steps: S1, channel adaptation is integrated into semantic communication using a multi-data adaptive algorithm, and a transmission model is adapted to low signal-to-noise ratio conditions based on channel conditions during data encoding and decoding, so that transmission effect is achieved; S2, bandwidth decision is created using deep reinforcement learning, and limited bandwidth is selected for data transmission according to the signal-to-noise ratio, data and data type of the current environment; by defining states, actions and rewards, the best bandwidth allocation strategy of the bandwidth decision is obtained by using deep Q learning; so as to adapt to a low bandwidth environment, and greatly improve the transmission effect in a low signal-to-noise ratio and low bandwidth environment; S3, bandwidth allocation of multi-class data transmission is simultaneously decided using deep reinforcement learning, and the adaptability and stability of the system to signal-to-noise ratio and bandwidth changes are reinforced. The transmission effect of multi-data under low signal-to-noise ratio and low bandwidth conditions is improved. The application realizes good low-delay communication.

Description

Technical Field

[0001] The present invention relates to the field of semantic communication in edge intelligence technology, and in particular to a semantic communication method based on multi-data adaptive technology. Background Art

[0002] Deep learning technology has achieved remarkable success in fields such as computer vision, natural language processing, and speech recognition, and is now attracting widespread attention in the field of intelligent communications. However, deep learning models typically require extensive computing resources, limiting their deployment on mobile devices. Summary of the Invention

[0003] The purpose of the present invention is to provide a semantic communication method based on multi-data adaptive technology, which can realize channel adaptation and bandwidth adaptation and improve transmission quality to achieve low-latency communication in an edge intelligent environment.

[0004] The technical solution adopted in the present invention is:

[0005] A semantic communication method based on multi-data adaptive technology, comprising the steps of:

[0006] S1: First, an adaptive semantic communication framework for processing multiple data is built to perform bandwidth-adaptive transmission of data of different types over different channels. The adaptive semantic communication framework consists of an encoder, a decoder, and a deep reinforcement learning network. The encoder and decoder are composed of a feature learning (FL) module and an attention feature (AF) module, respectively. The attention feature (AF) module is embedded in the encoder and alternately connected with the feature learning (FL) module of the encoder.

[0007] S2: semantic vectors extracted from raw data using the adaptive semantic communication framework;

[0008] S3: Based on the original data and signal-to-noise ratio, the adaptive semantic communication framework is trained with deep reinforcement learning (DQN) combined with the extracted semantic vector to obtain the required bandwidth for transmission as output;

[0009] S4: Perform semantic communication transmission under the corresponding bandwidth, and then decode and recover the required data through the decoder.

[0010] Furthermore, the adaptive semantic communication framework in S1 is built using pytorch; multiple data refers to three types of data: images, text, and voice.

[0011] Furthermore, S2 specifically includes the following steps:

[0012] S2-1, the original data is input into the encoder, and the feature learning (FL) module first extracts information from the data.

[0013] S2-2 then fuses the extracted information with the signal-to-noise ratio (SNR) to form contextual information and passes it to the next attention feature (AF) module.

[0014] S2-3, the next attention feature (AF) module further extracts information from the context information and passes it to the next feature learning (FL) module;

[0015] S2-4, repeat the cycle S2-1 to S2-3 four times to obtain the semantic vector extracted from the original data.

[0016] Specifically, the present invention proposes a channel adaptation method. This method is capable of operating under varying signal-to-noise ratios during transmission. It dynamically adjusts the source coding compression ratio and channel coding rate based on the channel signal-to-noise ratio. This is achieved through an attention mechanism. The present invention trains under a range of signal-to-noise ratios to adaptively encode data in different channel environments, improving data transmission quality even in conditions of poor channel quality and low signal-to-noise ratio.

[0017] Furthermore, S3 uses the original data and signal-to-noise ratio as the input state of deep reinforcement learning (DQN), selects different bandwidths of the extracted semantic vectors as the actions of deep reinforcement learning (DQN), and uses the average minimum mean square error, average peak signal-to-noise ratio, structural similarity, and bilingual evaluation substitutes as reward indicators of deep reinforcement learning (DQN); in order to obtain the maximum reward, the deep reinforcement learning (DQN) is trained to obtain the corresponding output, that is, the bandwidth required for transmission.

[0018] Furthermore, step S3 specifically includes the following steps:

[0019] In step S3-1, the bandwidth decision is defined by the tuple (S, A, r), where S is the state set, A is the action set, and r: S×A→R is the reward function; specifically, the state at time step n is defined as:

[0020] Sn={data n , snr n , Speech_n, snr_n}

[0021] Among them, Text_n represents text data, Image_n represents image data, Speech_n represents audio data, and snr_n represents the current signal-to-noise ratio;

[0022] The action set A is defined as follows:

[0023] An={1, 2, 3, ..., k}

[0024] Where k represents the upper limit of the allocable bandwidth;

[0025] Step S3-2, observe the state S at each time step n n ∈S and select action a according to the corresponding strategy π: S→A n ∈A;

[0026] Step S3-3, convert the state to S according to the feedback of data transmission quality n+1 , and receives a reward r n ;

[0027] Step S3-4, performing reinforcement learning on bandwidth decision making with the goal of maximizing the expected sum of rewards;

[0028] In step S3-5, a1, a2, and a3 represent different bandwidths allocated to different data. Different bandwidth limits, i.e., different k values, are allocated to different a values ​​according to the importance of the data in real life. The specific expression is as follows:

[0029] a1={1,2,3…k1}

[0030] a2={1,2,3…k2}

[0031] a3={1,2,3…k3}

[0032] Step S3-6: After the data is transmitted, the transmission quality and the average mean square error Mse are fed back. The higher the transmission quality, the smaller the Mse. The average minimum mean square error Mse of N data is defined as follows:

[0033]

[0034] Mse=a1*mse1+a2*mse2+a3*mse3

[0035] Step S3-7, setting the reward r based on the average minimum mean square error Mse, specifically:

[0036] r = 100-mse

[0037] Step S3-8, in order to learn the optimal allocation strategy, deep Q learning is used, where the network qψ seeks to approximate the Q function Q:S×A→R; the purpose of the Q function is to map each state and action pair to a Q value, which represents the given state and action pair (s n , a n ) is as follows:

[0038]

[0039] Among them, S×A is the set of state and action space, r i represents the reward of step i, s nIndicates the set of the entire state in the nth step, a n represents the set of actions to be performed in the nth step, and n represents the nth step of training;

[0040] Step S3-9, based on the typical deep reinforcement learning (DQN) method, a replay buffer, a target network, and an ε-greedy strategy are used to enhance the learning of the Q function; the replay buffer R stores the experience (s n , a n , r n , s n+1 ), and uniformly sample to update the parameter ψ; the loss function for deep reinforcement learning (DQN) is calculated as follows:

[0041]

[0042] For q ψ , q is the function that the neural network needs to fit, and ψ is the parameter of the function.

[0043] Furthermore, when a single text semantic transmission or image semantic bandwidth transmission is performed, step S3-5 includes the following steps:

[0044] S3-11, obtain the European Parliament's proceedings as a dataset, the dataset is preprocessed into sentences of 4 to 30 words in length, and is split into training data and test data;

[0045] S3-12, using Pytorch to train the adaptive semantic communication framework using the Adam optimizer;

[0046] S3-13, the two trained models are subjected to bandwidth testing, and four groups are divided according to the test signal-to-noise ratio. The bandwidth ratio during transmission is adjusted for each group of experiments, and the transmission quality of the corresponding group is observed with variables.

[0047] Furthermore, the Adam optimizer was trained at a learning rate of 0.0001, and the test signal-to-noise ratios were four groups: snr = 0; snr = 6; snr = 12; snr = 18. The bilingual evaluation substitute BLEU score was used to represent the text transmission quality. For a sent sentence of length l and a decoded sentence of length l, the bilingual evaluation substitute BLEU is expressed as

[0048]

[0049] Furthermore, when a single audio semantic is transmitted, S3-5 includes the following steps:

[0050] S3-21, a speech dataset from the Edinburgh Data Commons, is preprocessed into sentences ranging from 4 to 30 words in length and split into training and test data.

[0051] S3-22, the voice sample is sampled down to 8KHz; the input is a voice sample sequence, s = [s1, s2,..., sW], there are W samples; the voice sample sequence S is divided into m parts as the m training sets for training; in the simulation, W = 6, 384 is fixed; each sample sequence in the training set is composed of frames F = 128, and the frame length L = 128;

[0052] S3-23, the adaptive semantic communication framework is trained by using Pytorch, and the Adam optimizer is used for training;

[0053] S3-24, the trained two groups of models are tested for bandwidth, and according to the test signal-to-noise ratio, they are divided into four groups, and the bandwidth ratio during transmission is adjusted for each group of experiments to observe the transmission quality of the corresponding group as a variable;

[0054] Further, the Adam optimizer is trained at a learning speed of 0.0001, and the test signal-to-noise ratio is divided into snr = 0; snr = 6; snr = 12; snr = 18 four groups; the transmission effect is evaluated by using SSIM (structural similarity), and the SSIM is defined as follows:

[0055] SSIM(x, y) = [l(x, y)] α [c(x, y)] β [s(x, y)] γ

[0056]

[0057]

[0058]

[0059] Wherein, x and y are input images, μ x is the average value of x, μ y is the average value of y, is the variance of x; is the variance of y, σ xy is the covariance of x and y; c1, c2, c3 are constants, and α, β, γ are greater than 0; SSIM is a number between 0 and 1, the greater the number, the smaller the difference between the output image and the undistorted image, that is, the better the image quality. Further, α = β = γ = 1.

[0060] This invention adopts the above technical solutions and introduces channel adaptation and deep reinforcement learning technologies. This architecture improves adaptability to complex communication environments, thereby improving transmission efficiency and quality. In particular, under low signal-to-noise ratio and low bandwidth conditions, this architecture demonstrates superior performance in effectively handling multi-task communications. Therefore, applying this framework to edge computing and combining it with its near-real-time capabilities can further improve transmission efficiency and communication performance. Future research can focus on further optimizing the algorithm, enhancing the adaptability and stability of the system, and promoting its widespread application. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0062] Fig. 1 This is an overall flow chart of a semantic communication method based on multi-data adaptive technology according to an embodiment of the present invention;

[0063] Fig. 2 Schematic diagram of a framework of a semantic communication method based on multi-data adaptive technology according to an embodiment of the present invention;

[0064] Fig. 3 This is a specific flow chart of a semantic communication method based on multi-data adaptive technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0066] The rise of edge intelligence technology has injected new impetus into semantic communication. This technology emphasizes the application of intelligent computing and decision-making in edge devices and networks to reduce latency, improve efficiency and reduce dependence on cloud resources. In this context, semantic communication, as an intelligent communication method, achieves low-latency communication by processing and compressing data at the semantic level, and better adapts to the edge intelligence environment. This paper introduces a semantic communication architecture based on adaptive deep learning, which is designed for transmitting various data types. The architecture combines deep neural networks (DNN) with channel adaptation and bandwidth adaptation methods to improve transmission efficiency and promote natural and efficient information exchange. Semantic transmission of multiple data types such as images, text, and audio is achieved, and deep reinforcement learning (DQN) is used to dynamically allocate bandwidth to optimize bandwidth allocation for different data types to improve transmission quality. Overall, the proposed adaptive deep learning semantic communication architecture is of key significance in the field of edge intelligence, providing technical support for multi-data communication and helping to achieve low-latency communication.

[0067] like Figs. 1-3As shown in one of the figures, the present invention discloses a semantic communication method based on multi-data adaptive technology, which includes the steps of:

[0068] S1: First, an adaptive semantic communication framework for processing multiple data is built to perform bandwidth-adaptive transmission of data of different types over different channels. The adaptive semantic communication framework consists of an encoder, a decoder, and a deep reinforcement learning network. The encoder and decoder are composed of a feature learning (FL) module and an attention feature (AF) module, respectively. The attention feature (AF) module is embedded in the encoder and alternately connected with the feature learning (FL) module of the encoder.

[0069] S2: semantic vectors extracted from raw data using the adaptive semantic communication framework;

[0070] S3: Based on the original data and signal-to-noise ratio, the adaptive semantic communication framework is trained with deep reinforcement learning (DQN) combined with the extracted semantic vector to obtain the required bandwidth for transmission as output;

[0071] S4: Perform semantic communication transmission under the corresponding bandwidth, and then decode and recover the required data through the decoder.

[0072] Furthermore, the adaptive semantic communication framework in S1 is built using pytorch; multiple data refers to three types of data: images, text, and voice.

[0073] Furthermore, S2 specifically includes the following steps:

[0074] S2-1, the original data is input into the encoder, and the feature learning (FL) module first extracts information from the data.

[0075] S2-2 then fuses the extracted information with the signal-to-noise ratio (SNR) to form contextual information and passes it to the next attention feature (AF) module.

[0076] S2-3, the next attention feature (AF) module further extracts information from the context information and passes it to the next feature learning (FL) module;

[0077] S2-4, repeat the cycle S2-1 to S2-3 four times to obtain the semantic vector extracted from the original data.

[0078] Specifically, the present invention proposes a channel adaptation method. This method is capable of operating under varying signal-to-noise ratios during transmission. It dynamically adjusts the source coding compression ratio and channel coding rate based on the channel signal-to-noise ratio. This is achieved through an attention mechanism. The present invention trains under a range of signal-to-noise ratios to adaptively encode data in different channel environments, improving data transmission quality even in conditions of poor channel quality and low signal-to-noise ratio.

[0079] Furthermore, S3 uses the original data and signal-to-noise ratio as the input state of deep reinforcement learning (DQN), selects different bandwidths of the extracted semantic vectors as the actions of deep reinforcement learning (DQN), and uses the average minimum mean square error, average peak signal-to-noise ratio, structural similarity, and bilingual evaluation substitutes as reward indicators of deep reinforcement learning (DQN); in order to obtain the maximum reward, the deep reinforcement learning (DQN) is trained to obtain the corresponding output, that is, the bandwidth required for transmission.

[0080] Furthermore, step S3 specifically includes the following steps:

[0081] In step S3-1, the bandwidth decision is defined by the tuple (S, A, r), where S is the state set, A is the action set, and r: S×A→R is the reward function; specifically, the state at time step n is defined as:

[0082] Sn={data n , snr n , Speech_n, snr_n}

[0083] Among them, Text_n represents text data, Image_n represents image data, Speech_n represents audio data, and snr_n represents the current signal-to-noise ratio;

[0084] The action set A is defined as follows:

[0085] An={1, 2, 3, ..., k}

[0086] Where k represents the upper limit of the allocable bandwidth;

[0087] Step S3-2, observe the state S at each time step n n ∈S and select action a according to the corresponding strategy π: S→A n ∈A;

[0088] Step S3-3, convert the state to S according to the feedback of data transmission quality n+1 , and receives a reward r n ;

[0089] Step S3-4, performing reinforcement learning on bandwidth decision making with the goal of maximizing the expected sum of rewards;

[0090] In step S3-5, a1, a2, and a3 represent different bandwidths allocated to different data. Different bandwidth limits, i.e., different k values, are allocated to different a values ​​according to the importance of the data in real life. The specific expression is as follows:

[0091] a1={1,2,3…k1}

[0092] a2={1,2,3…k2}

[0093] a3={1,2,3…k3}

[0094] Step S3-6: After the data is transmitted, the transmission quality and the average mean square error Mse are fed back. The higher the transmission quality, the smaller the Mse. The average minimum mean square error Mse of N data is defined as follows:

[0095]

[0096] Mse=a1*mse1+a2*mse2+a3*mse3

[0097] Step S3-7, setting the reward r based on the average minimum mean square error Mse, specifically:

[0098] r = 100-mse

[0099] Step S3-8, in order to learn the optimal allocation strategy, deep Q learning is used, where the network qψ seeks to approximate the Q function Q: S×A→R; the purpose of the Q function is to map each state and action pair to a Q value, which represents the given state and action pair (s n , a n ) is as follows:

[0100]

[0101] Among them, S×A is the set of state and action space, r i represents the reward of step i, s n Indicates the set of the entire state in the nth step, a n represents the set of actions to be performed in the nth step, and n represents the nth step of training;

[0102] Step S3-9, based on the typical deep reinforcement learning (DQN) method, a replay buffer, a target network, and an ε-greedy strategy are used to enhance the learning of the Q function; the replay buffer R stores the experience (s n , a n , r n , s n+1 ), and uniformly sample to update the parameter ψ; the loss function for deep reinforcement learning (DQN) is calculated as follows:

[0103]

[0104] For q ψ , q is the function that the neural network needs to fit, and ψ is the parameter of the function.

[0105] Furthermore, when transmitting a single text semantic or image semantic bandwidth, step S3-5 includes the following steps:

[0106] S3-11, obtain the European Parliament's proceedings as a dataset, the dataset is preprocessed into sentences of 4 to 30 words in length, and is split into training data and test data;

[0107] S3-12, using Pytorch to train the adaptive semantic communication framework using the Adam optimizer;

[0108] S3-13, the two trained models are subjected to bandwidth testing, and four groups are divided according to the test signal-to-noise ratio. The bandwidth ratio during transmission is adjusted for each group of experiments, and the transmission quality of the corresponding group is observed with variables.

[0109] Furthermore, the Adam optimizer was trained at a learning rate of 0.0001, and the test signal-to-noise ratios were four groups: snr = 0; snr = 6; snr = 12; snr = 18. The bilingual evaluation substitute BLEU score was used to represent the text transmission quality. For a sent sentence of length l and a decoded sentence of length l, the bilingual evaluation substitute BLEU is expressed as

[0110]

[0111] Furthermore, when a single audio semantic is transmitted, S3-5 includes the following steps:

[0112] S3-21, a speech dataset from the Edinburgh Data Commons, is preprocessed into sentences ranging from 4 to 30 words in length and split into training and test data.

[0113] S3-22, downsample the speech samples to 8 kHz; the input is a speech sample sequence, s = [s1, s2, ..., sW], with W samples; the speech sample sequence S is divided into m parts, which serve as the m training sets; in the simulation, W is fixed at 6,384; each sample sequence in the training set consists of 128 frames, with a frame length of L = 128;

[0114] S3-23, using Pytorch to train the adaptive semantic communication framework using the Adam optimizer;

[0115] S3-24, the two trained models were subjected to bandwidth testing. Four groups were divided according to the test signal-to-noise ratio. The bandwidth ratio during transmission was adjusted for each group of experiments, and the transmission quality of the corresponding group was observed with the variable.

[0116] Furthermore, the Adam optimizer was trained at a learning rate of 0.0001, and the test signal-to-noise ratio was divided into four groups: snr = 0; snr = 6; snr = 12; snr = 18. The transmission effect was evaluated using SSIM (structural similarity), which is defined as follows:

[0117] SSIM(x,y)=[l(x,y)] α [c(x, y)] β [s(x, y)] γ

[0118]

[0119]

[0120]

[0121] Where x and y are the input images, μ x is the mean value of x, μ y is the mean value of y, is the variance of x; is the variance of y, σ xy is the covariance of x and y; c1, c2, and c3 are all constants, and α, β, and γ are all greater than 0. SSIM is a number between 0 and 1; a larger value indicates a smaller difference between the output image and the undistorted image, i.e., better image quality. Furthermore, α = β = γ = 1.

[0122] The specific working principle of the present invention is described in detail below: Fig. 1 , embodiment 1 of the present invention is:

[0123] The present invention provides a semantic communication method based on multi-data adaptive technology. By introducing channel adaptation and deep reinforcement learning technology, the architecture improves adaptability to complex communication environments and improves transmission efficiency and quality. In particular, under low signal-to-noise ratio and low bandwidth conditions, the architecture demonstrates superior performance in effectively handling multi-task communications. Therefore, applying this framework to the field of edge computing and combining it with its near real-time performance can further improve transmission efficiency and communication performance. Future research can focus on further optimizing the algorithm, enhancing the adaptability and stability of the system, and promoting its widespread application.

[0124] like Fig. 1 As shown, a semantic communication method based on multi-data adaptive technology in this embodiment includes the following steps:

[0125] S1. Integrate channel adaptation into semantic communication, taking channel conditions into account when encoding and decoding data, so that the transmission model can better adapt to low signal-to-noise ratio conditions, and improve transmission effects;

[0126] S2, using a deep reinforcement learning (DQN) bandwidth allocation decision module, further integrates bandwidth adaptation into semantic communication, enabling the model to not only adapt to low signal-to-noise ratios but also better adapt to low bandwidth environments, greatly improving the transmission effect of the model in low signal-to-noise ratio and low bandwidth environments;

[0127] S3, using DQN to simultaneously decide the bandwidth allocation problem of multi-class data transmission, reinforcing the adaptability and stability of the system to signal-to-noise ratio and bandwidth changes, and improving the transmission effect of multi-data in low signal-to-noise ratio and low bandwidth conditions.

[0128] It is noted that the present application provides a semantic communication method based on multi-data adaptive technology. By introducing channel adaptation and deep reinforcement learning technology, the architecture significantly improves the adaptability to complex communication environments, thereby improving the transmission efficiency and quality. Notably, under low signal-to-noise ratio and low bandwidth conditions, the architecture exhibits excellent performance and effectively handles multi-task communication.

[0129] That is, in the present example, the framework is applied to the field of edge computing and combined with its proximity and real-time performance, which is expected to further improve transmission efficiency and communication performance. Future research directions can focus on further optimizing algorithms, enhancing the adaptability and stability of the system, and promoting its application in a wide range of fields.

[0130] Please refer to Figs. 1-3 Embodiment two of the present application is:

[0131] Based on the above-mentioned embodiment one, in the semantic communication method based on multi-data adaptive technology of the present embodiment, the data adaptive algorithm is based on four indicators: average minimum mean square error, average peak signal-to-noise ratio, structural similarity, and bilingual evaluation substitution.

[0132] In the step S1 of the above-mentioned embodiment one, the classification using the data adaptive algorithm in data decoding and encoding specifically includes the following steps:

[0133] The present application proposes a channel adaptation method. The method can work at different signal-to-noise ratio levels during transmission. It dynamically adjusts the compression ratio of the signal source coding and the channel coding rate according to the channel signal-to-noise ratio. This is achieved through an attention mechanism.

[0134] The present application uses an AF module to process the signal-to-noise ratio (SNR) feedback from the channel, and then embeds the AF module into the encoder and connects it alternately with the FL module of the encoder.

[0135] The encoder and the decoder of the application are respectively composed of a feature learning (FL) module and an attention feature (AF) module. The FL module first extracts information from the data, then fuses it with the SNR to form context information, and provides it to the next AF module. Then, taking the context information as input, the attention mask of the FL feature is generated in the factor prediction network, and the FL feature is scaled according to the attention weight. The output of each AF module is then fed back to the next FL module. In this way, 4 cycles of processing are performed to obtain the final scaled data.

[0136] Training this structure in a certain range of signal-to-noise ratio makes it adaptive to encode data in different channel environments, and realizes the improvement of data transmission quality in the case of poor channel quality and low signal-to-noise ratio. In the experiment, the channel adaptive model is compared with three models trained in a fixed signal-to-noise ratio to prove the superiority of its performance (especially in the case of poor channel quality).

[0137] That is, in the embodiment, the transmission quality of the data is improved in the case of poor channel quality and low signal-to-noise ratio.

[0138] In the embodiment, step S2 further includes:

[0139] In the traditional semantic communication, the encoder encodes all the data and then transmits all of them. This traditional full-bandwidth transmission can restore the real data as much as possible. However, in the case of poor channel quality, all the data will be transmitted, and not all the data is important, so it is not possible to effectively use the limited bandwidth resources to transmit more important data. The application proposes to use deep reinforcement learning to create a bandwidth decision, and select limited bandwidth for data transmission by learning the channel signal-to-noise ratio SNR, data and data type in the current environment.

[0140] The application divides the available bandwidth allocation of the data as a decision-making process, and uses deep reinforcement learning to solve the optimal bandwidth allocation strategy. The decision module is defined by a tuple (S, A, r), where S is the state set, A is the action set, and r: SxA→R is the reward function. In each time step n, the state S n ∈S is observed, and an action a n ∈A is selected according to its policy π: S→A. Then, according to the feedback of the data transmission quality, the state is converted to S n+1 , and a reward r n is received. The goal of this reinforcement learning network is to maximize the expected sum of returns.

[0141] In the dynamic bandwidth allocation problem, the state of the time step n is defined as

[0142] Sn={data n , snr n , Speech_n, snr_n}.

[0143] Where Text_n represents text data, Image_n represents image data, Speech_n represents audio data, and snr_n represents the current signal-to-noise ratio

[0144] The present invention defines the action set A as follows, where k represents the upper limit of the allocable bandwidth.

[0145] An={1,2,3…k}

[0146] Where a1, a2, and a3 represent different bandwidths allocated to different data. In addition, the present invention allocates different bandwidth upper limits, i.e., different k values, to different a values ​​according to the importance of data in real life applications.

[0147] a1={1,2,3…k1}

[0148] a2={1,2,3…k2}

[0149] a3={1,2,3…k3}

[0150] The present invention obtains transmission quality and mean square error (MSE) as feedback after data transmission. The higher the transmission quality, the smaller the MSE.

[0151] The average minimum mean square error mse of N data is defined as follows:

[0152]

[0153] Mse=a1*mse1+a2*mse2+a3*mse3

[0154] The smaller Mse is, the greater the reward is. The present invention defines the reward r as:

[0155] r = 100-mse

[0156] To learn the optimal allocation strategy, we use deep Q-learning, where the network qψ seeks to approximate the Q-function Q: S×A→R. The purpose of the Q-function is to map each state and action pair to a Q-value, which represents the state and action pair (s n , a n ) is the total discounted reward for step n. It is as follows:

[0157]

[0158] The application adopts a typical DQN method, adopts a replay buffer, a target network and an epsilon-greedy policy to enhance the learning of a Q function. n n n n+1 ), and uniformly samples to update the parameters ψ.The application adopts the following loss function to calculate the loss of DQN.

[0159]

[0160] That is, in the embodiment, a bandwidth decision is created using deep reinforcement learning, and by learning the channel signal-to-noise ratio SNR, data and data type in the current environment, a limited bandwidth is selected, and more effective data transmission can be performed.

[0161] In step S3, the following steps are further included:

[0162] The application performs text, image and audio semantic transmission experiments. For text, the PyTorch training model is used, the Adam optimizer and the learning speed of 0.0001 are used, and the data set is the proceedings of the European Parliament. The image and audio experiments use similar deep learning frameworks and Adam optimizers, the image data set is the same as the text, and the audio data set is from the Edinburgh data sharing. The transmission effect is evaluated using SSIM (structural similarity), wherein the larger the SSIM value is, the smaller the gap between the output and the distortion-free image or audio is, that is, the better the transmission effect is.

[0163] The step S3 of the above embodiment is specifically as follows:

[0164] For a single text semantic transmission experiment, the application uses Pytorch to train the model of the application, and uses the Adam optimizer to train at a learning speed of 0.0001. The data set used is the proceedings of the European Parliament, which includes about 2 million sentences and 53 million words. The data set is preprocessed into a sentence length of 4 to 30 words, and is divided into training data and test data.

[0165] For the image semantic bandwidth transmission experiment, the deep learning framework and the experimental data set used by the application are basically the same as those in the channel adaptation part above. Similarly, the Adam optimizer is used to train at a learning speed of 0.0001. The same bandwidth decision is still used for bandwidth selection.

[0166] ​​For a single audio semantic transmission experiment, the present invention uses a speech dataset from the Edinburgh Data Commons, which includes a training set of over 10,000 .wav files and a test set of over 800 .wav files, with a sampling rate of 16 kHz. The present invention downsamples the speech samples to 8 kHz. The present invention's input is a speech sample sequence, s = [s1, s2, ..., sW], with W samples. S is constructed as a trainable size m. In the simulation, the present invention fixes W = 16,384, and each sample sequence in m consists of frames F = 128, with a frame length L = 128. The same experimental model is adopted as for text and images.

[0167] The transmission effect of images and audio is evaluated by SSIM (structural similarity), which is defined as follows:

[0168] SSIM(x,y)=[l(x,y)] α [c(x, y)] β [s(x, y)] γ

[0169]

[0170]

[0171]

[0172] Among them, a is brightness comparison, b is contrast comparison, and c is structure comparison. a and b represent the mean of x and y respectively, a and b represent the standard deviation of x and y respectively. a represents the covariance of x and y. And c is a constant

[0173] SSIM is a number between 0 and 1. The larger the SSIM is, the smaller the difference between the output image and the undistorted image is, that is, the better the image quality is.

[0174] The present invention adopts the above technical solution, introduces the concept of semantic communication, and provides a new way to provide low-latency communication in an edge computing environment through semantic-level data compression. However, under low signal-to-noise ratio and low bandwidth conditions, the transmission effect of semantic communication is limited. To solve this problem, the present invention proposes a semantic communication architecture based on adaptive deep learning, which makes full use of the synergy of technologies such as deep neural networks, channel adaptation and bandwidth adaptation. In this architecture, efficient transmission of multiple data types is achieved through the synergy of deep neural networks. Deep reinforcement learning (DQN) is introduced to dynamically allocate bandwidth and determine the optimal data type bandwidth allocation strategy according to different signal-to-noise ratio conditions. This enables higher transmission quality to be achieved under lower bandwidth conditions, significantly improves bandwidth utilization, and improves transmission quality under various channel conditions.

[0175] Overall, the adaptive deep learning semantic communication architecture proposed by the application has important significance in the field of multi-data communication. Through a series of experiments, it is proved that under the condition of low bandwidth and low signal-to-noise ratio, the architecture can realize better low-delay communication, provide fast high-performance low-delay service in the edge computing environment, and thus improve the user experience.

[0176] Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

Claims

1. A semantic communication method based on multi-data adaptive technology, characterized by: It includes the steps of: S1: First, an adaptive semantic communication framework for processing multiple data types is built to adaptively transmit data of different types across different channels. The adaptive semantic communication framework consists of an encoder, a decoder, and a deep reinforcement learning network. The encoder and decoder are composed of a feature learning module and an attention feature module, respectively. The attention feature module is embedded in the encoder and alternately connected with the encoder's feature learning module. Multiple data refers to three types of data: images, text, and speech. S2: semantic vectors extracted from raw data using the adaptive semantic communication framework; S3: Use the original data and signal-to-noise ratio as the input state of deep reinforcement learning, select different bandwidths of the extracted semantic vectors as deep reinforcement learning actions, and use average minimum mean square error, average peak signal-to-noise ratio, structural similarity, and bilingual evaluation as deep reinforcement learning reward indicators. In order to maximize the reward, the deep reinforcement learning is trained to obtain the corresponding output, that is, the required transmission bandwidth. S4: Perform semantic communication transmission under the corresponding bandwidth, and then decode and recover the required data through the decoder.

2. The semantic communication method based on multi-data adaptive technology according to claim 1, characterized in that: The adaptive semantic communication framework in S1 is built using pytorch.

3. The semantic communication method based on multi-data adaptive technology according to claim 1, characterized in that: S2 specifically includes the following steps: S2-1, the original data is input into the encoder, and the feature learning module first extracts information from the data. S2-2, then fuses the extracted information with the signal-to-noise ratio (SNR) to form context information and passes it to the next attention feature module. S2-3, the next attention feature module further extracts information from the context information and passes it to the next feature learning module. S2-4, repeats the cycle S2-1 to S2-3 four times to obtain the semantic vector extracted from the original data.

4. The semantic communication method based on multi-data adaptive technology according to claim 1, characterized in that: Step S3 specifically includes the following steps: In step S3-1, the bandwidth decision is defined by the tuple (S, A, r), where S is the state set, A is the action set, and r:S×A→R is the reward function; specifically, the state at time step n is defined as: Sn={Text_n,Image_n,Speech_n,snr_n} Among them, Text_n represents text data, Image_n represents image data, Speech_n represents audio data, and snr_n represents the current signal-to-noise ratio; The action set A is defined as follows: An={1,2,3,...,k} Where k represents the upper limit of the allocable bandwidth; Step S3-2, observe the state S at each time step n n ∈S and select action a according to the corresponding strategy π:S→A n ∈A; Step S3-3, convert the state to S according to the feedback of data transmission quality n+1 , and receives a reward r n ; Step S3-4, performing reinforcement learning on bandwidth decision making with the goal of maximizing the expected sum of rewards; In step S3-5, a1, a2, and a3 represent different bandwidths allocated to different data. Different bandwidth limits, i.e., different k values, are allocated to different a values ​​according to the importance of the data in real life. The specific expression is as follows: a1={1,2,3...k1} a2={1,2,3...k2} a3={1,2,3...k3} Step S3-6: After the data is transmitted, the transmission quality and the average mean square error Mse are fed back. The higher the transmission quality, the smaller the Mse. The average minimum mean square error Mse of N data is defined as follows: Mse=a1*mse1+a2*mse2+a3*mse3 Step S3-7, setting the reward r based on the average minimum mean square error Mse, specifically: r = 100-mse Step S3-8, in order to learn the optimal allocation strategy, deep Q learning is used, where the network qψ seeks to approximate the Q function Q:S×A→R; the purpose of the Q function is to map each state and action pair to a Q value, which represents the given state and action pair (s n , a n ) is as follows: Step S3-9, based on the typical deep reinforcement learning method, a replay buffer, a target network and an ε-greedy strategy are used to enhance the learning of the Q function; the replay buffer R stores the experience (s n , a n , r n , s n+1 ), and uniformly sample to update the parameter ψ; the loss function for deep reinforcement learning is calculated as follows:

5. The semantic communication method based on multi-data adaptive technology according to claim 4, characterized in that: When a single text semantic transmission or image semantic bandwidth transmission is performed, step S3-5 includes the following steps: S3-11, obtain the European Parliament's proceedings as a dataset, the dataset is preprocessed into sentences of 4 to 30 words in length, and is split into training data and test data; S3-12, using Pytorch to train the adaptive semantic communication framework using the Adam optimizer; S3-13, the two trained models are subjected to bandwidth testing, and four groups are divided according to the test signal-to-noise ratio. The bandwidth ratio during transmission is adjusted for each group of experiments, and the transmission quality of the corresponding group is observed with variables.

6. The semantic communication method based on multi-data adaptive technology according to claim 5, characterized in that: The Adam optimizer was trained at a learning rate of 0.0001, and the test signal-to-noise ratios were four groups: snr = 0; snr = 6; snr = 12; and snr = 18. The bilingual evaluation substitute BLEU score was used to represent the text transmission quality. For a sent sentence of length l and a decoded sentence of length l, the bilingual evaluation substitute BLEU is expressed as 7. The semantic communication method based on multi-data adaptive technology according to claim 4, characterized in that: When transmitting a single audio semantic, S3-5 includes the following steps: S3-21, a speech dataset from the Edinburgh Data Commons, is preprocessed into sentences ranging from 4 to 30 words in length and split into training and test data. S3-22, downsample the speech samples to 8 kHz; the input is a speech sample sequence, s = [s1, s2, ..., sW], with W samples; the speech sample sequence S is divided into m parts, which serve as the m training sets; in the simulation, W is fixed at 6,384; each sample sequence in the training set consists of 128 frames, with a frame length of L = 128; S3-23, using Pytorch to train the adaptive semantic communication framework using the Adam optimizer; S3-24, the two trained models are subjected to bandwidth testing, and four groups are divided according to the test signal-to-noise ratio. The bandwidth ratio during transmission is adjusted for each group of experiments, and the transmission quality of the corresponding group is observed with variables.

8. The semantic communication method based on multi-data adaptive technology according to claim 7, characterized in that: The Adam optimizer was trained at a learning rate of 0.0001, and the test signal-to-noise ratio was divided into four groups: snr = 0; snr = 6; snr = 12; snr = 18. The transmission effect was evaluated using the structural similarity SSIM, which is defined as follows: SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ Where x and y are the input images, μ x is the mean value of x, μ y is the mean value of y, is the variance of x; is the variance of y, σ xy is the covariance of x and y; c1, c2, c3 are all constants, α, β, γ are all greater than 0; SSIM is a number between 0 and 1. The larger it is, the smaller the difference between the output image and the undistorted image, that is, the better the image quality.

9. The semantic communication method based on multi-data adaptive technology according to claim 8, characterized in that: α=β=γ=1.

Images