Semantic communication method, system and equipment based on personalized federal learning

Abstract

The invention discloses a semantic communication method, system and equipment based on personalized federated learning. The method comprises the following steps: constructing a federated semantic communication scene of multiple vehicle users; constructing a semantic communication model based on semantic perception vector quantization variational self-encoding; deploying the constructed semantic communication model locally to all vehicle users, and initializing model parameters; an aggregation network is constructed, the aggregation network comprises a super network and an average aggregation network, and the super network is used for generating personalized parameters for a local semantic encoder and a semantic decoder of each vehicle user; the average aggregation network is used for updating parameters except the self-attention layer in the local semantic communication model of each vehicle user; constructing a personalized federated semantic communication model by taking the minimum global loss function as an overall optimization target; and training the personalized federal semantic communication model by adopting a joint training method. The method provided by the invention has excellent adaptability to different channel conditions, and can solve the problem of different structures among users.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and particularly to a semantic communication method, system, and device based on personalized federated learning. Background Art

[0002] In recent years, semantic communication, as a key technology to meet the task requirements in the era of intelligent communication, can overcome the challenges associated with traditional communication systems and has received extensive attention in the industry. Classical semantic communication architectures often assume that the transmitter and receiver share a common database. However, this assumption is difficult to achieve in practical application scenarios due to privacy and other issues. Usually, in actual scenarios, users are only allowed to use local data for training. To address such limitations, the federated learning method has emerged. In this method, local users only update their local models to the central server, thereby achieving the purpose of protecting privacy. Considering the diversity of data sources in the Internet of Vehicles, the semantic communication method based on federated learning provides a new approach for realizing practical and feasible communication.

[0003] In the implementation of the federated learning method, a key problem is the heterogeneity of data among users, also known as the non-independent and identically distributed problem. Under the condition of non-independent and identically distributed, training a single model alone may not effectively capture different data distributions. Personalized federated learning is proposed to solve this problem. It generates personalized models for each user, rather than relying on a single model only. This method can promote the adaptation to the unique data characteristics of different customers and improve the performance of the model in actual scenarios. Current federated learning methods based on Transformer usually regard the attention layer as an independent whole and ignore the potential internal feature connections among users.

[0004] Although there has been increasing attention in recent years on the integration of semantic communication architectures and actual communication models, and significant achievements have been made in semantic communication architectures based on personalized federated learning, there are still many factors in actual scenarios that need to be considered. Otherwise, it is bound to make the model a castle in the air, and once applied in practice, it will lead to problems such as reduced communication reliability and efficiency. Summary of the Invention

[0005] Object of the Invention: The object of the present invention is to provide a semantic communication method based on personalized federated learning (HPFL) to efficiently execute target tasks while considering the diversity of data characteristics among users and privacy issues.

[0006] Technical Solution: The method of the present invention includes the following steps: Construct a federated semantic communication scenario for multiple vehicle users; Construct a semantic communication model based on a semantic-aware vector quantization variational autoencoder, including a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to obtain a discrete feature index. The semantic decoder restores the discrete feature index to a feature vector and performs downstream semantic tasks; Deploy the constructed semantic communication model locally for all vehicle users and initialize the model parameters; Construct an aggregation network, including a hypernetwork and an average aggregation network. The hypernetwork is used to generate personalized parameters for the self-attention layer of the local semantic communication model for each vehicle user. The average aggregation network is used to update the global shared parameters of the local semantic communication models of each vehicle user except for the self-attention layer; Construct a global loss function and minimize the global loss function as the optimization objective. The global loss function is the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork; Adopt a joint training method of the local semantic communication model and the aggregation network to train and obtain an optimal personalized semantic communication model, realizing semantic communication based on personalized federated learning.

[0007] Furthermore, the method of semantic encoding is as follows: According to the cosine similarity matrix of the feature pairs in the input feature vector calculate the importance weight of each feature vector: , where is the importance weight of the th feature vector, is the th feature vector and the th feature vector is the number of feature vectors, corresponding to the number of image patches; Retain the indices of the highest-weight feature vectors , where represents the compression rate. Set the mask positions corresponding to the feature vectors with the retained feature vector indices to 1 and the rest to 0 to obtain a mask matrix ; The mask matrix for the th feature vector is represented as: , Multiply by to obtain the masked feature vector :​ ; And perform semantic encoding operation on to obtain the compressed feature vector : , wherein, is the compressed feature vector, represents semantic encoding, are the parameters of the semantic encoder.

[0008] Furthermore, the semantic decoding process is as follows: Denote the feature vector that the decoder needs to recover as , the input feature vector of the semantic decoder and are mapped to a high-dimensional feature space through the embedding layer, and then input into the attention layer. The attention layer uses the cross-attention mechanism to recover the masked feature vector, and the formula is expressed as: , wherein, represents the multi-head attention mechanism, , , and are respectively the query, key, value and output mapping matrix parameters of the th head of the multi-head attention mechanism, is the number of heads, is the embedding dimension of the head, that is, the feature vector dimension, represents the concatenation operation; and After being processed by the cross-attention mechanism, they are sent to the forward feedback layer; the forward feedback layer processes , and after recovering , the forward feedback layer connects through the mask matrix and ; The output of the forward feedback layer is sent to a prediction head after being normalized, mapped to the original feature space, and all decoding operations are completed. The entire decoding process is expressed as: , wherein, is the output of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

[0009] Furthermore, the hypernetwork includes an embedding layer, a multi-layer perceptron, and a linear layer. The process of generating personalized parameters for the local semantic communication model is as follows: The input of the hypernetwork is the index of each vehicle user. The index is mapped into a low-dimensional vector through the embedding layer, and the low-dimensional vector is then transformed into a high-dimensional feature representation through the multi-layer perceptron. Finally, the feature representation is converted into the query parameter, key parameter, and value parameter of the self-attention layer through the linear layer, which is expressed as: , where represents the hypernetwork, are the parameters of the hypernetwork, is the th parameter of the self-attention layer of the , and are the weight matrices corresponding to the query parameter, key parameter, and value parameter respectively, is the th index of the vehicle user.

[0010] Furthermore, the average aggregation network is used to update the parameters in the local semantic communication model of each vehicle user except for the self-attention layer, and the weighted aggregation method is adopted, which is expressed as: , where is the non-personalized parameter of the local semantic communication model of the th vehicle user, is the updated global parameter, is the data volume of vehicle user , is the total data volume of all vehicle users, is the number of vehicle users.

[0011] Furthermore, constructing a global loss function and minimizing the global loss function as the optimization goal includes: , where represents the parameters of the local semantic communication model, represents the parameters of the hypernetwork, is the data volume of vehicle user , is the total data volume of all vehicle users, is the number of vehicle users, is the reconstruction loss of the semantic encoder and decoder of the local semantic communication model of the th vehicle user, is the hyperparameter used to balance the weight relationship between the reconstruction loss and the discrete codebook loss, is the quantization loss of the discrete codebook for the th local semantic communication model of the vehicle user, is the loss of the hypernetwork; The reconstruction loss is expressed as: , where and represent the output of the semantic decoder and the original image features respectively, represents the number of image patches; The quantization loss of the discrete codebook is expressed as: , where is the basis vector of the normalized discrete codebook, is transpose of, is the compressed feature vector, is the input feature vector of the semantic decoder, represents gradient clipping; The loss of the hypernetwork is expressed as: , where is the parameter generated by the hypernetwork for the vehicle user , is the parameter of the vehicle user after local training.

[0012] Furthermore, a personalized semantic communication model is trained using the joint training method of the local semantic communication model and the aggregation network, including: (1) The hypernetwork generates initialized local semantic communication model parameters for each vehicle user according to the index of the vehicle user; (2) Each vehicle user performs local training using local data based on the initialized local model parameters and updates the local semantic communication model parameters; (3) Calculate the parameter change amount of the self-attention layer in the local semantic communication model according to the initialized parameters of the local semantic communication model of the vehicle user and the parameters of the local semantic communication model of the vehicle user after rounds of local training; (4) Upload the parameter change amounts of all local semantic communication models and the parameters except the self-attention layer to the edge server for aggregation; (5) Based on the parameters of the local semantic communication model of the vehicle user with respect to the parameters of the hypernetwork The gradient of, and the change in the parameters of the self-attention layer in the local semantic communication model, calculate the gradient of the loss of the hypernetwork with respect to the parameters of the hypernetwork : (6) Use the adaptive moment estimation method to update the parameters of the hypernetwork; (7) Use the updated hypernetwork to generate the parameters of the self-attention layer for each vehicle user, and then use the weighted average function to generate the updated other parameters: (8) Distribute the parameters of the self-attention layer and other parameters to all vehicle users for the next large round of training; (9) Repeat steps (2) to (8) until the set number of training times is reached, obtain the global approximate optimal solution, and generate a personalized semantic communication model.

[0013] Further, step (2) includes the following steps: (21) For each training round , the vehicle user randomly extracts B batch feature vectors from the input data for training; where is the total number of training times for local training; (22) Perform semantic perception-based compression masking processing on the input feature vectors to obtain the masked feature vectors ; (23) Send into the semantic encoder to extract semantic information and obtain the compressed feature vectors ; (24) Map to the discrete codebook to obtain the feature index set ; (25) Pass through the time physical channel including the Doppler effect to obtain the noise-affected feature index set ; (26) Compare with the discrete codebook to obtain the corresponding feature vectors ; (27) Send the feature vectors into the semantic decoder for decoding to obtain the output of the semantic decoder ; (28) Calculate the loss function of the local semantic communication model according to the discrete codebook quantization loss and the reconstruction loss; (29) Use the adaptive moment estimation method to update the parameters of the local semantic communication model, and the model parameters update rule is: , wherein, is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, , are correction values, is a constant, is the learning rate of the local semantic communication model; (210) Repeat steps (21) to (29) until all training rounds are completed to obtain an approximate optimal solution of the local semantic communication model.

[0014] The semantic communication system based on personalized federated learning according to the present invention includes: A scenario construction unit for constructing a federated semantic communication scenario for multiple vehicle users; A semantic communication model construction unit for constructing a semantic communication model based on a semantic-aware vector quantization variational autoencoder, including a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector, the discrete codebook maps the feature vector to obtain a discrete feature index, and the semantic decoder restores the discrete feature index into a feature vector and performs downstream semantic tasks; A model deployment unit for deploying the constructed semantic communication model locally on all vehicle users and initializing the model parameters; An aggregation network construction unit for constructing an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication models of each vehicle user except for the self-attention layer; An optimization objective construction unit for constructing a global loss function and taking minimizing the global loss function as the optimization objective. The global loss function is the weighted average of the losses of all local semantic communication models and the sum of the losses of the super network; A joint training unit for training to obtain a personalized semantic communication model by using a joint training method of the local semantic communication model and the aggregation network, so as to implement semantic communication based on personalized federated learning.

[0015] The electronic device according to the present invention includes a memory, a processor, and a computer program / instructions stored on the memory and executable on the processor. When the computer program / instructions are executed by the processor, the steps of the semantic communication method based on personalized federated learning are implemented.

[0016] Beneficial effects: Compared with the prior art, the remarkable technical effects of the present invention are as follows: A task-oriented semantic communication framework based on personalized federated learning is proposed. By simulating the channel and data heterogeneity in the actual situation, a communication method that fits the actual scenario is realized; through the structural design of the aggregation network and generating personalized local models for different users, the optimal performance of the local models is achieved; by introducing a mode of jointly training the local models and the aggregation network with the goal of minimizing the aggregation loss, the approximation to the global optimal parameters is realized. Brief Description of the Drawings

[0017] Figure 1 It is a flowchart of the semantic communication method based on personalized federated learning according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the federated semantic communication scenario of the vehicle networking of the present invention; Figure 3 It is a schematic diagram of the personalized semantic communication model of the present invention. Detailed Embodiments

[0018] The present invention will be described in detail below with reference to the drawings and specific embodiments. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0019] Figure 1 It is a flowchart of a semantic communication method based on personalized federated learning disclosed according to this embodiment. As Figure 1 shown, the semantic communication method based on personalized federated learning includes the following steps: Step 1: Construct a federated semantic communication scenario for multiple vehicle users; Build a vehicle networking federated semantic communication scenario as Figure 2 shown, that is, a federated semantic communication scenario for multiple vehicle users. In the training stage, all intelligent vehicles are within the coverage of the edge server, and the communication between vehicles is realized by a stable semantic communication system. Each vehicle is locally equipped with a semantic encoder and a semantic decoder. During the communication process, the sender uses the semantic encoder to extract the key features of the input data and transmits them to the receiver through the physical channel. The receiver uses the semantic decoder to reconstruct the features and perform downstream tasks. In each communication cycle, the edge server accepts the communication requests of the vehicle users and obtains the set of vehicle users, where is the index of the th vehicle user, , is the number of vehicle users. The edge server generates personalized semantic encoders and decoders for all vehicles by using a hypernetwork and updates the global parameters of the local semantic communication model with an average aggregation network.

[0020] Step 2: Construct a semantic communication model based on a semantic-aware vector quantization variational autoencoder; The semantic-aware vector quantization variational autoencoder, as Figure 3 shown, mainly consists of a semantic encoder, a discrete codebook, and a semantic decoder, and its processing process is as follows: First, the semantic encoder includes a compressor based on semantic importance and semantic encoding. The input image (the image is taken from the local training dataset of each vehicle user) is divided into image patches, and embedded encoding is performed on each patch to obtain the feature vectors of all image patches, that is, the input feature vectors , where is the number of feature vectors, equal to the number of image patches, is the dimension of the feature vector, and the cosine similarity matrix of all feature pairs can be calculated by the following formula: (1), where is the transpose matrix of.

[0021] After obtaining the similarity matrix , the importance weight of each feature vector can be obtained by the following formula: (2), where is the importance weight of the th feature vector, is the th feature vector and the th feature vector

[0022] Then, use the Top-K strategy to retain the indices of the top highest-weight feature vectors, where represents the compression rate. Set the mask positions corresponding to the feature vectors with the retained feature vector indices to 1, and the remaining positions to 0. Then the mask matrix for the th feature vector is represented as: (3), where is the mask matrix corresponding to the th feature vector.

[0023] For the above input feature vectors​ Multiply with the obtained mask matrix to obtain the masked feature vector : (4), After compressing and masking the input feature vector , perform semantic encoding on the obtained masked feature vector . Use a Vision Transformer (ViT) to implement semantic encoding, and the output of semantic encoding can be expressed as: (5), where is the compressed feature vector, represents semantic encoding, are the parameters of the semantic encoder

[0024] After the input image is processed by the above semantic encoder, the compressed feature vector is obtained, where is the th compressed feature vector , and then map it into discrete indices through a discrete codebook. The discrete codebook can be expressed as: (6), where is the discrete codebook, is the th feature vector in the discrete codebook, is the size of the codebook

[0025] After that, use the nearest neighbor search method to correspond each feature vector with the feature vectors in the discrete codebook, as shown in the following formula: (7), where is the index of the feature vector in the discrete codebook corresponding to , is the th feature vector in the discrete codebook. Then the discrete feature index obtained after the above mapping operation on the compressed feature vector can be expressed as: (8), where is the set of feature indices corresponding to the compressed feature vector ​

[0026] The semantic decoder is mainly composed of an attention layer and a feed-forward network, and its input is a set of feature indices output by the discrete codebook. A set of feature indices affected by noise obtained after passing through the channel. : (9), Among them, is the time-varying channel coefficient, is the phase rotation caused by the Doppler frequency shift, is independent and identically distributed (i.i.d) complex Gaussian noise, is the noise variance, is the identity matrix. The Doppler frequency shift can be calculated by the following formula: (10), Among them, is the relative speed of the vehicle, is the speed of light, is the carrier frequency, is the angle of signal arrival.

[0027] Then, by indexing the discrete codebook through the set of feature indices affected by noise the corresponding feature vector can be recovered, that is, the input feature vector of the semantic decoder. Denote the feature vector that the decoder needs to recover as , and are mapped to a high-dimensional feature space through an embedding layer and then fed into the attention layer.

[0028] Different from the self-attention operation, in order to make the decoder focus on recovering the masked feature vector, the attention layer adopts the cross-attention mechanism designed in the present invention, and the specific process can be expressed by the following formula: (11), Among them, represents the multi-head attention mechanism, , , and are the query, key, value, and output mapping matrix parameters of the -th head of the multi-head attention mechanism respectively, is the number of heads, is the embedding dimension of the head, that is, the feature vector dimension, represents the concatenation operation.

[0029] Feature vector and After being processed by the cross-attention mechanism, it is sent to the forward feedback layer. Similarly, in order to make the decoder focus on the recovery of the masked feature vectors, the forward feedback layer only processes . After recovering , the forward feedback layer connects with and through the masking matrix.

[0030] The output of the forward feedback layer is sent to a prediction head after being normalized, mapped to the original feature space, and all decoding operations are completed. The entire decoding process can be represented by the following formula: (12), where is the output of the semantic decoder, is the input feature vector of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

[0031] Step 3: Deploy the semantic communication model constructed in Step 2 on all vehicle users locally and initialize the model parameters.

[0032] Step 4: Construct an aggregation network; as shown in Appendix Figure 3 , the aggregation network consists of a hypernetwork and an average aggregation network.

[0033] Among them, the role of the hypernetwork is to generate personalized parameters for the local semantic encoder and semantic decoder of each vehicle user. Specifically, the hypernetwork consists of an embedding layer, a multi-layer perceptron, and a linear layer. The process of generating personalized parameters for the local semantic communication model is as follows: The input of the hypernetwork is the index of each vehicle user. The index is mapped into a low-dimensional vector through an embedding layer, and this low-dimensional vector is then transformed into a high-dimensional feature representation through a multi-layer perceptron. Finally, a linear layer is used to convert the feature representation into the query (Q), key (K), and value (V) parameters of the self-attention layer. The above process can be represented by the following formula: (13), where represents the hypernetwork, are the parameters of the hypernetwork, is the th vehicle user's self-attention layer parameters, , and are the weight matrices corresponding to the above-mentioned query parameter (Q), key (K), and value (V) parameters respectively, is the Index of a vehicle user.

[0034] The average aggregation network is used to update the parameters in the local semantic communication model of each vehicle user except for the self-attention layer, adopting a weighted aggregation method, and the principle is as follows: (14), where, is the non-personalized parameter of the local semantic communication model of the th vehicle user, is the updated global parameter, is the data volume of vehicle user , is the total data volume of all vehicle users, is the number of vehicle users.

[0035] Step 5, The entire personalized semantic communication model is specifically as Figure 3 shown. Based on the federated semantic communication system, a multi-objective joint optimization problem is proposed, mainly including parameter optimization of the local semantic communication model and parameter optimization of the hypernetwork on the edge server. In addition to parameter optimization of the semantic encoder and decoder, the local semantic communication model also includes optimization of the discrete codebook. Specifically, the optimization objectives of the semantic encoder and decoder are to minimize the reconstruction loss function to restore the original image features and guide the completion of downstream semantic tasks. The optimization objective of the discrete codebook is to minimize the loss function of the discrete codebook so that the basis vectors of the discrete codebook are as close as possible to the output features of the encoder. The optimization objective of the hypernetwork is to minimize the global weighted loss by generating personalized self-attention parameters adapted to the data distribution of each client, so as to achieve efficient personalized federated learning.

[0036] The reconstruction loss adopts the mean square error loss (MSE), which can be expressed as: (15), where, and represent the output of the semantic decoder and the original image features respectively; represents the number of image patches, is the reconstruction loss.

[0037] The discrete codebook quantization loss can be expressed as: (16), where, is the basis vector of the normalized discrete codebook, is the transpose of , is the compressed feature vector, is the input feature vector of the semantic decoder, Indicates the gradient cutoff, which is the quantization loss of the discrete codebook.

[0038] Then the loss of the local semantic communication model is the sum of the quantization loss of the discrete codebook and the reconstruction loss, that is: (17), where is a hyperparameter used to balance the weight relationship between the reconstruction loss and the discrete codebook loss.

[0039] The loss of the hypernetwork is the minimum mean square error loss between the output of the hypernetwork and the local training objective, defined as follows: (18), where is the loss of the hypernetwork, is the parameter generated by the hypernetwork for the vehicle user is the parameter of the vehicle user after local training, is the vehicle user data volume, is the total data volume of all vehicle users. The loss function of the hypernetwork is mainly used to train the hypernetwork and generate personalized weight parameters for the self-attention layer of the local semantic communication model of each vehicle user.

[0040] Finally, the global loss function of personalized federated learning can be defined as the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork, that is: (19), where is the reconstruction loss of the semantic encoder and decoder of the local semantic communication model of the th vehicle user, is the quantization loss of the discrete codebook of the local semantic communication model of the th vehicle user.

[0041] Then the overall optimization goal of personalized federated learning is to minimize the global loss function, which can be expressed as: (20), where represents the parameters of the local semantic communication model, including the semantic encoder, decoder, and discrete codebook, represents the parameters of the hypernetwork.

[0042] The constructed personalized semantic communication model structure is as Figure 3 shown. ​

[0043] Step 6: To solve the above optimization problem, an end-to-end joint training method is proposed, and the specific training process is as follows: Step 6-1: The hypernetwork generates initialized local semantic communication model parameters for each vehicle user according to the index of the vehicle user; Step 6-2: Based on the initialized local model parameters, each vehicle user uses local data (i.e., the local training dataset) for local training and updates the local semantic communication model parameters. The local training steps are as follows: Step 6-2-1: For each training round , the vehicle user randomly extracts B batches of feature vectors from the input data for training. Among them, is the total number of training times for local training.

[0044] Step 6-2-2: Perform semantic-aware compression masking processing on the input feature vector to obtain the masked feature vector .

[0045] Step 6-2-3: Send into the semantic encoder to extract semantic information: (21), Step 6-2-4: Map to the discrete codebook to obtain the feature index set .

[0046] Step 6-2-5: Pass through the time-varying physical channel including the Doppler effect. The input at the receiving end can be expressed as: (22), where is the time-varying channel coefficient, is the phase rotation caused by the Doppler shift, is independent and identically distributed (i.i.d) complex Gaussian noise, is the noise variance, is the identity matrix.

[0047] Step 6-2-6: Compare with the discrete codebook to obtain the corresponding feature vector .

[0048] Step 6-2-7: Send the feature vector into the semantic decoder for decoding: (23), Step 6-2-8: Calculate the loss function of the local semantic communication model as follows: (24), Step 6-2-9: Use the Adaptive Moment Estimation (ADMA) method to update the parameters of the local model. The specific process is as follows: In an example of the present invention, the parameters of the local semantic communication model are set as , is a sample of the local dataset of the th vehicle user. The first moment of the loss function of the local semantic communication model with respect to the model parameters and the codebook is denoted as , and the second moment is denoted as which can be expressed as follows: (25), (26), where and represent the decay rates of the first moment and the second moment respectively, and are the first moment and the second moment of the th round respectively, represents the training of the local semantic communication model in the th round, is the total number of rounds of training of the local semantic communication model, is the gradient calculation for , is the parameter of the local semantic communication model of the th vehicle user during the training in the th round, is the th vehicle user's training loss of the local semantic communication model.

[0049] Through equations (25) and (26), the update rule of the parameters of the local semantic communication model can be obtained as follows: (27), where is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, , is the correction value, is a constant used to avoid division by zero, is the learning rate of the local semantic communication model.

[0050] Step 6-2-10: Repeat Steps 6-2-1 to 6-2-9 until all training rounds are completed to obtain an approximate optimal solution of the local semantic communication model.

[0051] Step 6-3: Calculate the parameter change of the self-attention layer in the local semantic communication model: (28), where, is the change in the parameters of the self-attention layer in the local semantic communication model of the th vehicle user during the training of the th round, is the parameter initialized for the local semantic communication model of the th vehicle user, is the parameter of the local semantic communication model of the th vehicle user after local training in the th round.

[0052] Step 6-4: Upload the and the parameters other than the self-attention layer of all local semantic communication models to the edge server for aggregation.

[0053] Step 6-5: Calculate the gradient of the loss of the hypernetwork with respect to the parameters of the hypernetwork: (29), where, is the gradient of the loss of the hypernetwork with respect to the parameters of the hypernetwork, is the gradient of the parameters of the local semantic communication model of the th vehicle user with respect to the parameters of the hypernetwork, is the data volume of vehicle user , while is the total data volume of all vehicle users, is the set of all vehicle users.

[0054] After obtaining this gradient, the parameters of the hypernetwork are updated using the Adaptive Moment Estimation (ADMA) method as follows: First, the first moment and the second moment of the gradient of the loss of the hypernetwork with respect to the parameters of the hypernetwork can be expressed as: (30), where, is the first moment obtained after the th round of training of the hypernetwork, is the first moment obtained after the th round of training, is the decay factor of the first moment.

[0055] (31), where is the second moment obtained after the th round of training of the hypernetwork, is the second moment obtained after the th round of training, is the decay factor of the second moment.

[0056] Next, the update rule of the parameters of the hypernetwork can be obtained: (32), where is the parameter of the hypernetwork after the th round of training, is the parameter of the hypernetwork after the th round of training, , is the correction value, is the learning rate of the hypernetwork.

[0057] Step 6-7: Use the updated hypernetwork to generate the self-attention layer parameters of each vehicle user , and then use the weighted average function to generate the updated other parameters: (33), where is the non-personalized parameter of the th vehicle user's local semantic communication model, is the updated global parameter, is the vehicle user 's data volume, is the total data volume of all vehicle users, is the number of vehicle users.

[0058] Step 6-8: Distribute the self-attention layer parameters and the other parameters to all vehicle users for the next large round of training.

[0059] Step 6-9: Repeat Steps 6-2 to 6-8 until the set number of training times is reached, obtain the global approximate optimal solution, and generate a personalized semantic communication model.

[0060] The semantic communication method based on personalized federated learning proposed in the embodiments of the present invention proposes a task-oriented semantic communication framework for semantic communication based on personalized federated learning. By designing a joint training mode of the aggregation network and the local network, with the goal of minimizing the loss of the local semantic communication model and the loss of the hypernetwork, while efficiently completing the communication task, it also takes into account some factors that should be considered in the actual application scenario. Compared with the existing communication architectures, the proposed semantic communication method based on personalized federated learning has excellent adaptability to different channel conditions and data heterogeneity, and has a better fit to the actual vehicle-to-vehicle communication scenario.

[0061] The embodiments of the present invention also propose a system corresponding to the method, including: A scenario construction unit for constructing a federated semantic communication scenario for multiple vehicle users; A semantic communication model construction unit for constructing a semantic communication model based on a semantic-aware vector quantization variational autoencoder, including a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector, the discrete codebook maps the feature vector to obtain a discrete feature index, and the semantic decoder restores the discrete feature index into a feature vector and performs downstream semantic tasks; A model deployment unit for deploying the constructed semantic communication model locally on all vehicle users and initializing the model parameters; An aggregation network construction unit for constructing an aggregation network, including a hypernetwork and an average aggregation network. The hypernetwork is used to generate personalized parameters for the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication models of each vehicle user except for the self-attention layer; An optimization objective construction unit for constructing a global loss function and taking minimizing the global loss function as the optimization objective. The global loss function is the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork; A joint training unit for training a personalized semantic communication model by using a joint training method of the local semantic communication model and the aggregation network to realize semantic communication of personalized federated learning.

[0062] The embodiments of the present invention also disclose an electronic device, including a memory, a processor, and a computer program / instructions stored on the memory and executable on the processor. When the computer program / instructions are executed by the processor, the steps of the semantic communication method based on personalized federated learning are implemented.

[0063] An embodiment of the present invention also discloses a computer-readable storage medium storing computer instructions, which are used to execute the steps of the semantic communication method based on personalized federated learning when called.

[0064] An embodiment of the present invention also discloses a computer program product including computer programs / instructions, which implement the steps of the semantic communication method based on personalized federated learning when executed by a processor.

[0065] The method and system of the present invention belong to the field of wireless communication technology, and propose a task-oriented semantic communication framework based on personalized federated learning. By designing a trainable hypernetwork and an average aggregation function, the personalization of the parameters of the local semantic communication model is realized. At the same time, a multi-objective joint training method is proposed to embody the communication process between vehicles. By minimizing the aggregated loss function, an approximate optimal solution of the aggregated network is obtained. Compared with the existing federated learning methods, the proposed scheme has superior adaptability to different channel conditions and can solve the heterogeneity problem among users.

[0066] In the description of this specification, the description referring to terms such as "an embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or N embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0067] Where the present invention is not described in detail, it is all well-known technology to those skilled in the art.

Claims

1. A semantic communication method based on personalized federated learning, characterized in that, It includes the following steps: Construct a federated semantic communication scenario for multi-vehicle users; Construct a semantic communication model based on semantic-aware vector quantization variational autoencoder, including a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to obtain discrete feature indices. The semantic decoder restores the discrete feature indices into a feature vector and performs downstream semantic tasks; Deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters; Construct an aggregation network, including a hypernetwork and an average aggregation network. The hypernetwork is used to generate personalized parameters for the self-attention layer of the local semantic communication model of each vehicle user. The average aggregation network is used to update the global shared parameters of the local semantic communication models of each vehicle user except for the self-attention layer; Construct a global loss function and take minimizing the global loss function as the optimization objective. The global loss function is the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork; Adopt a joint training method of the local semantic communication model and the aggregation network to train and obtain an optimal personalized semantic communication model, realizing semantic communication based on personalized federated learning.

2. The semantic communication method based on personalized federated learning according to claim 1, wherein The method of semantic encoding is: According to the input feature vectors in the cosine similarity matrix of feature pairs, calculate the importance weights of each feature vector: ， Among them, is the importance weight of the th eigenvector, is the cosine similarity between the th eigenvector and the th eigenvector, is the number of eigenvectors, corresponding to the number of image patches; Keep the first indexes of the eigenvectors with the highest weights , where represents the compression rate. Set the mask positions corresponding to the eigenvectors with the reserved indexes to 1 and the rest to 0 to obtain the mask matrix ; The mask matrix for the th eigenvector in the mask matrix is expressed as: ， Multiply with to obtain the masked feature vector : ; And perform semantic encoding operations on to obtain a compressed feature vector : ， Among them, is the compressed feature vector, represents the semantic encoding, is the parameter of the semantic encoder.

3. The semantic communication method based on personalized federated learning according to claim 1, wherein The semantic decoding process is: Denote the feature vector that the decoder needs to recover as , the input feature vector of the semantic decoder and are mapped to a high-dimensional feature space through an embedding layer, and then input into an attention layer. The attention layer uses a cross-attention mechanism to recover the masked feature vector, which is expressed by the formula as: ， Among them, represents the multi-head attention mechanism, , , and are the query, key, value, and output mapping matrix parameters of the th head of the multi-head attention mechanism respectively, is the number of heads, is the embedding dimension of the head, that is, the feature vector dimension, represents the concatenation operation; and After being processed by the cross-attention mechanism, it is sent to the forward feedback layer; the forward feedback layer processes and, after restoring , the forward feedback layer connects through the mask matrix and ; The output of the forward feedback layer is sent to a prediction head after normalization processing, mapped to the original feature space, and all decoding operations are completed. The entire decoding process is expressed as: ， Among them, is the output of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

4. The semantic communication method based on personalized federated learning according to claim 1, wherein The hypernetwork includes an embedding layer, a multi-layer perceptron, and a linear layer. The process of generating personalized parameters for the local semantic communication model is: The input of the hypernetwork is the indices of each vehicle user. The indices are mapped into low-dimensional vectors through the embedding layer. The low-dimensional vectors are then transformed into high-dimensional feature representations through the multi-layer perceptron. Finally, the feature representations are converted into query parameters, key parameters, and value parameters of the self-attention layer through the linear layer, expressed as: ， Among them, represents the hypernetwork, are the parameters of the hypernetwork, is the parameter of the self-attention layer of the th vehicle user, and are the weight matrices corresponding to the query parameter, key parameter, and value parameter respectively, is the index of the th vehicle user.

5. The semantic communication method based on personalized federated learning according to claim 1, characterized in that, The average aggregation network is used to update the parameters of the local semantic communication models of each vehicle user except for the self-attention layer, adopting a weighted aggregation method, expressed as: ， Among them, is the non-personalized parameter of the th local semantic communication model of vehicle users, is the updated global parameter, is the data volume of vehicle users, is the total data volume of all vehicle users, is the number of vehicle users.

6. The semantic communication method based on personalized federated learning according to claim 1, wherein Construct a global loss function and take minimizing the global loss function as the optimization objective, including: ， Among them, represents the parameters of the local semantic communication model, represents the parameters of the hypernetwork, is the vehicle user data volume, is the total data volume of all vehicle users, is the number of vehicle users, is the reconstruction loss of the semantic encoder and decoder of the local semantic communication model of the is a hyperparameter used to balance the weight relationship between the reconstruction loss and the discrete codebook loss, is the discrete codebook quantization loss of the local semantic communication model of the is the loss of the hypernetwork; Reconstruction loss is expressed as: , Among them, and represent the output of the semantic decoder and the original image features respectively, represents the number of image patches; Discrete codebook quantization loss Expressed as: ， Among them, is the basis vector of the normalized discrete codebook, is transpose of is the compressed feature vector, is the input feature vector of the semantic decoder, represents gradient cutoff; Loss of the hypernetwork It is expressed as: ， Among them, are the parameters generated by the super network for vehicle users and are the parameters of vehicle users after local training.

7. The semantic communication method based on personalized federated learning according to claim 1, wherein Adopt a joint training method of the local semantic communication model and the aggregation network to train and obtain a personalized semantic communication model, including: (1) The hypernetwork generates initialized local semantic communication model parameters for each vehicle user according to the indices of the vehicle users; (2) Each vehicle user performs local training using local data based on the initialized local model parameters and updates the local semantic communication model parameters; (3)Parameters initialized according to the local semantic communication model of the vehicle user, and parameters after local training of the local semantic communication model of the vehicle user in the number of rounds of local training, calculate the parameter change amount of the self-attention layer in the local semantic communication model; (4) Upload the parameter change amounts of all local semantic communication models and the parameters except for the self-attention layer to the edge server for aggregation; (5)The gradients of the parameters of the local semantic communication model of the vehicle user with respect to the parameters of the hypernetwork, and the change amount of the parameters of the self-attention layer in the local semantic communication model, calculate the gradients of the loss of the hypernetwork with respect to the parameters of the hypernetwork : : (6) Use the adaptive moment estimation method to update the parameters of the hypernetwork; (7) Use the updated hypernetwork to generate the self-attention layer parameters of each vehicle user, and then use the weighted average function to generate the updated other parameters: (8) Distribute the self-attention layer parameters and other parameters to all vehicle users for the next large round of training; (9) Repeat steps (2) to (8) until the set number of training times is reached, obtain a global approximate optimal solution, and generate a personalized semantic communication model.

8. The semantic communication method based on personalized federated learning according to claim 7, wherein Step (2) includes the following steps: (21) For each training round the vehicle user randomly extracts B batch feature vectors from the input data for training; where is the total number of local training times; (22) Perform semantic-aware compression masking processing on the input feature vector to obtain the masked feature vector ; (23) Send into the semantic encoder to extract semantic information and obtain the compressed feature vector ; (24) Map to a discrete codebook to obtain a set of feature indices ; (25) Obtain a set of feature indices affected by noise through a time physical channel including the Doppler effect ; (26) Compare with the discrete codebook to obtain the corresponding feature vector ; (27) Send the feature vector to the semantic decoder for decoding to obtain the output of the semantic decoder ; (28) Calculate the loss function of the local semantic communication model according to the quantization loss of the discrete codebook and the reconstruction loss. (29) The adaptive moment estimation method is used to update the parameters of the local semantic communication model, and the model parameters The update rule is as follows: ， Among them, is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, is the parameter of the local semantic communication model of the th vehicle user after being updated in the th round of training, , is the correction value, is a constant, is the learning rate of the local semantic communication model; (210) Repeat steps (21) to (29) until all training rounds are completed to obtain an approximate optimal solution of the local semantic communication model.

9. A semantic communication system based on personalized federated learning, characterized in that, It includes: A scenario construction unit for constructing a federated semantic communication scenario for multi-vehicle users. A semantic communication model construction unit for constructing a semantic communication model based on a semantic-aware vector quantization variational autoencoder, including a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector, the discrete codebook maps the feature vector to obtain a discrete feature index, and the semantic decoder restores the discrete feature index into a feature vector and performs downstream semantic tasks. A model deployment unit for deploying the constructed semantic communication model locally on all vehicle users and initializing the model parameters. An aggregation network construction unit for constructing an aggregation network, including a hypernetwork and an average aggregation network. The hypernetwork is used to generate personalized parameters for the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication models of each vehicle user except for the self-attention layer. An optimization objective construction unit for constructing a global loss function and taking minimizing the global loss function as the optimization objective. The global loss function is the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork. A joint training unit for training to obtain a personalized semantic communication model by using a joint training method of the local semantic communication model and the aggregation network, and realizing semantic communication based on personalized federated learning.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program / instructions stored on the memory and executable on the processor. When the computer program / instructions are executed by the processor, the steps of the semantic communication method based on personalized federated learning according to any one of claims 1-8 are implemented.

Images