Semantic communication method, system and device based on personalized federated learning

Abstract

The present invention discloses a semantic communication method, system, and device based on personalized federated learning. The method includes: constructing a federated semantic communication scenario for multiple vehicle users; building a semantic communication model based on semantic-aware vector quantization variational autoencoding; deploying the constructed semantic communication model locally on all vehicle users and initializing the model parameters; constructing an aggregation network, including a supernetwork and an average aggregation network, wherein the supernetwork is used to generate personalized parameters for the local semantic encoder and semantic decoder of each vehicle user; and updating the parameters of the local semantic communication model of each vehicle user, except for the self-attention layer, on the average aggregation network; constructing a personalized federated semantic communication model with minimizing the global loss function as the overall optimization goal; and training the personalized federated semantic communication model using a joint training method. The method of the present invention has excellent adaptability to different channel conditions and can solve the problem of heterogeneity between users.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a semantic communication method, system and device based on personalized federated learning. Background Art

[0002] In recent years, semantic communication has garnered widespread industry attention as a key technology for meeting the demands of the intelligent communications era, overcoming the challenges associated with traditional communication systems. Classic semantic communication architectures often assume that both the sender and receiver share a common database. However, this assumption is difficult to implement in real-world scenarios due to privacy concerns. Typically, users are only allowed to use local data for training. To address these limitations, federated learning methods have emerged. In this method, local users only update their local models to a central server, thereby protecting privacy. Given the diverse data sources in the connected vehicle (IoV), semantic communication methods based on federated learning offer a new approach to achieving practical communication.

[0003] A key issue in implementing federated learning methods is the heterogeneity of data across users, also known as the non-IID problem. Under non-IID conditions, training a single model may not effectively capture the diverse data distributions. Personalized federated learning was proposed to address this issue by generating a personalized model for each user, rather than relying on a single model. This approach facilitates adaptation to the unique data characteristics of different customers and improves model performance in real-world scenarios. Current Transformer-based federated learning methods typically treat the attention layer as a single entity, ignoring the potential inherent feature connections between users.

[0004] Although people have paid more and more attention to the integration of semantic communication architecture and actual communication models in recent years, and the semantic communication architecture based on personalized federated learning has also achieved remarkable results, there are still many factors in actual scenarios that need to be considered. Otherwise, the model will inevitably become a castle in the air. Once applied in practice, it will lead to problems such as reduced communication reliability and efficiency. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a semantic communication method based on personalized federated learning (HPFL) to achieve efficient execution of target tasks while considering the diversity of data features and privacy issues among users.

[0006] Technical solution: The method of the present invention comprises the following steps:

[0007] Build a federated semantic communication scenario for multiple vehicle users;

[0008] A semantic communication model based on a semantic-aware vector quantized variational autoencoder is constructed, which includes a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to discrete feature indexes. The semantic decoder recovers the discrete feature indexes into feature vectors and performs downstream semantic tasks.

[0009] Deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters;

[0010] Construct an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication model of each vehicle user, except for the self-attention layer;

[0011] Construct a global loss function and minimize the global loss function as the optimization goal. The global loss function is the weighted average of all local semantic communication model losses and the sum of the hypernetwork losses.

[0012] The local semantic communication model and the aggregation network joint training method are used to train the optimal personalized semantic communication model, realizing semantic communication based on personalized federated learning.

[0013] Furthermore, the semantic encoding method is:

[0014] According to the input feature vector The cosine similarity matrix of the feature pairs in , calculates the importance weight of each feature vector:

[0015] ,

[0016] in, It is The importance weight of the feature vector, It is The eigenvector and The cosine similarity between feature vectors, is the number of eigenvectors, corresponding to the number of image patches;

[0017] Before Retention The highest weighted eigenvector index ,in Indicates the compression ratio, the feature vector index to be retained The mask position corresponding to the corresponding eigenvector is set to 1, and the remaining positions are 0, and the mask matrix is ​​obtained. ; Mask matrix Middle The mask matrix corresponding to the eigenvectors Expressed as:

[0018] ,

[0019] Will and Multiply to get the masked feature vector : ;

[0020] And Perform semantic encoding operation to obtain compressed feature vector :

[0021] ,

[0022] in, is the compressed feature vector, Represents semantic encoding, are the parameters of the semantic encoder.

[0023] Furthermore, the semantic decoding process is:

[0024] The feature vector that the decoder needs to recover is recorded as , the input feature vector of the semantic decoder and It is mapped to a high-dimensional feature space through the embedding layer and then input into the attention layer. The attention layer uses a cross-attention mechanism to restore the masked feature vector. The formula is expressed as:

[0025] ,

[0026] in, represents the multi-head attention mechanism, 、 、 and They are the first The query, key, value and output mapping matrix parameters of each header, is the number of heads, is the embedding dimension of the head, i.e. the feature vector dimension, Indicates a connection operation;

[0027] and After being processed by the cross attention mechanism, it is sent to the forward feedback layer; the forward feedback layer To process, After recovery, the forward feedback layer passes through the mask matrix Will and connect;

[0028] The output of the forward feedback layer is normalized and sent to a prediction head, mapped to the original feature space, and all decoding operations are completed. The entire decoding process is expressed as:

[0029] ,

[0030] in, is the output of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

[0031] Furthermore, the hypernetwork includes an embedding layer, a multilayer perceptron, and a linear layer, and the process of generating personalized parameters for the local semantic communication model is as follows:

[0032] The input of the hypernetwork is the index of each vehicle user. The index is mapped into a low-dimensional vector through the embedding layer. The low-dimensional vector is then converted into a high-dimensional feature representation through the multi-layer perceptron. Finally, the feature representation is converted into the query parameter, key parameter, and value parameter of the self-attention layer through the linear layer, which is expressed as:

[0033] ,

[0034] in, represents the super network, are the parameters of the hypernetwork, It is The parameters of the self-attention layer for each vehicle user, 、 and are the weight matrices corresponding to query parameters, key parameters, and value parameters, respectively. It is The index of the vehicle user.

[0035] Furthermore, the average aggregation network is used to update the parameters of the local semantic communication model of each vehicle user except the self-attention layer, using a weighted aggregation method, which is expressed as:

[0036] ,

[0037] in, It is Non-personalized parameters of the local semantic communication model for vehicle users, is the updated global parameter, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users.

[0038] Furthermore, a global loss function is constructed and minimizing the global loss function is used as the optimization goal, including:

[0039] ,

[0040] in, represents the parameters of the local semantic communication model, represents the parameters of the hypernetwork, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users, For the The reconstruction loss of the semantic encoder and decoder of the local semantic communication model for each vehicle user, is a hyperparameter used to balance the weight relationship between reconstruction loss and discrete codebook loss, For the The discrete codebook quantization loss of the local semantic communication model for vehicle users, is the loss of the super network;

[0041] Reconstruction losses Expressed as: ,

[0042] in, and Represent the output of the semantic decoder and the original image features, respectively. Represents the number of image blocks;

[0043] Discrete codebook quantization loss Expressed as:

[0044] ,

[0045] in, is the basis vector of the normalized discrete codebook, for The transpose of is the compressed feature vector, is the input feature vector of the semantic decoder, represents the gradient cutoff;

[0046] Hypernetwork Loss Expressed as:

[0047] ,

[0048] in, It is a super network for vehicle users The generated parameters, Is a vehicle user Parameters after local training.

[0049] Furthermore, a personalized semantic communication model is obtained by jointly training the local semantic communication model and the aggregated network, including:

[0050] (1) The super network generates initialized local semantic communication model parameters for each vehicle user based on the vehicle user’s index;

[0051] (2) Each vehicle user uses local data to perform local training based on the initialized local model parameters and updates the local semantic communication model parameters;

[0052] (3) According to the parameters initialized by the local semantic communication model of the vehicle user, and the local semantic communication model of the vehicle user in Calculate the parameter changes of the self-attention layer in the local semantic communication model after the parameters of the local training round;

[0053] (4) The parameter changes of all local semantic communication models and the parameters except the self-attention layer Upload to edge server for aggregation;

[0054] (5) Parameters of the local semantic communication model based on vehicle users with respect to the parameters of the hypernetwork The gradient of , as well as the parameter change of the self-attention layer in the local semantic communication model, calculate the loss of the hypernetwork with respect to the parameters of the hypernetwork Gradient:

[0055] (6) Update the parameters of the hypernetwork using the adaptive moment estimation method;

[0056] (7) Use the updated hypernetwork to generate the self-attention layer parameters for each vehicle user, and then use the weighted average function to generate the updated other parameters:

[0057] (8) Distribute the self-attention layer parameters and other parameters to all vehicle users for the next large round of training;

[0058] (9) Repeat steps (2) to (8) until the set number of training times is reached, and a global approximate optimal solution is obtained to generate a personalized semantic communication model.

[0059] Furthermore, step (2) includes the following steps:

[0060] (21) For each training round , the vehicle user randomly extracts B batch feature vectors from the input data Conduct training; among them, is the total number of training times for local training;

[0061] (22) Input feature vector Perform compression mask processing based on semantic perception to obtain the masked feature vector ;

[0062] (23) Send it to the semantic encoder to extract semantic information and obtain the compressed feature vector ;

[0063] (24) Map with the discrete codebook to obtain the feature index set ;

[0064] (25) Through the physical channel including the Doppler effect, the feature index set affected by noise is obtained ;

[0065] (26) Comparing with the discrete codebook, we get The corresponding eigenvector ;

[0066] (27) The eigenvector Send it to the semantic decoder for decoding and get the output of the semantic decoder ;

[0067] (28) Calculate the loss function of the local semantic communication model based on the discrete codebook quantization loss and reconstruction loss;

[0068] (29) Adaptive moment estimation method is used to update the parameters of the local semantic communication model. The model parameters The update rules are:

[0069] ,

[0070] in, It is The local semantic communication model parameters of vehicle users are The parameters after round training are updated, For the The local semantic communication model parameters of vehicle users are The parameters after round training are updated, 、 is the correction value, is a constant, is the learning rate of the local semantic communication model;

[0071] (210) Repeat steps (21) to (29) until all training rounds are completed and an approximately optimal solution for the local semantic communication model is obtained.

[0072] The semantic communication system based on personalized federated learning described in the present invention includes:

[0073] Scenario construction unit, used to build federated semantic communication scenarios for multiple vehicle users;

[0074] A semantic communication model construction unit is used to build a semantic communication model based on a semantic-aware vector quantization variational autoencoder. The model includes a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to a discrete feature index. The semantic decoder restores the discrete feature index to a feature vector and performs downstream semantic tasks.

[0075] Model deployment unit, used to deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters;

[0076] An aggregation network construction unit is used to construct an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication model of each vehicle user except the self-attention layer;

[0077] An optimization target construction unit is used to construct a global loss function and minimize the global loss function as the optimization target. The global loss function is the weighted average of all local semantic communication model losses and the sum of the hypernetwork losses.

[0078] The joint training unit is used to train a personalized semantic communication model using a local semantic communication model and an aggregate network joint training method, thereby realizing semantic communication based on personalized federated learning.

[0079] The electronic device described in the present invention includes a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor. When the computer program / instruction is executed by the processor, the steps of the semantic communication method based on personalized federated learning are implemented.

[0080] Beneficial effects: Compared with the existing technology, the significant technical effects of the present invention are: proposing a task-oriented semantic communication framework based on personalized federated learning, and realizing a communication method that fits the actual scenario by simulating the channel and data heterogeneity in actual situations; by designing the structure of the aggregation network and generating personalized local models for different users, the optimal performance of the local model is achieved; by introducing a mode of joint training of the local model and the aggregation network, with the goal of minimizing the aggregation loss, the global optimal parameters are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 Flowchart of a semantic communication method based on personalized federated learning according to an embodiment of the present invention;

[0082] Figure 2 This is a schematic diagram of the vehicle network federation semantic communication scenario of the present invention;

[0083] Figure 3 Schematic diagram of the personalized semantic communication model of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0085] Figure 1 FIG. 1 is a flowchart of a semantic communication method based on personalized federated learning disclosed in this embodiment. Figure 1 As shown, the semantic communication method based on personalized federated learning includes the following steps:

[0086] Step 1: Build a federated semantic communication scenario for multiple vehicle users;

[0087] Build as Figure 2 The federated semantic communication scenario of the Internet of Vehicles shown in the figure is a federated semantic communication scenario for multiple vehicle users. During the training phase, all intelligent vehicles are within the coverage of the edge server, and the communication between vehicles is realized by a stable semantic communication system. Each vehicle is locally equipped with a semantic encoder and a semantic decoder. During the communication process, the sender uses the semantic encoder to extract the key features of the input data and transmits it to the receiver through the physical channel. The receiver uses the semantic decoder to reconstruct the features and perform downstream tasks. In each communication cycle, the edge server accepts the communication request of the vehicle user and obtains the set of vehicle users. ,in It is The index of vehicle users, , is the number of vehicle users. The edge server generates personalized semantic encoders and decoders for all vehicles by using the hypernetwork and updates the global parameters of the local semantic communication model using the average aggregation network.

[0088] Step 2: Construct a semantic communication model based on semantic-aware vector quantization variational autoencoder;

[0089] Semantic-aware vector quantized variational autoencoder, such as Figure 3 As shown, it mainly consists of a semantic encoder, a discrete codebook and a semantic decoder. The processing process is as follows:

[0090] First, the semantic encoder consists of two parts: a compressor based on semantic importance and a semantic encoder. The input image (the image is taken from the local training dataset of each vehicle user) is divided into image blocks, and perform embedding coding on them respectively to obtain the feature vectors of all image blocks , that is, the input feature vector ,in is the number of eigenvectors, which is equal to the number of image patches, is the feature vector dimension, the cosine similarity matrix of all feature pairs It can be calculated by the following formula:

[0091] (1),

[0092] in, for The transposed matrix of .

[0093] Get the similarity matrix After that, the importance weight of each feature vector can be obtained as follows:

[0094] (2),

[0095] in, It is The importance weight of the feature vector, It is The eigenvector and The cosine similarity between the feature vectors.

[0096] Then, use the Top-K strategy to retain the top The highest weighted eigenvector index ,in Indicates the compression ratio. The feature vector index to be retained The mask position corresponding to the corresponding eigenvector is set to 1, and the rest of the positions are set to 0. Then the mask matrix Middle The mask matrix corresponding to the eigenvectors Expressed as:

[0097] (3),

[0098] in, For the The mask matrix corresponding to the eigenvectors.

[0099] The above input feature vector And the resulting mask matrix Multiply, and you get the masked feature vector :

[0100] (4),

[0101] In the input feature vector After compression masking, the masked feature vector is obtained Perform semantic encoding. Use Vision Transformer (ViT) to implement semantic encoding, and the output of semantic encoding can be expressed as:

[0102] (5),

[0103] in, is the compressed feature vector, Represents semantic encoding, are the parameters of the semantic encoder.

[0104] The input image is processed by the above semantic encoder to obtain the compressed feature vector ,in, For the The compressed feature vectors, ,Then Mapped into discrete indices through a discrete codebook. The discrete codebook can be expressed as:

[0105] (6),

[0106] in, is a discrete codebook, is the first feature vectors, is the size of the codebook.

[0107] Afterwards, each feature vector is searched by the nearest neighbor method. and the eigenvectors of the discrete codebook Correspondingly, it is shown as follows:

[0108] (7),

[0109] in, yes The index of the eigenvector in the corresponding discrete codebook, is the first eigenvectors. Then the compressed eigenvector The discrete feature index obtained after the above mapping operation can be expressed as:

[0110] (8),

[0111] in, is the compressed feature vector The corresponding feature index set.

[0112] The semantic decoder is mainly composed of an attention layer and a feedforward network, whose input is the feature index set output by the discrete codebook The feature index set affected by noise obtained after passing through the channel :

[0113] (9),

[0114] in, is the time-varying channel coefficient, is the phase rotation caused by Doppler shift, is independent and identically distributed (iid) complex Gaussian noise, is the noise variance, is the unit matrix. Doppler shift It can be calculated using the following formula:

[0115] (10),

[0116] in, is the relative speed of the vehicle, is the speed of light, is the carrier frequency, is the angle of arrival of the signal.

[0117] Then, the feature index set affected by the noise By indexing the discrete codebook, we can recover The corresponding eigenvector , which is the input feature vector of the semantic decoder. The feature vector that the decoder needs to recover is recorded as , and It is mapped to a high-dimensional feature space through an embedding layer and then fed into the attention layer.

[0118] Different from the self-attention operation, in order to make the decoder focus on recovering the masked feature vector, the attention layer adopts the cross attention mechanism designed by this invention. The specific process can be expressed as follows:

[0119] (11),

[0120] in, represents the multi-head attention mechanism, 、 、 and They are the first The query, key, value and output mapping matrix parameters of each header, is the number of heads, is the embedding dimension of the head, i.e. the feature vector dimension, Represents a join operation.

[0121] Eigenvector and After the cross attention mechanism is processed, it is sent to the forward feedback layer. Similarly, in order to make the decoder focus on recovering the masked feature vector, the forward feedback layer only To process. After recovery, the forward feedback layer passes through the mask matrix Will and Connect them.

[0122] The output of the forward feedback layer is normalized and sent to a prediction head, mapped to the original feature space, thus completing all decoding operations. The entire decoding process can be expressed as follows:

[0123] (12),

[0124] in, is the output of the semantic decoder, is the input feature vector of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

[0125] Step 3: Deploy the semantic communication model constructed in step 2 locally on all vehicle users and initialize the model parameters.

[0126] Step 4: Build an aggregation network; as shown in the attached Figure 3 As shown, the aggregate network consists of a super network and an average aggregate network.

[0127] The role of the hypernetwork is to generate personalized parameters for the local semantic encoder and semantic decoder of each vehicle user. Specifically, the hypernetwork consists of an embedding layer, a multi-layer perceptron, and a linear layer. The process of generating personalized parameters for the local semantic communication model is as follows:

[0128] The input to the hypernetwork is the index of each vehicle user. The index is mapped into a low-dimensional vector through an embedding layer. This low-dimensional vector is then converted into a high-dimensional feature representation through a multi-layer perceptron. Finally, a linear layer converts the feature representation into the query (Q) parameter, key (K) parameter, and value (V) parameter of the self-attention layer. The above process can be expressed as follows:

[0129] (13),

[0130] in, represents the super network, are the parameters of the hypernetwork, It is The parameters of the self-attention layer for each vehicle user, 、 and They are the weight matrices corresponding to the query parameter (Q), key (K) parameter, and value (V) parameter mentioned above. It is The index of the vehicle user.

[0131] The average aggregation network is used to update the parameters of the local semantic communication model of each vehicle user, except for the self-attention layer, using a weighted aggregation method. The principle is as follows:

[0132] (14),

[0133] in, It is Non-personalized parameters of the local semantic communication model for vehicle users, is the updated global parameter, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users.

[0134] Step 5: The entire personalized semantic communication model is as follows Figure 3 As shown. A multi-objective joint optimization problem is proposed based on the federated semantic communication system, which mainly includes the parameter optimization of the local semantic communication model and the parameter optimization of the hypernetwork on the edge server. In addition to the parameter optimization of the semantic encoder and decoder, the local semantic communication model also includes the optimization of the discrete codebook. Specifically, the optimization goal of the semantic encoder and decoder is to minimize the reconstruction loss function to restore the original image features and guide the completion of downstream semantic tasks. The optimization goal of the discrete codebook is to minimize the loss function of the discrete codebook so that the basis vectors of the discrete codebook are as close as possible to the output features of the encoder. The optimization goal of the hypernetwork is to minimize the global weighted loss by generating personalized self-attention parameters that adapt to the data distribution of each client, thereby achieving efficient personalized federated learning.

[0135] The reconstruction loss uses the mean square error loss (MSE), which can be expressed as:

[0136] (15),

[0137] in, and Represent the output of the semantic decoder and the original image features respectively; represents the number of image blocks, To rebuild the losses.

[0138] The discrete codebook quantization loss can be expressed as:

[0139] (16),

[0140] in, is the basis vector of the normalized discrete codebook, for The transpose of is the compressed feature vector, is the input feature vector of the semantic decoder, represents the gradient cutoff, Quantize the loss for discrete codebooks.

[0141] Then the loss of the local semantic communication model is the sum of discrete codebook quantization loss and reconstruction loss, that is:

[0142] (17),

[0143] in, is a hyperparameter used to balance the weight relationship between reconstruction loss and discrete codebook loss.

[0144] The loss of the hypernetwork is the minimum mean square error loss between the output of the hypernetwork and the local training target, which is defined as follows:

[0145] (18),

[0146] in, is the loss of the hypernetwork, It is a super network for vehicle users The generated parameters, Is a vehicle user After local training, the parameters Is a vehicle user The amount of data, is the total amount of data for all vehicle users. The loss function of the hypernetwork is mainly used to train the hypernetwork and generate personalized weight parameters for the self-attention layer of the local semantic communication model of each vehicle user.

[0147] Finally, the global loss function of personalized federated learning can be defined as the weighted average of the losses of all local semantic communication models and the sum of the losses of the hypernetwork, that is:

[0148] (19),

[0149] in, For the The reconstruction loss of the semantic encoder and decoder of the local semantic communication model for each vehicle user, For the Discrete codebook quantization loss for the local semantic communication model among vehicle users.

[0150] The overall optimization goal of personalized federated learning is to minimize the global loss function, which can be expressed as:

[0151] (20),

[0152] in, represents the parameters of the local semantic communication model, including the semantic encoder, decoder and discrete codebook, Represents the parameters of the hypernetwork.

[0153] The personalized semantic communication model structure is constructed as follows Figure 3 shown.

[0154] Step 6: In order to solve the above optimization problem, an end-to-end joint training method is proposed. The specific training process is as follows:

[0155] Step 6-1, the super network generates initialized local semantic communication model parameters for each vehicle user according to the vehicle user's index;

[0156] Step 6-2: Each vehicle user performs local training using local data (i.e., local training dataset) based on initializing local model parameters and updates the local semantic communication model parameters. The local training steps are as follows:

[0157] Step 6-2-1, for each training round , the vehicle user randomly extracts B batch feature vectors from the input data Conduct training. is the total number of training times for local training.

[0158] Step 6-2-2, input feature vector Perform compression mask processing based on semantic perception to obtain the masked feature vector .

[0159] Step 6-2-3, Send it to the semantic encoder to extract semantic information:

[0160] (twenty one),

[0161] Step 6-2-4, Map with the discrete codebook to obtain the feature index set .

[0162] Step 6-2-5, Through the physical channel including the Doppler effect, the input of the receiver can be expressed as:

[0163] (twenty two),

[0164] in, is the time-varying channel coefficient, is the phase rotation caused by Doppler shift, is independent and identically distributed (iid) complex Gaussian noise, is the noise variance, is the identity matrix.

[0165] Step 6-2-6, Comparing with the discrete codebook, we get The corresponding eigenvector .

[0166] Step 6-2-7, the feature vector Send to the semantic decoder for decoding:

[0167] (twenty three),

[0168] Step 6-2-8. Calculate the loss function of the local semantic communication model as follows:

[0169] (twenty four),

[0170] Step 6-2-9: Use the adaptive moment estimation (ADMA) method to update the parameters of the local model. The specific process is as follows:

[0171] In one embodiment of the present invention, the parameters of the local semantic communication model are set to , It is A sample of the local dataset of vehicle users, the first-order moment of the loss function of the local semantic communication model with respect to the model parameters and codebook is recorded as , the second-order moment is recorded as It can be expressed as follows:

[0172] (25),

[0173] (26),

[0174] in, and denote the decay rates of the first-order moment and the second-order moment, respectively. and They are The first and second moments of the round, Indicates the Round local semantic communication model training, is the total number of rounds of local semantic communication model training, For Perform gradient calculations, For the Vehicle users in The parameters of the local semantic communication model during round training, It is The training loss of the local semantic communication model for each vehicle user.

[0175] Through equations (25) and (26), we can get the parameters of the local semantic communication model: Update the rules as follows:

[0176] (27),

[0177] in, It is The local semantic communication model parameters of vehicle users are The parameters after round training are updated, For the The local semantic communication model parameters of vehicle users are The parameters after round training are updated, , is the correction value, is a constant used to avoid division by 0. is the learning rate of the local semantic communication model.

[0178] Step 6-2-10: Repeat steps 6-2-1 to 6-2-9 until all training rounds are completed to obtain the approximate optimal solution of the local semantic communication model.

[0179] Step 6-3: Calculate the parameter changes of the self-attention layer in the local semantic communication model:

[0180] (28),

[0181] in, It is The parameters of the self-attention layer in the local semantic communication model of vehicle users are The amount of change in training rounds, It is The parameters for initializing the local semantic communication model for each vehicle user, It is The local semantic communication model of vehicle users is Parameters after epochs of local training.

[0182] Step 6-4: All local semantic communication models and parameters except the self-attention layer Upload to the edge server for aggregation.

[0183] Step 6-5. Calculate the loss of the hypernetwork with respect to the parameters of the hypernetwork Gradient:

[0184] (29),

[0185] in, is the loss of the hypernetwork with respect to the parameters of the hypernetwork The gradient, It is The parameters of the local semantic communication model of vehicle users are related to the parameters of the hypernetwork The gradient, Is a vehicle user The amount of data, and is the total amount of data of all vehicle users, is the set of all vehicle users.

[0186] Step 6-6: After obtaining this gradient, the adaptive moment estimation (ADMA) method is also used to update the parameters of the hypernetwork, as follows:

[0187] First, the first-order moment and second-order moment of the gradient of the hypernetwork loss with respect to the hypernetwork parameters can be expressed as:

[0188] (30),

[0189] in, It is the super network The first-order moment obtained after rounds of training, It is The first-order moment obtained after rounds of training, is the decay factor of the first-order moment.

[0190] (31),

[0191] in, It is the super network The second-order moment obtained after rounds of training, It is The second-order moment obtained after rounds of training, is the attenuation factor of the second-order moment.

[0192] Then, the parameters of the hypernetwork can be obtained The update rule is:

[0193] (32),

[0194] in, It is The parameters of the hypernetwork after rounds of training, It is The parameters of the hypernetwork after rounds of training, , is the correction value, is the learning rate of the hypernetwork.

[0195] Steps 6-7: Use the updated hypernetwork to generate the self-attention layer parameters for each vehicle user , and then use the weighted average function to generate the updated other parameters:

[0196] (33),

[0197] in, It is Non-personalized parameters of the local semantic communication model for vehicle users, is the updated global parameter, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users.

[0198] Step 6-8: Set the self-attention layer parameters and other parameters Distribute to all vehicle users for the next large round of training.

[0199] Step 6-9: Repeat steps 6-2 to 6-8 until the set number of training times is reached, a global approximate optimal solution is obtained, and a personalized semantic communication model is generated.

[0200] The personalized federated learning-based semantic communication method proposed in an embodiment of the present invention proposes a task-oriented semantic communication framework based on personalized federated learning. By designing a joint training model for aggregated and local networks, the framework aims to minimize the loss of the local semantic communication model and the loss of the hypernetwork. While efficiently completing communication tasks, it also takes into account factors that should be considered in practical application scenarios. Compared with existing communication architectures, the proposed personalized federated learning-based semantic communication method has excellent adaptability to different channel conditions and data heterogeneity, and is better suited to actual vehicle-to-vehicle communication scenarios.

[0201] The embodiment of the present invention further provides a system corresponding to the method, including:

[0202] Scenario construction unit, used to build federated semantic communication scenarios for multiple vehicle users;

[0203] A semantic communication model construction unit is used to build a semantic communication model based on a semantic-aware vector quantization variational autoencoder. The model includes a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to a discrete feature index. The semantic decoder restores the discrete feature index to a feature vector and performs downstream semantic tasks.

[0204] Model deployment unit, used to deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters;

[0205] An aggregation network construction unit is used to construct an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication model of each vehicle user except the self-attention layer;

[0206] An optimization target construction unit is used to construct a global loss function and minimize the global loss function as the optimization target. The global loss function is the weighted average of all local semantic communication model losses and the sum of the hypernetwork losses.

[0207] The joint training unit is used to train a personalized semantic communication model using a local semantic communication model and an aggregate network joint training method, thereby realizing semantic communication of personalized federated learning.

[0208] An embodiment of the present invention also discloses an electronic device, including a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor. When the computer program / instruction is executed by the processor, the steps of the semantic communication method based on personalized federated learning are implemented.

[0209] An embodiment of the present invention further discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps of the semantic communication method based on personalized federated learning.

[0210] An embodiment of the present invention further discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the semantic communication method based on personalized federated learning.

[0211] The method and system described in this paper, belonging to the field of wireless communications technology, propose a task-oriented semantic communication framework based on personalized federated learning. By designing a trainable hypernetwork and an average aggregation function, the parameters of the local semantic communication model are personalized. A multi-objective joint training approach is also proposed to implement the vehicle-to-vehicle communication process. By minimizing the aggregate loss function, a near-optimal solution for the aggregate network is obtained. Compared with existing federated learning methods, the proposed scheme has superior adaptability to different channel conditions and can address heterogeneous user configurations.

[0212] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0213] Anything not described in detail in the present invention is well known to those skilled in the art.

Claims

1. A semantic communication method based on personalized federated learning, characterized by: The following steps are involved: Build a federated semantic communication scenario for multiple vehicle users; A semantic communication model based on a semantic-aware vector quantized variational autoencoder is constructed, which includes a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to discrete feature indexes. The semantic decoder recovers the discrete feature indexes into feature vectors and performs downstream semantic tasks. Deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters; Construct an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication model of each vehicle user, except for the self-attention layer; Construct a global loss function and minimize the global loss function as the optimization goal. The global loss function is the weighted average of all local semantic communication model losses and the sum of the hypernetwork losses. The local semantic communication model and the aggregated network joint training method are used to train the optimal personalized semantic communication model, realizing semantic communication based on personalized federated learning. include: (1) The super network generates initialized local semantic communication model parameters for each vehicle user based on the vehicle user’s index; (2) Each vehicle user uses local data to perform local training based on the initialized local model parameters and updates the local semantic communication model parameters; (3) According to the parameters initialized by the local semantic communication model of the vehicle user, and the local semantic communication model of the vehicle user in Calculate the parameter changes of the self-attention layer in the local semantic communication model after the parameters of the local training round; (4) The parameter changes of all local semantic communication models and the parameters except the self-attention layer Upload to edge server for aggregation; (5) Parameters of the local semantic communication model based on vehicle users with respect to the parameters of the hypernetwork The gradient of , as well as the parameter change of the self-attention layer in the local semantic communication model, calculate the loss of the hypernetwork with respect to the parameters of the hypernetwork Gradient: (6) Update the parameters of the hypernetwork using the adaptive moment estimation method; (7) Use the updated hypernetwork to generate the self-attention layer parameters for each vehicle user, and then use the weighted average function to generate the updated other parameters: (8) Distribute the self-attention layer parameters and other parameters to all vehicle users for the next large round of training; (9) Repeat steps (2) to (8) until the set number of training times is reached, and a global approximate optimal solution is obtained to generate a personalized semantic communication model.

2. The semantic communication method based on personalized federated learning according to claim 1, characterized in that: The semantic encoding method is: According to the input feature vector The cosine similarity matrix of the feature pairs in , calculates the importance weight of each feature vector: ， in, It is The importance weight of the feature vector, It is The eigenvector and The cosine similarity between feature vectors, is the number of eigenvectors, corresponding to the number of image patches; Before Retention The highest weighted eigenvector index ,in Indicates the compression ratio, the feature vector index to be retained The mask position corresponding to the corresponding eigenvector is set to 1, and the remaining positions are 0, and the mask matrix is ​​obtained. ; Mask matrix Middle The mask matrix corresponding to the eigenvectors Expressed as: ， Will and Multiply to get the masked feature vector : ; And Perform semantic encoding operation to obtain compressed feature vector : ， in, is the compressed feature vector, Represents semantic encoding, are the parameters of the semantic encoder.

3. The semantic communication method based on personalized federated learning according to claim 1, characterized in that: The semantic decoding process is: The feature vector that the decoder needs to recover is recorded as , the input feature vector of the semantic decoder and It is mapped to a high-dimensional feature space through the embedding layer and then input into the attention layer. The attention layer uses a cross-attention mechanism to restore the masked feature vector. The formula is expressed as: ， in, represents the multi-head attention mechanism, 、 、 and They are the first The query, key, value and output mapping matrix parameters of each header, is the number of heads, is the embedding dimension of the head, i.e. the feature vector dimension, Indicates a connection operation; and After being processed by the cross attention mechanism, it is sent to the forward feedback layer; the forward feedback layer To process, After recovery, the forward feedback layer passes through the mask matrix Will and connect; The output of the forward feedback layer is normalized and sent to a prediction head, mapped to the original feature space, and all decoding operations are completed. The entire decoding process is expressed as: ， in, is the output of the semantic decoder, represents the semantic decoder, are the parameters of the semantic decoder.

4. The semantic communication method based on personalized federated learning according to claim 1, characterized in that: The hypernetwork consists of an embedding layer, a multilayer perceptron, and a linear layer. The process of generating personalized parameters for the local semantic communication model is as follows: The input of the hypernetwork is the index of each vehicle user. The index is mapped into a low-dimensional vector through the embedding layer. The low-dimensional vector is then converted into a high-dimensional feature representation through the multi-layer perceptron. Finally, the feature representation is converted into the query parameter, key parameter, and value parameter of the self-attention layer through the linear layer, which is expressed as: ， in, represents the super network, are the parameters of the hypernetwork, It is The parameters of the self-attention layer for each vehicle user, 、 and are the weight matrices corresponding to query parameters, key parameters, and value parameters, respectively. It is The index of the vehicle user.

5. The semantic communication method based on personalized federated learning according to claim 1, characterized in that: The average aggregation network is used to update the parameters of the local semantic communication model of each vehicle user except the self-attention layer, using a weighted aggregation method, which is expressed as: ， in, It is Non-personalized parameters of the local semantic communication model for vehicle users, is the updated global parameter, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users.

6. The semantic communication method based on personalized federated learning according to claim 1, characterized in that Construct a global loss function and minimize the global loss function as the optimization goal, including: ， in, represents the parameters of the local semantic communication model, represents the parameters of the hypernetwork, Is a vehicle user The amount of data, is the total amount of data of all vehicle users, is the number of vehicle users, For the The reconstruction loss of the semantic encoder and decoder of the local semantic communication model for each vehicle user, is a hyperparameter used to balance the weight relationship between reconstruction loss and discrete codebook loss, For the The discrete codebook quantization loss of the local semantic communication model for vehicle users, is the loss of the super network; Reconstruction losses Expressed as: , in, and Represent the output of the semantic decoder and the original image features, respectively. Represents the number of image blocks; Discrete codebook quantization loss Expressed as: ， in, is the basis vector of the normalized discrete codebook, for The transpose of is the compressed feature vector, is the input feature vector of the semantic decoder, represents the gradient cutoff; Hypernetwork Loss Expressed as: ， in, It is a super network for vehicle users The generated parameters, Is a vehicle user Parameters after local training.

7. The semantic communication method based on personalized federated learning according to claim 1, characterized in that: Step (2) includes the following steps: (21) For each training round , the vehicle user randomly extracts B batch feature vectors from the input data Conduct training; among them, is the total number of training times for local training; (22) Input feature vector Perform compression mask processing based on semantic perception to obtain the masked feature vector ; (23) Send it to the semantic encoder to extract semantic information and obtain the compressed feature vector ; (24) Map with the discrete codebook to obtain the feature index set ; (25) Through the physical channel including the Doppler effect, the feature index set affected by noise is obtained ; (26) Comparing with the discrete codebook, we get The corresponding eigenvector ; (27) The eigenvector Send it to the semantic decoder for decoding and get the output of the semantic decoder ; (28) Calculate the loss function of the local semantic communication model based on the discrete codebook quantization loss and reconstruction loss; (29) Adaptive moment estimation method is used to update the parameters of the local semantic communication model. The model parameters The update rules are: ， in, It is The local semantic communication model parameters of vehicle users are The parameters after round training are updated, For the The local semantic communication model parameters of vehicle users are The parameters after round training are updated, 、 is the correction value, is a constant, is the learning rate of the local semantic communication model; (210) Repeat steps (21) to (29) until all training rounds are completed and an approximately optimal solution for the local semantic communication model is obtained.

8. A semantic communication system based on personalized federated learning, characterized by: include: Scenario construction unit, used to build federated semantic communication scenarios for multiple vehicle users; A semantic communication model construction unit is used to build a semantic communication model based on a semantic-aware vector quantization variational autoencoder. The model includes a semantic encoder, a discrete codebook, and a semantic decoder. The semantic encoder compresses and encodes the input image to obtain a feature vector. The discrete codebook maps the feature vector to a discrete feature index. The semantic decoder restores the discrete feature index to a feature vector and performs downstream semantic tasks. Model deployment unit, used to deploy the constructed semantic communication model locally on all vehicle users and initialize the model parameters; An aggregation network construction unit is used to construct an aggregation network, including a super network and an average aggregation network. The super network is used to generate personalized parameters of the self-attention layer of the local semantic communication model for each vehicle user; the average aggregation network is used to update the global shared parameters of the local semantic communication model of each vehicle user except the self-attention layer; An optimization target construction unit is used to construct a global loss function and minimize the global loss function as the optimization target. The global loss function is the weighted average of all local semantic communication model losses and the sum of the hypernetwork losses. A joint training unit is used to train a personalized semantic communication model using a local semantic communication model and an aggregated network joint training method, thereby realizing semantic communication based on personalized federated learning. include: (1) The super network generates initialized local semantic communication model parameters for each vehicle user based on the vehicle user’s index; (2) Each vehicle user uses local data to perform local training based on the initialized local model parameters and updates the local semantic communication model parameters; (3) According to the parameters initialized by the local semantic communication model of the vehicle user, and the local semantic communication model of the vehicle user in Calculate the parameter changes of the self-attention layer in the local semantic communication model after the parameters of the local training round; (4) The parameter changes of all local semantic communication models and the parameters except the self-attention layer Upload to edge server for aggregation; (5) Parameters of the local semantic communication model based on vehicle users with respect to the parameters of the hypernetwork The gradient of , as well as the parameter change of the self-attention layer in the local semantic communication model, calculate the loss of the hypernetwork with respect to the parameters of the hypernetwork Gradient: (6) Update the parameters of the hypernetwork using the adaptive moment estimation method; (7) Use the updated hypernetwork to generate the self-attention layer parameters for each vehicle user, and then use the weighted average function to generate the updated other parameters: (8) Distribute the self-attention layer parameters and other parameters to all vehicle users for the next large round of training; (9) Repeat steps (2) to (8) until the set number of training times is reached, and a global approximate optimal solution is obtained to generate a personalized semantic communication model.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program / instruction stored in the memory and executable on the processor, wherein when the computer program / instruction is executed by the processor, the steps of the semantic communication method based on personalized federated learning are implemented according to any one of claims 1 to 7.

Images