A semantic-based federated learning method for mobile recommendation

Abstract

The present application relates to a kind of mobile recommendation-oriented semantic-based federated learning method, it is intended to improve the performance of model under the premise of protecting user privacy, while reducing heavy computing and communication load on server.The method proposes a kind of semantic-based optimization strategy, use the semantic information contained in user trajectory data to guide the model training under federated learning, to improve the recommendation performance of model.At the same time, semantic knowledge does not contain user privacy information, and the learning process of semantics is irreversible, very good protection of user privacy.In addition, the method uses the three-layer federated architecture of "client-edge server", to alleviate the computing and communication pressure of server.The introduction of edge also provides more available computing and communication resources for semantic optimization.Experimental results show that, compared with traditional federated learning framework, SFL has achieved certain improvement in performance and efficiency.

Description

Technical Field

[0001] This invention relates to the field of mobile recommendation technology, and in particular to a semantic-based federated learning method for mobile recommendation. Background Technology

[0002] With the widespread use of mobile devices, mobile recommender systems have become an important research area. Traditional mobile recommender systems centrally store users' historical check-in data on a central server to train an efficient recommender model. However, users' historical check-in data contains their private information, such as location, social relationships, and occupation. This leads to a significant risk of privacy breaches. In recent years, federated learning has become an effective method to address user privacy and security issues. McMahan et al. proposed FedAVG, which allows multiple clients to collaboratively train a machine learning model without sharing local data. Duan et al. proposed a self-balancing federated learning framework, Astraea, which alleviates the local and global imbalance of training data through data expansion and multi-client rescheduling.

[0003] With the development of federated learning, user data privacy and security issues in the field of mobile recommendation have also attracted attention. Guo et al. proposed PREFR, an edge-accelerated federated learning framework for Points of Interest (POI) recommendation, which protects user privacy and improves model efficiency by decoupling the recommendation process. Huang et al. proposed a federated learning method for geographic POI recommendation, which quantifies location information in the Internet of Things (IoT) by constructing a geographic information matrix and introduces federated learning into the matrix factorization of POI recommendation in the IoT environment to protect user privacy.

[0004] However, directly applying general federated learning methods to mobile recommendation systems can lead to performance degradation and computational and communication overload. On one hand, in existing federated learning setups, client data is stored locally to protect user privacy. This results in isolated local data across clients, and many user features from local client data cannot be directly uploaded to the server for global model training. Some contextual information has been used to train an efficient mobile recommendation model; however, the rich semantic knowledge in user trajectory data remains underutilized. On the other hand, in federated learning, the server frequently experiences computational overload and communication bottlenecks due to the need to coordinate the computational and communication tasks of all clients. In mobile recommendation scenarios, clients are mostly mobile terminals such as smartphones, tablets, or in-vehicle systems. The number of clients is not only enormous, but they also frequently move dynamically across domains. This further exacerbates the computational and communication pressure on the server. Summary of the Invention

[0005] In view of the above-mentioned problems in the existing technology, the technical problem to be solved by the present invention is: how to alleviate the problems of model performance degradation and computational and communication overload in federated learning in mobile recommendation scenarios.

[0006] To address the aforementioned technical problems, this invention employs the following technical solution: a semantic-based federated learning method for mobile recommendation, comprising the following steps:

[0007] S1: The client uses local data to train a local model and obtains the parameters of the trained local model. At the same time, the client calculates the local trajectory semantic vector based on the local data. After local training is completed, the client first encrypts the local trajectory semantic vector and uploads it to the server.

[0008] S2: The server uses the decrypted client semantic vector and clusters the client based on similarity clustering to obtain the clustering results. Based on the clustering results, the client encrypts and uploads the trained local model parameters to the designated edge terminal.

[0009] S3: After receiving and decrypting the trained local model parameters, the edge terminal aggregates the trained local model parameters and loads the aggregated model parameters into the edge terminal model. Then, the edge terminal uses the public dataset to train the edge terminal model and obtains the trained edge terminal model parameters. Each edge terminal encrypts and uploads the trained model parameters.

[0010] S4: The server decrypts the model parameters trained on the edge and performs global aggregation to obtain global model parameters. The server then broadcasts the global model parameters to all clients and edge devices.

[0011] S5: Determine if the preset number of iterations has been reached. If not, return to S1; otherwise, end.

[0012] Preferably, the step of training the local model to obtain the trained local model parameters in S1 includes:

[0013] The client receives global model parameters from the server and initializes the client's `c` object. i The local model parameters are used by the client to train the model locally using a local dataset.

[0014]

[0015] in, Indicates client c i The parameters of the local model after training, It is client c i The local model parameters, where η is the learning rate, n h x represents the number of trajectory points. iand y i These are the trajectory data and tags of the i-th user, respectively. express gradient, It is cross-entropy loss. This indicates that when the model parameters are The output function of the local model at that time, where λ is the regularization factor.

[0016] Preferably, the step of calculating the local trajectory semantic vector in S1 includes:

[0017] The prompt project was used to reconstruct the user's trajectory data into text prompts: the day was divided into five time periods, namely midnight, early morning, noon, early morning, and night.

[0018] A text template was created and the input slots were filled, template = "User went to the[]on[][], the[]on[][], ..., and the[]on[][]". Then, the prompt text for the client CI was generated, i.e., the prompt sentence.

[0019]

[0020] in, f represents the input trajectory sequence. prompt (·) is a template sentence with a slot. The trajectory points will be filled into the corresponding slots.

[0021] We use a fine-tuned pre-trained language model to learn semantic feature representations of the prompt sentences:

[0022]

[0023] Among them, f PTM (·) is the pre-trained language model after fine-tuning. This represents the parameters of the pre-trained language model after fine-tuning. This represents the representation vector output by the PTM, i.e., the client c. i The trajectory semantic vector, where, b is the batch size, sl is the sequence length, d is the output feature dimension of the PTM, and R is the set of real numbers.

[0024] Preferably, the steps in S1 to obtain the fine-tuned pre-trained language model are as follows:

[0025] For user u in the public dataset itrajectory Reconstruct it into a natural language sentence using formula (1). E c Defined as a semantic location category, used to store semantic information in the trajectory. N c d represents the number of location categories, and d is the dimension. The location category of the last trajectory point in the trajectory is used as the label. c Then, calculate the location category distribution:

[0026]

[0027]

[0028] Among them, dis c f represents the distribution of location categories CE (·) represents the cross-entropy loss function.

[0029] For each trajectory, randomly select some labels from the location category set. c Location category, as a negative sample c For negative samples, c Calculate its location category distribution:

[0030]

[0031]

[0032] in, N represents the location category distribution of negative samples. neg f is the number of negative samples. BCE (·) denotes the binary cross-entropy loss function, O c Is with All-zero tensors of the same shape.

[0033] Finally, the parameters are learned by minimizing the objective function of the training data. The objective function is: Obtain the parameters of the fine-tuned pre-trained language model

[0034] Preferably, the encryption and decryption processes of S1-S4 are as follows:

[0035]

[0036] ct=g pt r N mod N 2 (9)

[0037]

[0038] Where pt represents plaintext, ct represents ciphertext, g represents randomly generated common parameters, and r is a randomly generated number. N is the product of two randomly selected prime numbers p and q, i.e., N = p × q. Modulo operation is used to represent modulo operations, L(·) represents a function without specific meaning, and x represents the input of the function.

[0039] Preferably, the step in S2 where the server clusters client-side clusters based on similarity is as follows:

[0040] The semantic similarity between clients is measured by the distance between trajectory semantic vectors:

[0041]

[0042] in, Indicates client c i and client c j trajectory semantic vector and The similarity between them, sqrt(x) is the operation of finding the square root of x. It is a calculation and The product of.

[0043] Then, the server uses the semantic similarity between clients to cluster the clients into K clusters:

[0044]

[0045]

[0046] Among them, CLU k This represents the k-th cluster. Indicates CLU k The mean of the client semantic vectors within the cluster, i.e., the cluster center, N k Indicates CLU k The number of clients within, express and The square of the distance between them.

[0047] Preferably, the step of aggregating the trained local model parameters in S3 includes:

[0048]

[0049] in, Indicates edge end e i The aggregated model parameters.

[0050] Preferably, the step of obtaining the global model parameters in S4 includes:

[0051]

[0052] Where, θ g N represents the global model parameters. e Indicates the number of edge ends. Indicates edge end e i The parameters of the trained model.

[0053] Compared with the prior art, the present invention has at least the following advantages:

[0054] 1. In SFL, a semantic-based optimization strategy was designed to improve the performance of the recommendation model. Semantics itself does not contain user privacy information, and the learning process of semantic knowledge is irreversible, effectively meeting privacy protection requirements. To alleviate the computational and communication burden on the server, a three-tier architecture of "client-edge-server" is used. Multiple edge devices are introduced to share some of the server's computational and communication tasks, providing available computational and communication resources for semantic optimization.

[0055] 2. In SFL, a semantic-based optimization strategy was designed. Rich semantic knowledge from user trajectories guides the server to cluster clients to uncover common features, thereby improving the aggregation effect of the edge model and training an efficient mobile recommendation model.

[0056] 3. Experimental results show that SFL's global model performance outperforms several existing strong baselines. Further analysis reveals SFL's advantages in server performance improvement rate, client performance, and computational and communication efficiency. Attached Figure Description

[0057] Figure 1 This is an overall flowchart of the method of the present invention.

[0058] Figure 2 A flowchart for learning trajectory semantic vectors for mobile clients.

[0059] Figure 3 Server performance curves for two sequence recommendation models, SASREC and SSEPT, obtained from experiments on two datasets, NYC and TKY. Figure 3 (a) and Figure 3 (b) Show the performance curves of SASREC and SSEPT on NYC, respectively. Figure 3 (c) and Figure 3 (d) represents the performance curves of SASREC and SSEPT on TKY, respectively.

[0060] Figure 4Performance graphs of two sequence recommendation models, SASREC and SSEPT, were obtained from experiments on two datasets, NYC and TKY. Figure 4 (a) and Figure 4 (b) Shows the performance graphs of SASREC and SSEPT on NYC, respectively. Figure 4 (c) and Figure 4 (d) shows the performance graphs of SASREC and SSEPT on TKY, respectively. Detailed Implementation

[0061] The present invention will now be described in further detail.

[0062] This invention presents a semantic-based federated learning method for mobile recommendation, abbreviated as SFL.

[0063] Define the trajectory dataset as D = (traj c traj u ), where traj c This represents the client's trajectory data. N represents the number of clients. traj u This represents the trajectory data of users in a public dataset. M represents the number of users in the public dataset. For example, Each check-in s within the trajectory i They are all tuples (u i p i cl i , t i ), which represents user u i At time t i Visited location p i Among them, cl i Indicates location p i Categories, such as cinemas, restaurants, etc.

[0064] The main task is to improve the performance of mobile recommendations while protecting privacy, and to alleviate the problem of server computation and communication overload under federated learning, thereby improving computational and communication efficiency. Therefore, in the trajectory dataset D, to achieve multiple objectives, a multi-objective optimization function F(D) is used to represent the multi-objective optimization task:

[0065] F(D)=F(f priv (D), f pefm (D), f effi (D)) (2)

[0066] Among them, f priv (D), f pefm (D), feffi (D) represents the three sub-objective optimization functions. priv (D) aims to protect user privacy when processing user data. pefm (D)←max(AUC(D)), f pefm (D) aims to improve the performance of recommendations. effi (D) aims to minimize the computational and communication overhead of the server during the training of the global model, thereby improving the efficiency of the federated method by minimizing the server's computation time and communication volume.

[0067] A semantic-based federated learning method for mobile recommendation includes the following steps:

[0068] S1: The client uses local data to train a local model and obtains the parameters of the trained local model. At the same time, the client calculates the local trajectory semantic vector based on the local data. After local training is completed, the client first encrypts the local trajectory semantic vector and uploads it to the server.

[0069] S2: The server uses the decrypted client semantic vector and clusters the client based on similarity clustering to obtain the clustering results. Based on the clustering results, the client encrypts and uploads the trained local model parameters to the designated edge terminal.

[0070] S3: After receiving and decrypting the trained local model parameters, the edge terminal aggregates the trained local model parameters and loads the aggregated model parameters into the edge terminal model. Then, the edge terminal uses the public dataset to train the edge terminal model and obtains the trained edge terminal model parameters. Each edge terminal encrypts and uploads the trained model parameters.

[0071] S4: The server decrypts the model parameters trained on the edge and performs global aggregation to obtain global model parameters. The server then broadcasts the global model parameters to all clients and edge devices.

[0072] The process of training at the edge is similar to the process of training a local model on the client using local data.

[0073] S5: Determine if the preset number of iterations has been reached. If not, return to S1; otherwise, end.

[0074] Specifically, the steps in S1 for training the local model to obtain the trained local model parameters include:

[0075] The client receives global model parameters from the server and initializes the client's `c` function. i The local model parameters are used by the client to train the model locally using a local dataset.

[0076]

[0077] in, Indicates client c i The parameters of the local model after training, It is client c i The local model parameters, where η is the learning rate, n h x represents the number of trajectory points. i and y i These are the trajectory data and tags of the i-th user, respectively. express gradient, It is cross-entropy loss. This indicates that when the model parameters are The output function of the local model at that time, where λ is a regularization factor used to prevent overfitting.

[0078] Specifically, the step of calculating the local trajectory semantic vector in S1 includes:

[0079] In mobile recommendation systems, user trajectory data contains rich semantic information, offering potential for improving the performance of recommendation models under federated learning. Therefore, in SFL, a semantic-based optimization strategy is designed to utilize the semantic information contained in user trajectory data to guide model training under federated learning. To mine and represent the contextual semantic information in user trajectory data, a "pre-train and prompt" paradigm is adopted. A prompt sentence is reconstructed using the prompt engineering method and used as input to the pre-trained language model to generate the user's trajectory semantic vector. Figure 2 This process can be described intuitively.

[0080] Pre-trained language models (PTMs) can learn the language structure and knowledge of large-scale text data to generate high-quality text representations. Therefore, the trajectory data is first reconstructed into text, also known as prompt sentences.

[0081] The prompt project is used to reconstruct the user's trajectory data into text prompts: the day is divided into five time periods, namely midnight (0:00-5:59), early morning (6:00-9:59), noon (10:00-12:59), afternoon (13:00-18:59), and night (19:00-23:59).

[0082] A text template was created and the input slots were filled, template = "User went to the[]on[][], the[]on[][], ..., and the[]on[][]". Then, the client c was generated. i The prompt text is the prompt sentence.

[0083]

[0084] in, f represents the input trajectory sequence. prompt (·) is a template sentence with a slot. The trajectory points will be filled into the corresponding slots.

[0085] We use a fine-tuned pre-trained language model to learn semantic feature representations of the prompt sentences:

[0086]

[0087] Among them, f PTM (·) is the pre-trained language model after fine-tuning. This represents the parameters of the pre-trained language model after fine-tuning. This represents the representation vector output by the PTM, i.e., the client c. i The trajectory semantic vector, where, b is the batch size, sl is the sequence length, d is the output feature dimension of the PTM, and R is the set of real numbers.

[0088] Specifically, the steps in S1 to obtain the fine-tuned pre-trained language model are as follows:

[0089] To make the pre-trained language model more adaptable to task and trajectory data, the pre-trained language model was fine-tuned using a public dataset before learning client trajectory semantic vectors.

[0090] For user u in the public dataset i trajectory Reconstruct it into a natural language sentence using formula (1). E c Defined as a semantic location category, used to store semantic information in the trajectory. N c d represents the number of location categories, and d is the dimension. The location category of the last trajectory point in the trajectory is used as the label. c Then, calculate the location category distribution:

[0091]

[0092]

[0093] Among them, dis c f represents the distribution of location categories CE (·) represents the cross-entropy loss function.

[0094] Negative sampling is used to improve the computational efficiency of the language model. Specifically, for each trajectory, some categories different from the label are randomly selected from the location category set. c Location category, as a negative sample c For negative samples, c Calculate its location category distribution:

[0095]

[0096]

[0097] in, N represents the location category distribution of negative samples. neg f is the number of negative samples. BCE (·) denotes the binary cross-entropy loss function, O c Is with All-zero tensors of the same shape.

[0098] Finally, the parameters are learned by minimizing the objective function of the training data. The objective function is: Obtain the parameters of the fine-tuned pre-trained language model

[0099] Specifically, to protect the security of parameter transmission between the client, edge, and server, homomorphic encryption technology is used to encrypt and decrypt the parameters during transmission. The encryption and decryption operations mainly involve model parameters, client trajectory semantic vectors, and clustering results. The encryption and decryption process for S1-S4 is as follows:

[0100]

[0101] ct=g pt r N mod N 2 (9)

[0102]

[0103] Where pt represents plaintext, ct represents ciphertext, g represents randomly generated common parameters, and r is a randomly generated number. N is the product of two randomly selected prime numbers p and q, i.e., N = p × q. Modulo operation is used to represent modulo operations, L(·) represents a function without specific meaning, and x represents the input of the function.

[0104] Specifically, the steps in S2 where the server clusters client connections based on similarity are as follows:

[0105] The purpose of clustering is to guide edge endpoints in selecting semantically similar clients to mine shared information. The semantic similarity between clients is measured using the distance between trajectory semantic vectors.

[0106]

[0107] in, Indicates client c i and client c j trajectory semantic vector and The similarity between them, sqrt(x) is the operation of finding the square root of x. It is a calculation and The product of.

[0108] Then, the server uses the semantic similarity between clients to cluster the clients into K clusters:

[0109]

[0110]

[0111] Among them, CLU k This represents the k-th cluster. Indicates CLU k The mean of the client semantic vectors within the cluster, i.e., the cluster center, N k Indicates CLU k The number of clients within the cluster is given by formula (14), which represents the clustering based on client semantic similarity. express and The square of the distance between them.

[0112] Specifically, the step of aggregating the trained local model parameters in S3 includes:

[0113]

[0114] in, Indicates edge end e i The aggregated model parameters.

[0115] Specifically, the step of obtaining the global model parameters in S4 includes:

[0116] After all edge devices have completed training, each edge device encrypts and uploads the trained model parameters to the server for global aggregation, resulting in the global model parameters θ. g :

[0117]

[0118] Where, θ g N represents the global model parameters. e Indicates the number of edge ends. Indicates edge end e i The parameters of the trained model.

[0119] experiment

[0120] 1. Dataset and Model

[0121] The proposed method was evaluated on two publicly available datasets, NYC and TKY. These are commonly used datasets in the mobile recommendation field, containing user check-in records from New York and Tokyo, respectively. The NYC dataset contains 1083 users, 5135 POIs, and 147938 check-in records. The TKY dataset contains 2293 users, 7873 POIs, and 447570 check-in records.

[0122] To evaluate the SFL and baseline models, two sequential recommendation models were used to validate SFL and the baseline: SASREC (Self-Attention based Sequential Recommendation) and SSEPT (Stochastic Shared Embeddings). SASREC predicts the next item by identifying related items from the target items based on the user's historical behavior. SSEPT enhances the model's personalization capabilities by using the Stochastic Shared Embeddings (SSE) technique. Because both sequential recommendation models have low computational costs and can simultaneously capture user contextual information, they are well-suited for federated recommendation tasks.

[0123] 2. Baseline

[0124] The existing models Centralized, FedAVG, SCAFFOLD, FedProx, FedAdam, FedDyn, and FedALA are referred to as the baselines.

[0125] 3. Evaluation Indicators

[0126] Two evaluation metrics, NDCG@k and HR@k, are used to assess SFL and baseline, where k values ​​include 5, 10, and 20. NDCG@k and HR@k are commonly used evaluation metrics in recommender systems. NDCG stands for Normalized Discounted Cumulative Gain, which assesses the discrepancy between the recommended results and the user's actual interaction list. HR stands for Hit Ratio, which assesses whether the recommended results contain the user's actual interactions.

[0127] Table 1 Global Model Results

[0128]

[0129] 4. Global Model Results

[0130] Experiments were conducted on the NYC and TKY datasets using two models, SASREC and SSEPT. The performance of centralized training, SFL, and all baselines was compared, and the results are shown in Table 1. In Table 1, underlined values ​​indicate the best values ​​among the baselines, bold values ​​represent the SFL results, and "improve" indicates the percentage improvement of SFL relative to the best value among the baselines. Table 1 shows that the model trained using centralized training performs best across all datasets and models. This is because the server can access all users' data during centralized training. SFL outperforms the baseline in all settings, with an average improvement of approximately 12%. When training the SSEPT model on the TKY dataset, the performance improvement was the largest at NDCG@5, at 24.46%.

[0131] 5. Server performance improvement rate

[0132] Rapidly improving and stabilizing the performance of the global model can save significant time during model training and improve the efficiency of federated learning. Therefore, the performance improvement rates of SFL and the baseline on the global model were compared. Figure 3 As shown, the results of NDCG@20 were selected for comparison, with the horizontal axis 0-800 representing 8 global iterations. The results indicate that the baseline's global model performance improvement rate is relatively slow. Furthermore, some baselines exhibit performance degradation in the later stages of global iterations, while SFL's global model performance shows an upward trend throughout the entire training process. SFL's global model performance improves rapidly in the first two global iterations and gradually stabilizes in subsequent iterations. Simultaneously, SFL outperforms the baseline in overall performance.

[0133] 6. Client performance

[0134] The performance of client-side models of SFL and the baseline was compared, and the results are as follows: Figure 4As shown in the figure, 35 client-side models were used for evaluation before the final round of global aggregation. The boxes in the figure represent the client-side model performance of different models, and the boxes and vertical lines represent different ranges: 25%–75% and 10%–90%. The median, mean, and outliers are also marked to visually demonstrate the performance differences between the SFL and the baseline client-side models. Figure 4 As can be seen, SFL exhibits the best client-side performance, with higher median, mean, and overall performance compared to the baseline. Furthermore, SFL has relatively fewer outliers compared to the baseline. Figure 4 (b) and (d) show that in the SSEPT models NYC and TKY, the SFL has a smaller box and more consistent client performance.

[0135] 7. Ablation test

[0136] To verify the effectiveness of each component in SFL, ablation experiments were conducted, and the results are shown in Table 2. SFL was compared with the centralized, classic federated learning method FedAVG and the best baseline model FedALA under equivalent settings. Different values ​​of 'semantic(traj_len)' represent different user trajectory lengths set in the semantic-based optimization strategy to learn different levels of semantic information. Table 3 shows that SFL's performance is significantly improved compared to the baseline model when using only the three-layer architecture. Introducing the semantic-based optimization strategy further improves SFL's performance. This indicates that both the optimization strategy and the three-layer architecture have a significant effect on improving model performance. Furthermore, it was found that both excessively short and excessively long trajectory data affect model performance, but overall, both result in some improvement.

[0137] Table 2 Ablation Experiment Results

[0138]

[0139] 8. Computational and communication efficiency

[0140] This section compares the computational and communication efficiency of SFL and the baseline, as shown in Table 3. Since the baseline model does not use edge computing, the edge computing time on the baseline is marked as 'None'. As can be seen from Table 3, the computation time on the SFL server (9.28s) is significantly shorter than the computation time on the baseline server (22.86s). This indicates that the introduction of edge computing in SFL greatly reduces the computational burden on the server. Furthermore, since the client undertakes the task of learning trajectory semantic vectors in SFL, the local computation time is increased. Therefore, the computation time on the SFL client (5.21s) is slightly higher than the baseline computation time (4.18s).

[0141] To facilitate the calculation and comparison of communication costs, the model parameter size was fixed at 11MB. Since the communication volume of semantic vectors and clustering results in SFL is very low (both less than 10KB), the communication costs of these two parts were ignored when calculating the communication volume. As can be seen from the results in Table 3, the introduction of edge devices in SFL distributes the communication tasks between the server and the client to several edge devices, greatly reducing the communication pressure on the server.

[0142] Table 3 Computational and Communication Efficiency

[0143]

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A semantic-based federated learning method for mobile recommendation, characterized in that: Includes the following steps: S1: The client uses local data to train a local model and obtains the parameters of the trained local model. At the same time, the client calculates the client trajectory semantic vector based on the local data. After local training is completed, the client first encrypts the client trajectory semantic vector and uploads it to the server. The steps in S1 for calculating the client trajectory semantic vector include: The prompt project is used to reconstruct user trajectory data into text prompts: the day is divided into five time periods, namely midnight, early morning, noon, afternoon and night; A text template was created and the input slots were filled; then, the client was generated. The prompt text is the prompt sentence. : （1） in, This represents the input trajectory sequence. It is a template sentence with slots. The trajectory points will be filled into the corresponding slots; We use a fine-tuned pre-trained language model to learn semantic feature representations of the prompt sentences: （3） in, It is a pre-trained language model that has been fine-tuned. This represents the parameters of the pre-trained language model after fine-tuning; This represents the representation vector output by the PTM, i.e., the client. The trajectory semantic vector, where, , It is the batch size. It is the sequence length. R is the output feature dimension of PTM, and R is the set of real numbers; S2: The server uses the decrypted client semantic vector and clusters the client based on similarity clustering to obtain the clustering results. Based on the clustering results, the client encrypts and uploads the trained local model parameters to the designated edge terminal. The steps in S2 where the server performs similarity-based clustering on the client are as follows: The semantic similarity between clients is measured by the distance between trajectory semantic vectors: (12) in, Indicates the client and client trajectory semantic vector and Similarity between them Is seeking The square root operation, It is a calculation and The product; Then, the server uses the semantic similarity between clients to cluster the clients into K clusters: (13) (14) in, This represents the k-th cluster. express The mean of the client semantic vectors within the cluster, i.e., the cluster center. express The number of clients within, express and The square of the distance between them; S3: After receiving and decrypting the trained local model parameters, the edge end aggregates the trained local model parameters and loads the aggregated model parameters into the edge end model. Then, the edge end uses the public dataset to train the edge end model and obtains the trained edge end model parameters. Each edge end encrypts and uploads the trained model parameters. S4: The server decrypts the model parameters trained on the edge and performs global aggregation to obtain global model parameters. The server then broadcasts the global model parameters to all clients and edge devices. S5: Determine if the preset number of iterations has been reached. If not, return to S1; otherwise, end.

2. The semantic-based federated learning method for mobile recommendation as described in claim 1, characterized in that: The steps in S1 for training the local model to obtain the trained local model parameters include: The client receives global model parameters from the server and initializes the client. The local model parameters are used by the client to train the model locally using a local dataset. (11) in, Indicates the client The parameters of the local model after training, It is a client Local model parameters, It's the learning rate. Indicates the number of trajectory points. and These are the trajectory data and tags of the i-th user, respectively. express gradient, It is cross-entropy loss; This indicates that when the model parameters are The output function of the local model at that time. It is a regularization factor.

3. The semantic-based federated learning method for mobile recommendation as described in claim 2, characterized in that: The steps for constructing the fine-tuned pre-trained language model are as follows: For users in the public dataset trajectory Reconstruct it into a natural language sentence using formula (1). ,Will Defined as a semantic location category, used to store semantic information in the trajectory. , Indicates the number of location categories. It is a dimension; the location category of the last trajectory point in the trajectory is used as the label, represented as... Then, calculate the location category distribution: (4) (5) in, Indicates the distribution of location categories. Represents the cross-entropy loss function; Randomly select some different locations from the location category set for each trajectory Location category, as negative sample For negative samples Calculate its location category distribution: (6) (7) in, This represents the location category distribution of negative samples. It is the number of negative samples. This represents the binary cross-entropy loss function. Is with All-zero tensors of the same shape; Finally, the parameters are learned by minimizing the objective function of the training data. The objective function is: The parameters of the fine-tuned pre-trained language model are obtained. .

4. The semantic-based federated learning method for mobile recommendation as described in claim 3, characterized in that: The encryption and decryption process of S1-S4 is as follows: (8) (9) (10) in, Indicates plain text, Indicates ciphertext, This represents randomly generated common parameters. The number is randomly generated; Two prime numbers are randomly selected. and The product of, i.e. ,use Represents modulo operations. Let x represent the function, and let x represent the input of the function.

5. A semantic-based federated learning method for mobile recommendation as described in claim 4, characterized in that: The step of aggregating the trained local model parameters in S3 includes: (15) in, Indicates edge end The aggregated model parameters.

6. The semantic-based federated learning method for mobile recommendation as described in claim 5, characterized in that: The steps in S4 to obtain the global model parameters include: (16) in, Represents global model parameters. Indicates the number of edge ends. Indicates edge end The parameters of the trained model.

Images