A large language model enhanced user privacy protection cross-domain service recommendation method

Abstract

The application discloses a large language model enhanced user privacy protection cross-domain service recommendation method, first, the data in the data set is screened, a text prompt is designed, a large language model is used to enhance service description text and user attribute information; the service description text is encoded by using a BERT language model to generate embedding; and a PQ method is used to map the text embedding into discrete codes, and a code dictionary is maintained; reading user historical service interaction data, generating user behavior sequence files in time stamp order; the service description information in the user behavior sequence is converted into discrete code representation, input sequence encoder for local model training, and the model with the optimal performance is saved; the parameter gradient of the local model is clipped and quantized, and then uploaded to the server; the model is fine-tuned by using soft prompt and hard prompt. The method reduces the semantic difference in cross-domain recommendation, and fine-tunes the model by using soft prompt and hard prompt, thereby enhancing the personalization and accuracy of recommendation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data mining and recommendation system, and particularly relates to a large language model enhanced user privacy protection cross-domain service recommendation method. BACKGROUND

[0002] Nowadays, with the rapid development of information technology and the advent of the big data era, data is concentrated in some popular fields. Cross-domain service recommendation (Service Recommendation) is committed to using user service interaction data collected from the source domain to mine the implicit user preferences and recommend appropriate services to the target domain users, so as to meet the personalized needs of users in multiple fields, and is an effective way to overcome information imbalance. However, traditional recommendation methods face many challenges in cross-domain scenarios, including insufficient user privacy protection, low data utilization efficiency, and lack of explainability of the recommendation process.

[0003] Recently, many researchers have proposed many cross-domain recommendation models based on knowledge transfer mapping, collaborative matrix decomposition and representation sharing, which transfer user preferences for services according to the commonality between domains. Most existing recommendation methods integrate data from multiple domains in a centralized manner, which not only poses a serious risk of privacy leakage, but also increases users' suspicion of the system, thereby affecting the effectiveness of the recommendation. In addition, the recommendation model is difficult to effectively model the user's preferences in multiple domains, especially in the case of data sparsity or lack of unified semantic representation, which further limits the performance of the recommendation system. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and proposes a large language model enhanced user privacy protection cross-domain service recommendation method to improve the recommendation performance of the model in the field. Two fine-tuning methods, soft prompt and hard prompt, are designed: soft prompt is based on the common preferences of domain users, and hard prompt is optimized using the user's historical interaction sequence.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows:

[0006] A large language model enhanced user privacy protection cross-domain service recommendation method, comprising the following steps:

[0007] Step 1, screen the data in the data set, and design prompt to enhance the service description text and user attribute information using GPT language model;

[0008] Step 2, encode the service description text using the BERT language model to generate an embedding;

[0009] Step 3, use the PQ method to map the text embedding into discrete codes and maintain a code dictionary;

[0010] Step 4, read and process the user's historical service interaction data, sort the historical service interaction records of each user according to the timestamp order, and generate the corresponding user behavior sequence file. Each line in the file represents a user's service interaction sequence. Then use the leave-N-out strategy to divide the training, validation and test data sets;

[0011] Step 5, represent each service description information in the sequence file by discrete code, then use the processed historical service interaction data for local training, and save the best model according to the index performance;

[0012] Step 6, only upload the gradient of the model parameters to the server locally, and the server updates the code dictionary after fusing the gradient and synchronizes it to the local;

[0013] Step 7, soft prompt, hard prompt fine-tuning

[0014] As a preferred, in step 1, the service data set needs to be filtered before enhancement, removing interaction data with small interaction and service metadata with missing data, and designing appropriate prompts for large language model enhancement and fusion:

[0015]

[0016]

[0017] Wherein the text prompt and are used for attribute optimization and content enhancement of user u and item i respectively, A u and A i are the user / item features enhanced by large language model (LLM), D u and D i represent the text attributes of the data set itself.

[0018] As a preferred, in step 2, when the transformer (BERT) represented by the bidirectional encoder is used to encode the service description text, a special input mark [CLS] is added in front of the initialization encoding T i = {w1, w2, …, w j , …, w c} of each service description text, that is, x i = BERT ([CLS; w1; …; w c ]), where [;] represents the splicing operation, w j is the content word in natural language, and c is the truncation length of the project text. The text encoding vector represents the representation of the given text.

[0019] As a preference, in step 3, the product quantization method (PQ) is:

[0020] 1) Sub-vectorization: Define D groups of vectors, where each vector corresponds to a dimension d W / D and contains M c centroids. The text encoding vector x i is first divided into D sub-vectors, i.e.,

[0021] x i = [x i,1 ,…, x i,D ]

[0022] 2) Discrete mapping: Let a k,j ∈ d W / D denote the j-th centroid embedding in the k-th group of vectors. For each k-th sub-vector x i,k , PQ selects the nearest centroid embedding from the corresponding centroid set, generating the discrete encoding of x i,k :

[0023]

[0024] where c i,k is the k-th dimensional discrete encoding vector of item i.

[0025] 3) Codebook maintenance: The text encoding vector x i is split into D sub-vectors, each representing the projection of the original vector in a different subspace. A codebook (a set of pre-defined centroids) is created for each subspace, and these codebooks collectively form the code embedding dictionary. Each sub-vector is quantized to the nearest centroid in the codebook, and the index of the centroid serves as the "code" for that sub-vector. The quantization results of all sub-vectors are combined into a code vector, which is used to represent the original high-dimensional vector.

[0026] The corresponding centroids are retrieved from the global code embedding dictionary through lookup operations. These centroids are then combined into the final item representation vector through average pooling methods. These embedding representations are stored in the code embedding dictionary for subsequent retrieval and matching.

[0027] Each domain maintains a code embedding dictionary where d v denotes the dimension of the item embedding. There are D code embedding matrices in T, each matrix is shared among all items in the server (not limited to a single domain). This feature allows us to align different domains and embed general domain information into the item embeddings.

[0028] For retrieval operations on T, the code embedding of item i can be represented as where is the embedding vector of the (k) th row in matrix T The code embedding is obtained by average pooling the final representation vector of item i:

[0029]

[0030] where v i is the final representation of the item, is the average pooling method over D dimensions, and the code embedding of item i can be represented as

[0031] As preferred, in step 4, the service IDs and corresponding categories interacted by the user user in each service interaction sequence are sorted in chronological order of interaction, separated by commas; the form is S u = {(ID of service 1), (ID of service 2), …, (ID of service n)} = {(i u,1 ), (i u,2 ), …, (i u,n )}. According to the number of interaction services and the leave-N-out strategy, the training, validation and test data sets are divided, a suitable N value is set, N interaction data is reserved as the validation set, N interaction data is reserved as the test set, and the rest is used as the training set.

[0032] As preferred, in step 5, each service description information in the behavior sequence file is subjected to nonlinear transformation by the multi-head attention layer (MH) and the feed-forward neural network (FFN) in the Transformer, and the generated hidden state at the last position n of the sequence is used as the representation form of the entire sequence:

[0033]

[0034] Then, the softmax function is used to score the sequence According to the formula:

[0035]

[0036] The score prediction is performed, where v A represents all items in A. Finally, the cross-entropy loss function is used to train in the field A. Save the model with the best performance on the specified indicators, and do the same for other fields.

[0037] As preferred, in step 6, the gradient is first clipped to the range [-τ, τ] and quantized to a discrete value:

[0038]

[0039] Then upload to the server, the server will be from the gradient of the field to flatten and splice:

[0040]

[0041] The server determines the weight according to the data volume ratio of the field, and uses the weighted sum average algorithm for gradient fusion:

[0042]

[0043] Update the code dictionary in the server and synchronize to the local client to enhance the local training before the G needs to be reshaped operation, reshape to the initial shape and decode to the original range.

[0044] As a preferred, the step 7 contains two prompting methods: soft prompting and hard prompting.

[0045] Soft prompting: i.e. field prompting P domain , which is composed of d W context words and field prompting encoder, and encoded through a multi-head attention layer (MA). The encoding process takes the sequence embedding h in the pre-trained model as the query, and calculates it by combining the prompt content to generate multiple attention heads, and gets the final prompt encoding through linear transformation. The calculation of each attention head includes linear transformation of query, key and value, and attention weight calculation.

[0046] Hard prompting: based on soft prompting, combined with user preference simulation construction, through LLM as a knowledge perception sampler, generate user-item pairs from training data. That is, identify hard samples with high prediction scores in the candidate set C u after soft prompting, which contains valuable positive and negative samples. It mainly includes the following sub-steps:

[0047] Step 7-2-1: Generate positive and negative samples

[0048] LLM selects positive samples and negative samples from the candidate items of user u according to the input prompt (contains task description, historical interaction, candidate item and output format), so as to construct enhanced user-item interaction triplets

[0049] Step 7-2-2: Construct enhanced dataset

[0050] The generated enhanced dataset ε A contains positive and negative sample triplets These samples are sampled by LLM and integrated into the original training set ε to form the joint training set ε∪ε A .

[0051] Step 7-2-3: Train the model

[0052] Optimize the loss function L based on contrast ranking using the joint dataset BPR . The loss function is optimized by the difference between the positive sample prediction score and the negative sample prediction score , and the formula is as follows:

[0053] Where σ(·) is the sigmoid activation function, |Θ| 2 is the weight decay regularization, and the weight is weighted by the parameter ω1. The augmented dataset ε A is a subset of LLM generated data, and its size is controlled by the batch size B and the rate ω2.

[0054] The present application has the following characteristics and beneficial effects:

[0055] The technical scheme is adopted, semantic information based on a large language model (LLM) is fully utilized, cross-domain knowledge transfer is realized, and performance and user experience of a recommendation system in multiple fields are improved. The LLM is used for deep semantic enhancement of item description text, specific field vocabulary is removed, general explanations are added, and text representation effect is optimized on the basis of preserving original information, so that application capability of semantic ID in different fields is improved, and knowledge transfer efficiency between fields is significantly enhanced. In order to uniformly model items in different fields, the text description of the item is quantitatively processed to be mapped to the same semantic space, and the item is represented by using the semantic ID. This method improves the generalization of item representation, and effectively protects the privacy information of users by combining the training framework of federated learning, avoiding the privacy leakage problem that may occur in traditional centralized training. In order to fully utilize field professional knowledge, two fine-tuning strategies, soft prompt and hard prompt, are designed. The soft prompt extracts common preferences shared by all users to fine-tune the recommendation result, and improve the overall recommendation quality. The hard prompt combines the LLM to screen the candidate recommendation result, so as to filter out unsuitable results, effectively reduce the influence of false positives (FP) and false negatives (FN), and further improve the accuracy and individualization performance of the recommendation system. The present application solves the problems of insufficient utilization of service text data, easy leakage of privacy data and low interpretability of recommendation results. The experiments on real data sets show that the user privacy protection cross-domain service recommendation method based on large language model enhancement has better recommendation ability than the traditional single-domain and cross-domain recommendation methods, and exceeds the traditional service recommendation algorithm in multiple indicators (Precision, Recall, etc.). Especially in the case of dealing with complex field differences and protecting user privacy, the method shows stronger robustness and practicality, and proves its superiority and broad prospects in practical application. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0057] Figure 1 The framework diagram of the large language model enhanced user privacy protection cross-domain service recommendation method embodiment of the present application.

[0058] Figure 2 The content schematic diagram designed for the large language model prompt in the present application embodiment. DETAILED DESCRIPTION

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0060] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0061] In the description of the present application, it should be noted that unless otherwise specified and limited, the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0062] The present application provides a large language model enhanced user privacy protection cross-domain service recommendation method, which improves the field similarity of the source domain and the target domain by using the world knowledge of the large language model (LLM). In order to transfer the user's preference without touching the user's sensitive privacy information, we use the federated learning framework to represent the project with semantic ID, and only upload the semantic ID information to the center server for fusion. Finally, in order to further improve the recommendation performance of the model in the field, we design two prompt fine-tuning methods: soft prompt and hard prompt. Soft prompt fine-tunes all users' common preferences in the field, and hard prompt fine-tunes according to each user's historical interaction sequence. The model diagram is as shown in Figure 1 The method comprises the following steps:

[0063] Step 1, screen the user's historical service interaction data and service metadata, where the interaction data includes user ID, service ID, service information (rating, etc.), and the time of interaction; the metadata contains the description information of the service, remove the interaction data with less than a specified number of interactions and the service without metadata. Design prompt, and input into LLM in turn with historical service interaction data and service metadata to obtain historical service interaction data containing user preferences and expanded enhanced service metadata.

[0064] Specifically, the text prompt includes a user attribute optimization prompt and an item content enhancement prompt The user attribute optimization prompt and the item content enhancement prompt User enhancement features A generated by a large language model u and item enhancement features A i Finally, the user enhancement features A u and item enhancement features A i are fused with the original features D u and D i of the service data set, and the formula is:

[0065]

[0066] where, represents feature concatenation or weighted fusion operation, and LLM(·) represents a large language model.

[0067] Step 2, when encoding the service description text using a transformer represented by a bidirectional encoder (BERT), a special input marker [CLS] is added in front of the initialization encoding T i ={w1,w2,…,w j ,…,w c} of each service description text, and then input into the BERT large language model to obtain the text embedding, i.e. x i =BERT([CLS;w1;…;w c ]), where [;] represents concatenation operation, w j is a content word in natural language, c is the truncation length of the item text, and vector represents the representation of the given text.

[0068] Step 3, convert the text embedding into discrete code using the PQ method, represent the service with code, and the product quantization method includes: divide the text embedding vector into D sub-vectors, select the nearest neighbor centroid index from the centroid set as the discrete code for each sub-vector, and perform pooling through the centroid embedding in the code dictionary to generate the final item representation vector.

[0069] In particular, the product quantization method (PQ) is:

[0070] 1) Sub-vectorization: Define D groups of vectors, where each vector corresponds to a dimension d W / D and contains M c centroids. The text encoding vector x i is first divided into D sub-vectors, i.e.,

[0071] x i = [x i,1 ,…, x i,D ]

[0072] 2) Discrete mapping: Let a k,j ∈ d W / D denote the j-th centroid embedding in the k-th group of vectors. For each k-th sub-vector x i,k , PQ selects the nearest centroid embedding from the corresponding centroid set, generating the discrete encoding of x i,k :

[0073]

[0074] where c i,k is the k-th dimensional discrete encoding vector of item i.

[0075] 3) Maintain codebook: The text encoding vector x i is split into D sub-vectors, each representing the projection of the original vector in a different subspace. A codebook (a set of pre-defined centroids) is created for each subspace, and these codebooks collectively form the code embedding dictionary. Each sub-vector is quantized to the nearest centroid in the codebook, and the index of the centroid serves as the “code” for that sub-vector. The quantization results of all sub-vectors are combined into a code vector, which is used to represent the original high-dimensional vector.

[0076] The corresponding centroids are retrieved from the global code embedding dictionary through lookup operations. These centroids are then combined into the final item representation vector through average pooling methods. These embedding representations are stored in the code embedding dictionary for subsequent retrieval and matching.

[0077] Each domain maintains a code embedding dictionary where d v denotes the dimension of the item embedding. There are D code embedding matrices in T, each matrix is shared among all items in the server (not limited to a single domain). This feature allows us to align different domains and embed general domain information into the item embeddings.

[0078] For retrieval operations on T, the code embedding of item i can be represented as where is the matrix T (k) th row of the embedding matrix V The code embedding is obtained by average pooling the final representation vector of item i:

[0079]

[0080] where v i is the final representation of the item, is the average pooling method in D dimensions, and the code embedding of item i can be represented as

[0081] Step 4, read and process the user's historical service interaction data, sort the historical service interaction records of each user according to the timestamp order, and generate the corresponding user behavior sequence file. Each line in the file represents a user's service interaction sequence. Then use the leave-N-out strategy to divide the training, validation and test data sets.

[0082] Specifically, the service IDs corresponding to the user user interaction and the corresponding categories are sorted in the order of interaction time, separated by commas; the form is:

[0083] S u ={(service 1 ID), (service 2 ID), …, (service n ID)}={(i u,1 ), (i u,2 ), …, (i u,n )}

[0084] According to the number of interactive services and the leave-N-out strategy, divide the training, validation and test data sets, set a suitable N value, reserve N interaction data as the validation set, reserve N interaction data as the test set, and the rest as the training set.

[0085] Step 5, convert the service description information in the user behavior sequence into discrete code representation, input the sequence encoder based on Transformer for local model training, and save the model with the best performance.

[0086] Specifically, in step 5, each service description information in the behavior sequence file is represented and converted to the code {v1, v2, …, v i ,…, v n} that can be used by the Transformer sequence encoder. For each input item representation v i , add the corresponding position embedding p j (j is the position of item i in the sequence): Then The multi-head attention layer (MH) and feed-forward neural network (FFN) in the Transformer are passed to perform non-linear transformations, as follows: H l+1 = FFN(MH(H l )), l denotes a layer in the feed-forward neural network. After the transformation, at the last position n of the sequence, the generated hidden state is used as a representation of the entire sequence. This representation combines all the information in the input sequence and takes into account the dependencies between positions:

[0087]

[0088] The softmax function is then used to predict the scores for the sequence According to the formula:

[0089]

[0090] The score prediction is made, where v A represents all the items in A. Finally, the cross-entropy loss function is used to train in the field A. The model with the best performance on the specified indicators is saved, and the same is done for other fields.

[0091] Step 6: Under the federated learning framework, each client uploads the quantized and clipped parameter gradients of the local model to the server. The server fuses the gradients according to the data volume weight and updates the code dictionary, which is synchronized to each client.

[0092] Specifically, the gradient needs to be clipped to the range [-τ, τ] and quantized to a discrete value:

[0093]

[0094] Then upload to the server, the server will flatten and concatenate the gradients from each field:

[0095]

[0096] The server determines the weight according to the data volume ratio of the field and uses the weighted summation average algorithm to fuse the gradients:

[0097]

[0098] Update the code dictionary in the server and synchronize it to the local client to enhance local training. Before updating the code dictionary in the server and synchronizing it to the local client to enhance local training, the G needs to be reshaped to the initial shape and decoded to the original range. Update the code dictionary in the server and synchronize it to the local client to enhance local training. Repeat the above operations until the pre-training phase model converges.

[0099] Step 7, fine-tuning the model with soft and hard prompts.

[0100] Specifically, soft prompt: extract specific content from the common preferences of all users in each field, and encode the field prompt P with context words and field prompt encoder domain , which consists of d W context words and is encoded through a multi-head attention layer (MA). The encoding process takes the sequence embedding h in the pre-trained model as the query, and combines the prompt content to calculate multiple attention heads, and through linear transformation to get the final prompt encoding. The calculation of each attention head includes linear transformation of query, key and value, and attention weight calculation, so as to capture the correlation between sequence embedding and field prompt.

[0101] Hard prompt: based on soft prompt, combined with user preference simulation construction, through LLM as knowledge perception sampler, generate user-item pairs from training data. Specifically, the hard prompt provides auxiliary information (such as year, category) for each user's historical interaction and candidate item pool in the form of text C u = {i u,1 , i u,2 ,..., i u,|Cu| ,}. Since LLM cannot comprehensively rank all items, it will identify hard samples with high prediction scores in the candidate set C u after soft prompt. These hard samples contain valuable positive and negative samples, providing reinforced training signals for model training.

[0102] The main steps of the hard prompt process include:

[0103] 1. Generate positive and negative samples

[0104] LLM selects positive samples and negative samples from the candidate items of user u according to the input prompt (containing task description, historical interaction, candidate item and output format), so as to construct enhanced user-item interaction triplets

[0105] 2. Construct enhanced dataset

[0106] The generated enhanced dataset ε A contains positive and negative sample triplets These samples are sampled by LLM and integrated into the original training set ε to form the joint training set ε∪ε A .

[0107] 3. Train the model

[0108] Optimizing the loss function L based on contrastive ranking using joint dataset BPR The loss function is optimized by the difference between the positive sample prediction score and the negative sample prediction score , as follows:

[0109] Where σ(·) is the sigmoid activation function, |Θ| 2 is the weight decay regularization, and the weights are weighted by the parameter ω1. The augmented dataset ε A is a subset of the LLM generated data, whose size is controlled by the batch size B and the rate ω2.

[0110] Based on the user privacy protection cross-domain service recommendation method provided by the large language model enhancement of the embodiment, a specific simulation example is provided: the cross-domain recommendation model used is denoted as PCR-LLM.

[0111] The original data used in this embodiment are five widely used real scenario data sets: Scientific, Arts, Pantry, Movies, and Online Retail. Among them, Scientific, Art, Pantry, Movie are all from Amazon, and Online dataset is used to verify the robustness of the model, to avoid all data coming from Amazon. In the Scientific dataset, there are 8,443 users and 4,385 services with 50,981 interactions, and the data sparsity rate is 99.8623%; In the Arts dataset, there are 45,487 users and 21,019 services with 349,664 interactions, and the data sparsity rate is 99.9634%; In Pantry, there are 13,102 users and 4,898 users with 113,861 interactions, and the data sparsity rate is 99.8226%; In the Movies dataset, there are 281,701 users and 59,204 services with 2,942,031 interactions, and the data sparsity rate is 99.9925%; In the Online Retail dataset, there are 16,521 users and 3,469 services with 503,386 interactions, and the data sparsity rate is 99.1217%. All datasets are filtered and screened to remove user interaction records with few interactions and services with missing metadata.

[0112] In this embodiment, the partial parameters of PCR-LLM are selected as follows: the Token of each input in the pre-trained language model is mapped to a 768-dimensional vector; the number of subspaces in the product quantization method is set to 32; the iteration termination condition set in the pre-training stage of the cross-domain recommendation model is that the total loss function value converges or the iteration round reaches 100 rounds; and the iteration termination condition in the model fine-tuning stage is that there is no improvement in the evaluation index within 10 rounds or the iteration round reaches 100 rounds.

[0113] In addition, the experiments of this embodiment also compare the method of the application with several traditional prediction methods. The single-domain traditional prediction methods used as controls are: (1) SASRec: a self-attention sequential recommendation method; (2) BERT4Rec: a transformer-based bidirectional encoder representation sequential recommendation method;

[0114] (3) GRU4Rec: a session-based recommendation method using a recurrent neural network. The multi-domain traditional prediction methods are: (1) CCDR: a contrastive cross-domain recommendation method in matching; (2) RecGURU: an adversarial learning method for generalized user representation for cross-domain recommendation. This embodiment uses Recall and NDCG as the evaluation indicators of the prediction model. Recall (Recall) measures the proportion of relevant K items recommended by the model in all relevant items, and is used to evaluate the coverage ability of the recommendation system. NDCG (Normalized Discounted Cumulative Gain) is a ranking quality evaluation indicator, which measures the relevance and ranking quality of the recommendation results. In the experiment, K = {10, 50} is set.

[0115] Table 1 and Table 2 Comparison of experimental results of the method of the application and the control method

[0116] Table 1: Performance Comparison on Pantry and Online

[0117]

[0118] Table 2: Performance Comparison on Scientific and Arts

[0119]

[0120] The final experimental results are shown in Tables 1 and 2, and it can be seen that the PCR-LLM in the method of the application achieves better results on the Pantry-Online and Scientific-Arts data sets compared with the control model. Specifically, in the Recall index, the PCR-LLM model is improved by at least 1.30% and at most 32.43% compared with the single-domain recommendation method, and is improved by at least 5.50% and at most 35.46% compared with the multi-domain recommendation method; in the NDCG index, it is improved by at least 1.68% and at most 37.92% compared with the single-domain recommendation method, and is improved by at least 2.22% and at most 37.95% compared with the multi-domain recommendation method. It is worth mentioning that after two fine-tuning of soft prompt and hard prompt, the performance is steadily improved, with a minimum improvement of 9.77% and a maximum improvement of 36.90%. It can be seen that the large language model significantly improves the model recommendation performance.

[0121] The embodiments of the application are described in detail above with reference to the drawings, but the application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments including components can be made without departing from the principles and spirits of the application, and still fall within the protection scope of the application.

Claims

1. A large-scale language model-enhanced user privacy-preserving cross-domain service recommendation method, characterized in that: The following steps are involved: Step 1: Filter the service datasets of the source and target domains. The service datasets include service description text and user attribute information. First, filter the service datasets. Then, use the preset text prompts and a large language model to perform semantic enhancement and fusion on the filtered service datasets. Step 2: Use the transformer represented by the bidirectional encoder to encode the enhanced service description text to generate a high-dimensional text embedding; Step 3: Mapping the high-dimensional text embedding into discrete codes through a product quantization method to construct a code dictionary, wherein the code dictionary includes a set of centroids of multiple subspaces; Step 4: Read the user's historical service interaction data, generate a user behavior sequence file in timestamp order, and use the leave-N-out strategy to divide it into training set, validation set and test set; Step 5: Convert the service description information in the user behavior sequence into a discrete code representation, input it into the Transformer-based sequence encoder for local model training, and save the model with the best performance; Step 6: Under the federated learning framework, each client clips and quantizes the parameter gradients of the local model and uploads them to the server. The server fuses the gradients based on the data volume weights and updates the code dictionary, which is then synchronized to each client. Step 7: Fine-tune the model using soft and hard hints.

2. The method according to claim 1, characterized in that In step 1, the filtering method is to remove interaction data with interaction volume lower than a preset threshold and services with missing metadata.

3. The method according to claim 1, characterized in that In step 1, the text prompt includes a user attribute optimization prompt And item content enhancement prompt P i P , the user attribute optimization tips And item content enhancement prompt P i P User-enhanced features A generated by a large language model u and item enhancement feature A i , and finally the user enhanced feature A u and item enhancement feature A i and the original features D of the service dataset u and D i To perform fusion, the formula is: in, represents feature concatenation or weighted fusion operation, and LLM(·) represents a large language model.

4. The method according to claim 1, wherein In step 2, when the transformer represented by the bidirectional encoder is used to encode the service description text, the initialization code T of the description text of each service is i ={w1,w2,…,w j ,…,w c } Add a special input tag [CLS] before it, that is, x i =BERT([CLS; w1; ...; w c ]), where [;] represents the splicing operation, w j is the content word in natural language, c is the truncated length of the item text, and the vector Represents a representation of the given text.

5. The method according to claim 1, wherein In step 3, the product quantization method includes: dividing the text embedding vector into D sub-vectors, selecting the nearest neighbor centroid index from the centroid set as a discrete code for each sub-vector, and pooling it through the centroid embedding in the code dictionary to generate a final item representation vector.

6. The method according to claim 5, characterized in that The pooling operation is average pooling, and the formula is: Among them, v i is the final representation of the item, It is an average pooling method in D dimensions. The code embedding of item i can be expressed as 7. The method according to claim 1, characterized in that In step 4, in each user behavior sequence file, the service IDs and corresponding categories that the corresponding user has interacted with are sorted in the order of interaction time and separated by commas; the format is: S u ={(ID of service 1),(ID of service 2),…,(ID of service n)}={(i u,1 ),(i u,2 ),…,(i u,n )} Divide the training, validation, and test data sets according to the number of interactive services and the leave-N-out strategy, set an appropriate N value, retain N interactive data as the validation set, retain N interactive data as the test set, and the rest as the training set.

8. The method according to claim 1, characterized in that In step 6, the local needs to first Clipped to the range [-τ,τ] and quantized to discrete values: Then upload it to the server, which flattens and concatenates the gradients from each field: The server determines the weights based on the data volume ratio of the fields and uses the weighted sum average algorithm for gradient fusion: Update the code dictionary in the server and synchronize it to the local client. Before strengthening local training, G needs to be reshaped to its initial shape and decoded to its original range.

9. The method according to claim 1, characterized in that In step 7, the soft prompt fine-tuning is to generate prompt context words based on the common preferences of users in the field, and optimize the recommendation results after encoding through the multi-head attention layer.

10. The method according to claim 1, characterized in that In step 7, the specific steps of fine-tuning the hard prompt include: Use a large language model to generate positive and negative sample pairs based on user historical interactions and candidate service information Then construct the enhanced dataset ε A , where the enhanced dataset ε A Contains positive and negative sample triplets And combined with the original training set to optimize the comparison ranking loss function L BPR .

11. The method according to claim 10, characterized in that The formula of the comparative ranking loss function is: in, and denote the predicted scores of positive and negative samples respectively, ω1 is the weight parameter, σ(·) is the sigmoid activation function, |Θ| 2 Heavy decay regularization.

12. The method according to claim 1, characterized in that In step 5, the sequence encoder uses a multi-head self-attention layer and a position-aware feedforward neural network and is trained using a cross-entropy loss function.

13. The method according to any one of claims 1 to 12, characterized in that The large language model includes GPT, BERT or its variants, which are used for text enhancement, semantic encoding or candidate service screening.

Images