A multi-modal recommendation method and system based on federated learning

Abstract

The application discloses a multi-modal recommendation method and system based on federated learning, relates to the technical field of computer information recommendation, and aims at solving the problems that the existing method cannot guarantee that the data of multiple parties does not leave the local, cannot cut off the privacy leakage of the data in the uploading process, and cannot realize high-precision recommendation. The application comprises the following steps: step 1, multiple clients respectively preprocess local data sets; step 2, a server end constructs a multi-modal model and a loss function and then delivers the multi-modal model and the loss function to the clients; step 3, the clients train the model, upload optimal model parameters to the server end; step 4, the server end aggregates the model parameters to obtain an aggregated model, trains the aggregated model, and then delivers the aggregated model to the clients again; and step 5, the clients verify and test the model again, obtain a multi-modal recommendation model, and realize federated multi-modal recommendation. The application effectively fuses the information of multiple distributed data sources and realizes personalized recommendation under the premise of protecting the privacy of users.

Description

Technical Field

[0001] This invention relates to the field of computer information recommendation technology, and more specifically, to a multimodal recommendation method and system based on federated learning. Background Technology

[0002] With the rapid development of big data technology and the improvement of residents' consumption concepts, more and more people are joining the "online shopping army." However, internet data is experiencing explosive growth and is still showing a trend of exponential increase. With the emergence of massive amounts of information, people find it difficult to find the products they want amidst this vast amount of data, thus leading to the development of personalized recommendation methods. However, most recommendation methods are based on unimodal information. Multimodal recommendation methods typically utilize randomly sampled negative examples in the Bayesian personalized ranking loss to guide the learning of user / item representations. This increases the computational cost of large graphs and may also introduce noisy supervision signals into the training process.

[0003] In most cases, data is scattered across various companies. These companies hope to collaborate with other companies (using the data held by each company) to train a model that can help them gain greater benefits, without disclosing their own data. However, with the enactment of relevant data privacy protection laws both domestically and internationally, companies directly using data from multiple parties has led to privacy leaks. As people's demand for privacy protection grows, protecting privacy and security has become increasingly important.

[0004] Therefore, how to ensure that data from multiple parties does not leave their local machines while simultaneously preventing privacy leaks during data upload and improving the accuracy of recommendation algorithms in the context of multimodal recommendation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] Existing multimodal recommendation methods struggle to ensure that data from multiple parties remains locally, prevent privacy leaks during data uploads, and achieve high-precision recommendations.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0008] This invention provides a multimodal recommendation method based on federated learning. The method is based on a federated learning model and includes one server and multiple clients, comprising the following steps:

[0009] Step 1: Multiple clients preprocess the local dataset, which includes text and images. The preprocessed dataset is divided into training set, validation set and test set.

[0010] Step 2: The server constructs a multimodal model and loss function, and then distributes the multimodal model and loss function to each client.

[0011] Step 3: Each client trains the model distributed by the server based on the training set, updates the network parameters through backpropagation, verifies the model performance using the validation set, tests the model in each round of training using the test set, and uploads the model parameters with the best test results to the server.

[0012] Step 4: After receiving the model parameters uploaded by each client, the server aggregates the model parameters uploaded by different clients to obtain an aggregated model. The server trains the aggregated model based on the local dataset and then sends the trained aggregated model back to the client.

[0013] Step 5: The client re-initializes the model sent by the server and validates and tests the model based on the validation set and test set to obtain a multimodal recommendation model. The model obtains the recommendation results of candidate items based on the inner product of the user embedding and the embeddings of all candidate items, thus realizing federated multimodal recommendation.

[0014] Further, in step 1, the data is preprocessed. The specific process is as follows: First, UI interaction graphs are extracted from the dataset, where U represents users and I represents products. Second, the k-score filtering method is used to redistribute the index values ​​of the interaction graphs to generate k-score interaction graphs. Based on the generated k-score interaction graphs, the image and text features of the products are matched one-to-one to form a dataset.

[0015] The text and images in the dataset are converted into continuous vectors, and then into embedding vectors respectively. The image features and text features are merged according to the same user ID to obtain the preprocessed dataset.

[0016] Furthermore, the k-score filtering method has a k value of 5, and the generated k-score interaction graph consists of userID, itemID, rating, timestamp, and x_label information, where userID represents the user ID, itemID represents the product ID, rating represents the user's rating of the product, timestamp represents the timestamp, and x_label represents the corresponding label.

[0017] Furthermore, the image and text features of the product are mapped one-to-one, and the generated dataset includes: itemID, asin, title, price, imUrl, related, brand, categories, salesRamk, and description;

[0018] In this context, asin represents the product code, title represents the product name, price represents the product price, imUrl represents the product image link, related represents related products, salesRank represents discount information, brand represents the product brand, and categories represents the category to which the product belongs.

[0019] Furthermore, the method for constructing the loss function in step 2 is as follows:

[0020] Step 21: Input the preprocessed user data and project data into the distributed model to obtain modality-specific feature representations. Where m∈M represents a specific mode from the complete mode M, d m The dimension of the feature is represented; it is projected onto a latent low dimension using a projection function fm based on a multilayer perceptron (MLP).

[0021] Step 22: Encode the structure of the UI interaction diagram using a backbone network with residual connections;

[0022] Step 23: Using a multimodal latent space transformer, obtain three types of latent embeddings, including: user ID embedding, item ID embedding, and unimodal item embedding;

[0023] Step 24: Input the three embedded inputs into the comparison view generator to generate three comparison views;

[0024] Step 25: Construct the final loss function based on the three views.

[0025] Furthermore, step 21 is based on the following method:

[0026] h m =e m W m +b m

[0027] in, Describe the projection function f m Linear transformation matrix and bias in MLP.

[0028] Furthermore, in step 24, the comparison view generator embedding rate p is used to compare the comparison view. The method is as follows:

[0029]

[0030] Wherein, the input vector h represents the feature or representation to be processed, Bernoulli(p) represents a random variable that follows a Bernoulli distribution, where p is the probability of success and is used to control the probability of discarding elements, and h·Bernoulli(p) represents the element-level random discarding operation on the input vector h, where each element is independently set to 0 with probability p and retained with probability (1-p).

[0031] Furthermore, the final loss function obtained in step 25 is:

[0032]

[0033] in and For regularization penalties;

[0034] For graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the following is a partial translation of the original text, but the full context is unclear: "for graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the remaining text is unclear: ... and Stop gradient operator:

[0035]

[0036] For cross-channel feature alignment loss, let cosine similarity be calculated for each single-modal latent embedding of item i:

[0037]

[0038] The loss is the internal modal feature masking loss, which represents the loss for random embedding h. m Perform masking. To represent sparse embedding, we obtain the mask loss:

[0039]

[0040] Furthermore, in step 4, the model is aggregated, specifically by weighting the model parameters of each client to obtain the aggregated model.

[0041] A multimodal recommendation method system based on federated learning is provided. The system has program modules corresponding to the steps of any of the above-mentioned technical solutions, and executes the steps of the multimodal recommendation method based on federated learning during runtime.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention presents a multimodal recommendation method and system based on federated learning. Leveraging the advantages of federated learning, it effectively integrates information from multiple distributed data sources while protecting user privacy, enabling personalized recommendations. This method not only provides more accurate and personalized recommendation results, further aligns learned features across different modalities, and reduces discrepancies between different augmented view representations from specific modalities, but also employs a simple latent representation exit mechanism, rather than graph augmentation, to generate target views of users or items for contrastive learning without negative samples, thus improving user experience and system efficiency.

[0044] The method of this invention can effectively process distributed data, and its scalability and adaptability make it suitable for various multimodal recommendation scenarios, including images and text. It has promising application prospects. Attached Figure Description

[0045] Figure 1 This is a flowchart of the multimodal recommendation method based on federated learning in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating the data preprocessing process in an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of the multimodal recommendation model in an embodiment of the present invention. Detailed Implementation

[0048] In the description of this invention, it should be noted that the terms "first," "second," and "third" mentioned in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," and "third" may explicitly or implicitly include one or more of that feature.

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] Specific Implementation Plan 1: Combining Figures 1 to 3 As shown, this invention provides a multimodal recommendation method based on federated learning. The method is based on a federated learning model and includes one server and multiple clients, comprising the following steps:

[0051] Step 1: Multiple clients preprocess the local dataset, which includes text and images. The preprocessed dataset is divided into training set, validation set and test set.

[0052] Step 2: The server constructs a multimodal model and loss function, and then distributes the multimodal model and loss function to each client.

[0053] Step 3: Each client trains the model distributed by the server based on the training set, updates the network parameters through backpropagation, verifies the model performance using the validation set, tests the model in each round of training using the test set, and uploads the model parameters with the best test results to the server.

[0054] Step 4: After receiving the model parameters uploaded by each client, the server aggregates the model parameters uploaded by different clients to obtain an aggregated model. The server trains the aggregated model based on the local dataset and then sends the trained aggregated model back to the client.

[0055] Step 5: The client re-initializes the model sent by the server and validates and tests the model based on the validation set and test set to obtain a multimodal recommendation model. The model obtains the recommendation results of candidate items based on the inner product of the user embedding and the embeddings of all candidate items, thus realizing federated multimodal recommendation.

[0056] Specific Implementation Plan Two: (e.g.) Figure 2 As shown, in step 1, the data is preprocessed. The specific process is as follows: First, UI interaction graphs are extracted from the dataset, where U represents users and I represents products. Second, the k-score filtering method is used to redistribute the index values ​​of the interaction graphs to generate k-score interaction graphs. Based on the generated k-score interaction graphs, the image and text features of the products are matched one-to-one to form a dataset.

[0057] The text and images in the dataset are converted into continuous vectors, and then further converted into embedding vectors. Image features and text features are merged based on the same user ID to obtain the preprocessed dataset. The rest of this implementation scheme is the same as in Implementation Scheme 1.

[0058] The preprocessing of the dataset in this implementation plan also includes modifying missing values, sorting out duplicate data, and adding or removing labels.

[0059] Specific Implementation Scheme 3: The k-score filtering method uses a value of 5 for k. The generated k-score interaction graph consists of userID, itemID, rating, timestamp, and x_label information. Here, userID represents the user ID, itemID represents the product ID, rating represents the user's rating of the product, timestamp represents the timestamp, and x_label represents the corresponding label. All other aspects of this implementation scheme are the same as in Specific Implementation Scheme 2.

[0060] In this implementation scheme, the generated 5-score interaction graph is divided into a training set, a validation set, and a test set, and the x_label is renumbered.

[0061] Specific implementation plan four: The image and text features of the product are mapped one-to-one, and the generated dataset includes: itemID, asin, title, price, imUrl, related, brand, categories, salesRamk, and description;

[0062] Wherein, asin represents the product code, title represents the product name, price represents the product price, imUrl represents the product image link, related represents related products, salesRank represents discount information, brand represents the product brand, and categories represents the product category. This implementation plan is otherwise the same as specific implementation plan three.

[0063] In this implementation, the asin column is used as the key and the itemID column is used as the value to form a dictionary, which is used to associate image features with corresponding items and save them as a file.

[0064] Specific implementation plan five: The method for constructing the loss function in step 2 is as follows:

[0065] like Figure 3 As shown, step 21 involves inputting the preprocessed user data and project data into the distributed model to obtain modality-specific feature representations. Where m∈M represents a specific mode from the complete mode M, d m The dimension of the feature is represented; it is projected onto a latent low dimension using a projection function fm based on a multilayer perceptron (MLP).

[0066] Step 22: Encode the structure of the UI interaction diagram using a backbone network with residual connections;

[0067] Step 23: Using a multimodal latent space transformer, obtain three types of latent embeddings, including: user ID embedding, item ID embedding, and unimodal item embedding;

[0068] Step 24: Input the three embedded inputs into the comparison view generator to generate three comparison views;

[0069] Step 25: Construct the final loss function based on the three views. This implementation scheme is otherwise the same as Implementation Scheme 1.

[0070] Specific implementation plan six: Step 21 is based on the following method:

[0071] h m=e m W m +b m

[0072] in, Describe the projection function f m The linear transformation matrix and bias in the MLP. This implementation scheme is otherwise the same as specific implementation scheme five.

[0073] In this implementation scheme, after the data is loaded, it is placed into the multimodal latent space converter. Indicates user Item i ∈ I, where d is the embedding dimension, and U and I are the sets of users and items, respectively. Their cardinality is set to |U| and |I|, respectively. This implementation represents the modality-specific features obtained from the pre-trained model as follows: Where m∈M represents a specific mode from the complete mode M, d m The size of the feature is represented by |M|. The cardinality of M is denoted by |M|. Consider two modes: visual v and text t. However, the model can be easily extended to scenarios with more than two modes. Since the multimodal feature spaces are different from each other, the multimodal features and ID embeddings are first transformed into the same latent space. The features of items obtained from different forms have different dimensions and different feature spaces.

[0074] In step 22, the structure of the UI interaction graph is encoded using a backbone network with residual connections. Let... It is a set of nodes Given a graph with edge set ε, the number of nodes is denoted by |V|, and the number of edges is denoted by |ε|. The adjacency matrix is ​​represented as... The diagonal degree matrix is ​​represented by D. In In the diagram, edges describe the observed user-item interactions. If a user interacts with an item, an edge is constructed between the user node and the item node. Furthermore, this implementation uses... This represents the ID embedding at the l-th layer, which is the sum of all embeddings of users and items at the l-th layer. Specifically, the initial ID embedding H... 0 It is an embedded e from all users and projects u and e i The set. Recursively compute a typical feedforward graph convolutional network embedding H in layer l+1. l+1 Hidden ID:

[0075]

[0076] Where σ(·) is a nonlinear function. For the renormalization of the adjacency matrix A, D is the diagonal matrix of A+I. For node classification, the last layer of the graph convolutional network is used to predict the node labels through a soft-maximization classifier.

[0077] Based on the ordinary graph network, the feature transformation W is removed. l The recommendation is simplified by using a non-linear activation σ(·) layer. However, these two layers negatively impact recommendation performance. The simplified graph convolutional layer is defined as follows:

[0078]

[0079] Where (l+1) represents the node embedding of the th hidden layer, obtained only through the transition matrix D. -1 / 2 AD -1 / 2 Linear aggregation from the l-th layer. The transition matrix is ​​the weighted adjacency matrix mentioned above.

[0080] All representations are aggregated into the hidden layer using a single read function to achieve the final representation of the user and the item. Oversmoothing issues may exist during this process, therefore, items are initially embedded in the item. Add a residual join to obtain the final representation of the project, as shown in the following formula:

[0081]

[0082]

[0083] The READout function can be any differentiable function. The default mean function of the Lightweight Graph Network is used for the final ID embedding update.

[0084] Step 23 utilizes a multimodal latent space transformer to obtain three types of latent embeddings: user ID embedding, item ID embedding, and unimodal item embedding. This simplifies the current SSL paradigm and delays data augmentation after online network encoding.

[0085] Specific Implementation Scheme Seven: Comparison Views under the Embedding Rate p of the Comparison View Generator in Step 24 The method is as follows:

[0086]

[0087] Wherein, the input vector h represents the feature or representation to be processed, Bernoulli(p) represents a random variable following a Bernoulli distribution, where p is the probability of success, used to control the probability of discarding elements, and h·Bernoulli(p) represents the element-level random discarding operation on the input vector h, where each element is independently set to 0 with probability p, and retained with probability (1-p). This implementation scheme is otherwise the same as specific implementation scheme five.

[0088] In this implementation scheme, in order to reduce computational complexity and memory costs, the multimodal recommendation model uses a simple stochastic latent embedding technique (similar to node embedding) to eliminate the need for graph augmentation.

[0089] Specific implementation plan eight: The final loss function obtained in step 25 is:

[0090]

[0091] in and For regularization penalties;

[0092] For graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the following is a partial translation of the original text, but the full context is unclear: "for graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the remaining text is unclear: ... and Stop gradient operator:

[0093]

[0094] For cross-channel feature alignment loss, let cosine similarity be calculated for each single-modal latent embedding of item i:

[0095]

[0096] The loss is the internal modal feature masking loss, which represents the loss for random embedding h. m Perform masking. To represent sparse embedding, we obtain the mask loss:

[0097] This implementation plan is otherwise the same as Specific Implementation Plan Five.

[0098] In this implementation, the stopping gradient is placed in the comparison view. At the same time, the original embedding h is input into the MLP predictor.

[0099]

[0100] in The prediction function f p The linear transformation matrix and deviation in the equation.

[0101] The model receives a positive user item pair (u, i) as input. It is based on an online representation. Generated comparison view Define a symmetric loss function as and Negative cosine similarity between them:

[0102]

[0103] In the above formula, the function C(·,·) is defined as:

[0104]

[0105] Where ||·||2 is the l2-norm. The total loss is the average of all user-item pairs. Then, the prediction of user u by the positive perturbation term i is maximized, and vice versa. The minimum possible value of this loss is -1.

[0106] Finally, we stop gradient propagation on the target network and force the loss to backpropagate only on the online network. The updated operator is as follows:

[0107]

[0108] With the stopping gradient operator, the target network will not receive... The gradient.

[0109] Furthermore, the multimodal features of an item are aligned with its target ID embedding. This alignment encourages embedding IDs close to each other on items with similar multimodal features. For each unimodal latent embedding h of item i... m The comparison view generator outputs its comparison pair as Using negative cosine similarity pairs and Compare:

[0110]

[0111] Finally, the multimodal recommendation model uses intramodal feature masking loss to further encourage the use of sparse representations of latent embeddings to learn the predictor. In large transformers, sparsity has proven to be scalable. Utilizing a contrastive view generator, the implicit embedding h is processed using a random latent embedding technique. m Perform random masking and represent sparse embedding as Intra-modal feature masking loss is defined as:

[0112]

[0113] Additionally, a regularization penalty (i.e., h) is added to the online embedding. u and h i ), thus obtaining the final loss function.

[0114] Specific Implementation Plan Nine: Step 4 involves aggregating the models, specifically by weighted averaging the model parameters from each client to obtain the aggregated model. The rest of this implementation plan is the same as Specific Implementation Plan One.

[0115] After receiving the uploaded model parameters and gradients, the server in this implementation scheme adds these parameters and gradients together and calculates the average aggregated parameters based on the number of clients, thus obtaining a new aggregated model.

[0116] In this implementation, the interaction score between the user and candidate items is first predicted. Then, the candidate items are sorted in descending order based on the predicted interaction score, and the top K items are selected as recommendations to the user. The top-K items are recommended by scoring the inner product of the user embedding and all candidate item embeddings. The multimodal recommendation model can predict the latent embeddings of users and items well, using the embeddings transformed by the predictor fp for the inner product. That is:

[0117]

[0118] Where h u and h i It is the input vector, which can be represented as the hidden state, feature vector, or other type of vector representation in a neural network. By calculating h... i and h i The inner product, i.e. You can obtain a scalar value that represents the similarity or correlation between two vectors.

[0119] Finally, a list of recommended products based on the ratings is generated. The higher the similarity and relevance, the higher the predicted user score for the product, and the higher the ranking score, indicating that the user prefers the item.

[0120] Specific Implementation Scheme 10: A multimodal recommendation method system based on federated learning, which has program modules corresponding to the steps of any one of the above specific implementation schemes 1 to 9, and executes the steps in the above multimodal recommendation method based on federated learning when running.

[0121] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multimodal recommendation method based on federated learning, characterized in that, The method is based on a federated learning model, includes one server and multiple clients, and includes the following steps: Step 1: Multiple clients preprocess the local dataset, which includes text and images. The preprocessed dataset is divided into training set, validation set and test set. Step 2: The server constructs a multimodal model and loss function, and then distributes the multimodal model and loss function to each client. Step 3: Each client trains the model distributed by the server based on the training set, updates the network parameters through backpropagation, verifies the model performance using the validation set, tests the model in each round of training using the test set, and uploads the model parameters with the best test results to the server. Step 4: After receiving the model parameters uploaded by each client, the server aggregates the model parameters uploaded by different clients to obtain an aggregated model. The server trains the aggregated model based on the local dataset and then sends the trained aggregated model back to the client. Step 5: The client re-initializes the model sent by the server and validates and tests the model based on the validation set and test set to obtain a multimodal recommendation model. The model obtains the recommendation results of candidate items based on the inner product of the user embedding and all candidate item embeddings, thus realizing federated multimodal recommendation. The method for constructing the loss function in step 2 is as follows: Step 21: Input the preprocessed user data and project data into the distributed model to obtain modality-specific feature representations. ,in Indicates origin from complete mode Specific modes, The dimension representing the feature; Using a projection function based on a multilayer perceptron (MLP) Project it onto a potential low dimension; Step 22: Encode the structure of the UI interaction diagram using a backbone network with residual connections; Step 23: Using a multimodal latent space transformer, obtain three types of latent embeddings, including: user ID embedding, item ID embedding, and unimodal item embedding; Step 24: Input the three embedded inputs into the comparison view generator to generate three comparison views; Step 25: Construct the final loss function based on the three views; Step 21 is based on the following method: in, , Represents the projection function Linear transformation matrix and bias in MLP; In step 24, compare the embedding rate of the view generator. Comparison view below The method is as follows: Here, the input vector h represents the feature or representation to be processed. Let represent a random variable that follows a Bernoulli distribution, where p is the probability of success, used to control the probability of discarding an element. This indicates that an element-level random discard operation is performed on the input vector h, where each element is independently set to 0 with probability p and retained with probability (1-p). The final loss function obtained in step 25 is: in and For regularization penalties; For graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the following is a partial translation of the original text, but the full context is unclear: "for graph reconstruction loss, let be the user and item pair (u, i) receiving positive feedback as input, based on the contrast view generated from the online representation, and define a symmetric loss function, then calculate the cosine similarity between them, and [the remaining text is unclear: ... and Stop gradient operator: For cross-channel feature alignment loss, it represents the loss for the project. Calculate the cosine similarity for each unimodal latent embedding: The loss is the inner modality feature mask, which represents the loss for random embeddings. Perform masking. To represent sparse embedding, we obtain the mask loss: 。 2. The multimodal recommendation method based on federated learning according to claim 1, characterized in that, Step 1 involves preprocessing the data. The specific process is as follows: First, UI interaction graphs are extracted from the dataset, where U represents users and I represents products. Second, the k-score filtering method is used to redistribute the index values ​​of the interaction graphs to generate k-score interaction graphs. Based on the generated k-score interaction graphs, the image and text features of the products are matched one-to-one to form a dataset. The text and images in the dataset are converted into continuous vectors, and then into embedding vectors respectively. The image features and text features are merged according to the same user ID to obtain the preprocessed dataset.

3. The multimodal recommendation method based on federated learning according to claim 2, characterized in that, The k-score filtering method uses a value of 5 for k. The generated k-score interaction graph consists of userID, itemID, rating, timestamp, and x_label information. Here, userID represents the user ID, itemID represents the product ID, rating represents the user's rating of the product, timestamp represents the timestamp, and x_label represents the corresponding label.

4. The multimodal recommendation method based on federated learning according to claim 3, characterized in that, The image and text features of the product are mapped one-to-one, and the generated dataset includes: itemID, asin, title, price, imUrl, related, brand, categories, salesRamk, and description; In this context, asin represents the product code, title represents the product name, price represents the product price, imUrl represents the product image link, related represents related products, salesRank represents discount information, brand represents the product brand, and categories represents the category to which the product belongs.

5. The multimodal recommendation method based on federated learning according to claim 1, characterized in that, In step 4, the model is aggregated, specifically by weighting the model parameters of each client to obtain the aggregated model.

6. A multimodal recommendation method system based on federated learning, characterized in that, The system has a program module corresponding to the steps of any one of claims 1 to 5 above, and executes the steps in the above-described federated learning-based multimodal recommendation method when running.

Images