Multimodal recommendation method and system based on intra-modal and inter-modal contrastive learning
Through the comparison learning method between modes and modes, the problems of noise interference and semantic differences in multimodal recommendation systems are solved, more accurate personalized recommendations are achieved, and recommendation performance is improved.
Patent Information
- Application Number
- CN202411240177.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-09-05
AI Technical Summary
Multimodal recommendation systems face challenges in noise interference, semantic differences and feature fusion when introducing multimodal information, resulting in poor recommendation performance.
Using a method based on in-modal and inter-modal comparison learning, high-order features are captured through graph convolution networks, in-modal feature enhancement and inter-modal feature alignment are performed, and user recommendations are generated based on pre-trained multi-modal recommendation network models. List.
It improves the prediction accuracy and personalized recommendation effect of the recommendation system, reduces the cost of model training, significantly improves performance, and is suitable for various multimodal recommendation scenarios.
Smart Images
Figure CN118760802B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing and recommendation systems, and specifically relates to a multimodal recommendation method and system based on intra-modal and inter-modal comparative learning. Background Art
[0002] Early recommendation methods generally used single information, such as user-history interaction behaviors. These methods ignored the multimodal information of items and could not fully capture the user's preference clues. With the development of deep learning technology, recommendation systems can use neural networks to learn the deep features and complex nonlinear relationships of multimodal data, giving birth to multimodal recommendation systems. Multimodal recommendation systems combine user behavior data and multimodal information of items. By learning the correlation between multiple data sources, they can more comprehensively understand the user's interests and preferences, thereby providing more accurate personalized recommendation results. The introduction of multimodal information enriches the feature representation of items, alleviates the impact of data sparsity problems to a certain extent, and can effectively improve the recommendation performance of recommendation models. However, it also brings corresponding challenges. First, there is a large amount of noise information in multimodal data, which will interfere with the improvement of recommendation performance; second, due to the differences in semantic information between different modalities, it is necessary to design a reasonable feature alignment method to enhance the semantic consistency between different modalities; finally, it is necessary to design a reasonable multimodal feature fusion mechanism to ensure that each modality can play a positive role in the recommendation process. How to solve the above problems has become a key technical problem that needs to be solved in multimodal recommendation technology. Summary of the invention
[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multimodal recommendation method and system based on intra-modal and inter-modal comparative learning are provided. The present invention aims to utilize multimodal information to improve the prediction accuracy of the recommendation system, and to solve the corresponding problems of introducing multimodal information based on intra-modal and inter-modal comparative learning tasks, so as to achieve more accurate personalized recommendations for users, and can be widely used in various multimodal recommendation scenarios.
[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0005] A multimodal recommendation method based on intra-modal and inter-modal contrastive learning includes using a pre-trained multimodal recommendation network model to obtain an output user's recommendation list using an image, a description text and an ID. The multimodal recommendation network model generates a user's recommendation list including the following steps:
[0006] S1, obtains the input image, description text and ID, and extracts image features and text features from the image and description text;
[0007] S2, embed the ID to obtain the ID embedding feature, and use the ID embedding feature and image feature to construct a user-item graph in the visual mode , embedding ID features and text features to construct a user-item graph in text mode ; For user-item graphs in visual mode and the user-item graph in text mode , respectively use graph convolutional networks to capture high-order features to obtain ID feature vectors , image feature vector and text feature vector ;
[0008] S3, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector ;
[0009] S4, the user feature vector With the item feature vector The inner product calculation is performed to obtain the user's predicted score for the project, and the predicted scores are sorted and the set of the top K projects is selected as the user's recommendation list output.
[0010] Optionally, extracting image features and text features from the image and description text respectively in step S1 includes inputting the image into a pre-trained ResNet50 network model to extract image features, and inputting the description text into a pre-trained BERT model to extract text features.
[0011] Optionally, when a graph convolutional network is used to capture high-order features in step S2, the graph convolutional network includes L graph convolutional network layers and a summation module, wherein the L graph convolutional network layers are cascaded in sequence, and the output ends of the L graph convolutional network layers are all connected to the input ends of the summation module, and any The function expression for calculating the feature vector of the graph convolutional network layer is:
[0012] ,
[0013] In the above formula, and Mode No. Layer and The feature vector output by the graph convolutional network layer, and Respectively represent user node u and item node i in the user-item graph One-hop neighbors in ; modality , Respectively represent visual, text and ID modes; the summation module counts the modes High-level features The function expression is:
[0014]
[0015] In the above formula, L is the number of graph convolutional network layers, and the modality High-level features Represents the final ID feature vector , image feature vector Or a text feature vector , and there are:
[0016] ,
[0017] ,
[0018] ,
[0019] In the above formula, for The final embedding of user IDs, for The final embedding of items, In visual mode The final embedding of the user, In visual mode The final embedding of items, In text mode The final embedding of the user, In text mode The final embedding of items, and are the number of users and the number of items respectively.
[0020] Optionally, in step S3, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector include:
[0021] S3.1, according to the image feature vector and text feature vector Construct the final embedding vector shown below :
[0022] ,
[0023] In the above formula, is the final embedding vector, For the visual mode The final embedding of the user, For text mode The final embedding of the user, For the visual mode The final embedding of items, For text mode The final embedding of the items;
[0024] S3.2, generate the user feature vector according to the following formula With the item feature vector :
[0025] , .
[0026] Optionally, in step S4, the user feature vector With the item feature vector The function expression of the user's predicted score for the project obtained by inner product calculation is:
[0027] ,
[0028] In the above formula, Rating the user's predictions for the item, for The final embedding of user IDs, for The final embedding of the item.
[0029] Optionally, before the image, description text and ID are obtained by using a pre-trained multimodal recommendation network model to output the user's recommendation list, the method further includes training and optimizing the network parameters of the multimodal recommendation network model by using a back propagation algorithm and a mirror gradient optimization algorithm, and the loss function used when training and optimizing the network parameters of the multimodal recommendation network model is:
[0030] ,
[0031] In the above formula, is the loss function, , and are weight parameters, is the loss of feature alignment between modalities, is the loss for intra-modal feature enhancement, is the regularization loss, is the Bayesian personalized ranking loss.
[0032] Optionally, the calculation function expression of the loss of the inter-modal feature alignment is:
[0033] ,
[0034] In the above formula, N is the total number of samples, B is the batch size when calculating the loss, is the similarity calculation function, is the image feature of the i-th sample, is the text feature of the i-th sample, is the image feature of the jth sample, is the text feature of the jth sample;
[0035] The calculation function expression of the loss of the intra-modal feature enhancement is:
[0036] ,
[0037] ,
[0038] ,
[0039] ,
[0040] ,
[0041] In the above formula, , , and is the intermediate variable, are the items in item set I, is a user in the user set U, is the hyperparameter used to balance the loss function, is the cosine similarity function, and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features;
[0042] The calculation function expression of the regularization loss is:
[0043] ,
[0044] In the above formula, is the image feature vector, is the text feature vector;
[0045] The calculation function expression of the Bayesian personalized ranking loss is:
[0046] ,
[0047] In the above formula, is the user u, item p and item in the training set D The triplet formed by p and n respectively represents the items that have interacted with user u and the items that have not interacted with user u. is the sigmoid activation function, and are user u for item p and item The prediction score of .
[0048] In addition, the present invention also provides a multimodal recommendation system based on intra-modal and inter-modal comparative learning, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal recommendation method based on intra-modal and inter-modal comparative learning.
[0049] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal recommendation method based on intra-modal and inter-modal comparative learning through a processor.
[0050] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal recommendation method based on intra-modality and inter-modality contrast learning through a processor.
[0051] Compared with the prior art, the present invention mainly has the following advantages: the multimodal recommendation method based on intra-modal and inter-modal comparative learning of the present invention includes obtaining the output user's recommendation list by using a pre-trained multimodal recommendation network model for images, description texts and IDs: obtaining the input image, description text and ID, extracting image features and text features from the image and description text; constructing a user-item graph under visual modality and text modality, and then using a graph convolutional network to capture high-order features to obtain image and text feature vectors respectively; performing intra-modal feature enhancement and inter-modal feature alignment to obtain user and item feature vectors and calculating the user's predicted score for the item to generate a recommendation list. The present invention solves the corresponding problems of introducing multimodal information based on intra-modal and inter-modal comparative learning tasks, uses multimodal information to improve the prediction accuracy of the recommendation system, can achieve more accurate personalized recommendations for users, and can effectively align and fuse features. Not only can the model training cost be reduced, but the performance of the model can also be significantly improved, and it can be widely used in various multimodal recommendation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.
[0053] Figure 2 Schematic diagram of the working principle of the multimodal recommendation network model in an embodiment of the present invention.
[0054] Figure 3 To obtain the final user feature vector in the embodiment of the present invention With the item feature vector Schematic diagram of the process.
[0055] Figure 4 The figure is a flow chart of optimizing a multimodal recommendation network model through training by a back propagation algorithm and a mirror gradient optimization algorithm in an embodiment of the present invention.
[0056] Figure 5 is the different loss weights within the modality on the baby data set in the embodiment of the present invention The performance of the model below.
[0057] Figure 6 is the different loss weights within the modality on the video dataset in the embodiment of the present invention The performance of the model below.
[0058] Figure 7 is the different loss weights between modalities on the baby data set in the embodiment of the present invention The performance of the model below.
[0059] Figure 8 is the different loss weights between modalities on the video dataset in the embodiment of the present invention The performance of the model below. DETAILED DESCRIPTION
[0060] Multimodal recommendation systems improve recommendation accuracy by introducing multimodal information as auxiliary information, but the introduction of multimodal information also brings corresponding challenges to the recommendation system. First, there will be a lot of noise information in the multimodal data, and this noise information will affect the overall performance of the recommendation system, resulting in poor final recommendation results; secondly, because the multimodal recommendation system uses multiple modal information to simulate user preferences for items, and the representation of the same item in different modalities may be different, resulting in inaccurate final prediction results. This embodiment proposes an intra-modal and inter-modal comparative learning method, uses an intra-modal comparative learning task to enhance the features of a specific modality, and then uses an inter-modal comparative learning task to align the feature representations of different modalities to enhance semantic consistency, ultimately achieving the purpose of improving the recommendation accuracy of the multimodal recommendation system. Specifically, Figure 1 and Figure 2 As shown, the multimodal recommendation method based on intra-modal and inter-modal comparative learning in this embodiment includes using a pre-trained multimodal recommendation network model to obtain an output user's recommendation list using an image, a description text, and an ID. The multimodal recommendation network model generates a user's recommendation list including the following steps:
[0061] S1, obtains the input image, description text and ID, and extracts image features and text features from the image and description text;
[0062] S2, embed the ID to obtain the ID embedding feature, and use the ID embedding feature and image feature to construct a user-item graph in the visual mode , embedding ID features and text features to construct a user-item graph in text mode ; For user-item graphs in visual mode and the user-item graph in text mode , respectively use graph convolutional networks to capture high-order features to obtain ID feature vectors , image feature vector and text feature vector ;
[0063] S3, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector ;
[0064] S4, the user feature vector With the item feature vector The inner product calculation is performed to obtain the user's predicted score for the project, and the predicted scores are sorted and the set of the top K projects is selected as the user's recommendation list output.
[0065] like Figure 2 As shown, in step S1 of this embodiment, extracting image features and text features from the image and description text respectively includes inputting the image into a pre-trained ResNet50 network model to extract image features, and inputting the description text into a pre-trained BERT model to extract text features.
[0066] In step S2 of this embodiment, the ID is embedded to obtain the ID embedding feature, and the ID embedding feature and the image feature are used to construct a user-item graph in the visual mode. , embedding ID features and text features to construct a user-item graph in text mode When constructing the user-item graph, users and items are represented as nodes respectively. The connection relationship between users and items is constructed based on the user's behavioral association with the item. The visual features and text features are associated with the corresponding user and item nodes respectively to form a multimodal user-item graph (UI graph), which is recorded as , , representing visual, textual and ID modes respectively, including the user-item graph in visual mode And the user-item graph in text mode In this embodiment, the ID modality only participates in the final prediction and scoring stage and does not play a role in the feature enhancement and feature alignment stages.
[0067] When a graph convolutional network is used to capture high-order features in step S2 of this embodiment, the graph convolutional network includes L graph convolutional network layers and a summation module, wherein the L graph convolutional network layers are sequentially cascaded, and the output ends of the L graph convolutional network layers are all connected to the input ends of the summation module, and any The function expression for calculating the feature vector of the graph convolutional network layer is:
[0068] ,
[0069] In the above formula, and Mode No. Layer and The feature vector output by the graph convolutional network layer, and Respectively represent user node u and item node i in the user-item graph One-hop neighbors in ; modality , Respectively represent visual, text and ID modes; the summation module counts the modes High-level features The function expression is:
[0070]
[0071] In the above formula, L is the number of graph convolutional network layers, and the modality High-level features Represents the final ID feature vector , image feature vector Or a text feature vector , and there are:
[0072] ,
[0073] ,
[0074] ,
[0075] In the above formula, for The final embedding of user IDs, for The final embedding of items, In visual mode The final embedding of the user, In visual mode The final embedding of items, In text mode The final embedding of the user, In text mode The final embedding of items, and are the number of users and the number of items respectively.
[0076] In this embodiment, the final embedding vector is obtained based on the feature vector after the intra-modality feature enhancement task and the inter-modality feature alignment task. In this embodiment, the final embedding vector is represented by a matrix, which is formed by concatenating the user feature vector and the item feature vector in rows. Column and back The columns represent user feature vectors and item feature vectors respectively. and are the number of users and items in the data set respectively. Figure 3 As shown, in step S3 of this embodiment, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector include:
[0077] S3.1, according to the image feature vector and text feature vector Construct the final embedding vector shown below :
[0078] ,
[0079] In the above formula, is the final embedding vector, For the visual mode The final embedding of the user, For text mode The final embedding of the user, For the visual mode The final embedding of items, For text mode The final embedding of the items;
[0080] S3.2, generate the user feature vector according to the following formula With the item feature vector :
[0081] , .
[0082] In step S4 of this embodiment, the user feature vector With the item feature vector The function expression of the user's predicted score for the project obtained by inner product calculation is:
[0083] ,
[0084] In the above formula, Rating the user's predictions for the item, for The final embedding of user IDs, for The final embedding of the item.
[0085] In this embodiment, before the image, description text and ID are obtained by using the pre-trained multimodal recommendation network model to obtain the output user's recommendation list, the network parameters of the multimodal recommendation network model are trained and optimized by the back propagation algorithm and the mirror gradient optimization algorithm, and the loss function used when training and optimizing the network parameters of the multimodal recommendation network model is:
[0086] ,
[0087] In the above formula, is the loss function, , and are weight parameters, is the loss of feature alignment between modalities, is the loss for intra-modal feature enhancement, is the regularization loss, is the Bayesian personalized ranking loss.
[0088] In this embodiment, the calculation function expression of the loss of inter-modal feature alignment is:
[0089] ,
[0090] In the above formula, N is the total number of samples, B is the batch size when calculating the loss, is the similarity calculation function, is the image feature of the i-th sample, is the text feature of the i-th sample, is the image feature of the jth sample, is the text feature of the jth sample. The function expression of the similarity calculation function in this embodiment is:
[0091] = ,
[0092] In the above formula, is a hyperparameter used to balance the loss function.
[0093] In this embodiment, the calculation function expression of the loss of intra-modal feature enhancement is:
[0094] ,
[0095] ,
[0096] ,
[0097] ,
[0098] ,
[0099] In the above formula, , , and is the intermediate variable, are the items in item set I, is a user in the user set U, is the hyperparameter used to balance the loss function, is the cosine similarity function, and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features. Figure 3 As shown, in the image feature vector and text feature vector Different random noises are added to construct the noise view, which can be expressed as follows:
[0100] ,
[0101] In the above formula, and is the image feature vector Add different random noise and The noise view obtained after adding different random noises (image features obtained after adding different random noises), and is the text feature vector Add different random noise and The noise view obtained after adding different random noises (text features obtained after adding different random noises) can be expressed as: , and there is , .Will and , and As a comparison view, the intra-modal contrast learning can be performed to calculate the loss of intra-modal feature enhancement The intermediate variables , , and .
[0102] In this embodiment, the calculation function expression of the regularization loss is:
[0103] ,
[0104] In the above formula, is the image feature vector, is the text feature vector;
[0105] In this embodiment, the calculation function expression of the Bayesian personalized ranking loss is:
[0106] ,
[0107] In the above formula, is the user u, item p and item in the training set D The triplet formed by p and n respectively represents the items that have interacted with user u and the items that have not interacted with user u. is the sigmoid activation function, and are user u for item p and item The prediction score of .
[0108] This embodiment uses the back propagation algorithm and the mirror gradient optimization algorithm to train and optimize the network parameters of the multimodal recommendation network model. When , an epoch consists of a complete forward propagation and back propagation, and the model training is completed when the predetermined number of iterations is reached or when the prediction results no longer change significantly. Figure 4 As shown, in this embodiment, the steps of training and optimizing the network parameter θ of the multimodal recommendation network model by using the back propagation algorithm and the mirror gradient optimization algorithm include:
[0109] Step 1, initialize the iteration variable t to 1;
[0110] Step 2: determine whether the iteration variable t is less than or equal to the preset number of iterations T. If so, jump to step 3; otherwise, output the network parameters of the multimodal recommendation network model. , end and exit;
[0111] Step 3, determine whether the iteration variable t mod β is equal to 0, if true, jump to step 4; otherwise jump to step 5; where mod is the modular operation, β is a constant parameter;
[0112] Step 4: Update the network parameters of the multimodal recommendation network model according to the function expression shown in the following formula: :
[0113] ,
[0114] ,
[0115] In the above formula, is the intermediate variable of the tth iteration, and are the network parameters for the tth and t-1th iterations , and Hyperparameters, is the learning rate, is the network parameter The gradient of and is the loss function of the tth and t-1th iterations, jump to step 6;
[0116] Step 5: Update the network parameters of the multimodal recommendation network model according to the function expression shown in the following formula: :
[0117] ,
[0118] In the above formula, and are the network parameters for the tth and t-1th iterations , is the learning rate, is the network parameter The gradient of is the loss function of the t-1th iteration, jump to step 6;
[0119] Step 6, increase the iteration variable t by 1 and jump to step 2.
[0120] In order to verify the multimodal recommendation method based on intra-modal and inter-modal contrast learning in this embodiment, the dataset used in the experiment in this embodiment is the Amazon product review dataset, specifically including the baby dataset and the video dataset. The evaluation indicators used are the recall rate Recall@K and the normalized discounted cumulative gain NDCG@K, where K represents the number of items recommended to the user, for example, K=20 in this embodiment. Figure 5 are the different loss weights on the baby data set in this embodiment The model performance under Figure 6 are the different loss weights on the video dataset in this embodiment The model performance under Figure 7 are the different loss weights on the baby data set in this embodiment The model performance under Figure 8 are the different loss weights on the video dataset in this embodiment The model performance under , where Recall@20 is the recall rate when the number of items recommended to the user K is 20, and NDCG@20 is the normalized discounted cumulative gain when the number of items recommended to the user K is 20, see Figures 5 to 8 It can be seen that under different loss weights and The performance of the recommendation model will fluctuate, and the optimal loss weight value is usually inconsistent for different data sets.
[0121] In addition, in this embodiment, LightGCN, DualGNN, SLMRec and BM3 are used as comparisons of the method of this embodiment (abbreviated as IGCL), and the indicators used include: R@10, R@20, N@10 and N@20. LightGCN is a general single-modal recommendation method. For details, see the literature: He X, Deng K, Wang X, et al. Lightgcn: Simplifying and powering graph convolution network for recommendation[C] / / Proceedings of the 43rdInternational ACM SIGIR conference on research and development in InformationRetrieval. 2020: 639-648. DualGNN, SLMRec and BM3 are multimodal recommendation methods. For details of DualGNN, please refer to the literature: Wang Q, Wei Y, Yin J, et al. Dualgnn: Dual graph neural network formultimedia recommendation[J]. IEEE Transactions on Multimedia, 2021, 25:1074-1084., for details of SLMRec, please refer to the literature: Tao Z, Liu X, Xia Y, et al. Self-supervisedlearning for multimedia recommendation[J]. IEEE Transactions on Multimedia,2022, 25: 5107-5116., for details of BM3, please refer to the literature: Zhou X, Zhou H, Liu Y, et al. Bootstraplatent representations for multi-modal recommendation[C] / / Proceedings of theACM Web Conference 2023. 2023: 845-854., R@10 and R@20 represent the recall rate Recall@10 and Recall@20, N@10 and N@20 represent the normalized discounted cumulative gain NDCG@10 and NDCG@20 respectively. The final performance comparisons on the baby dataset and the video dataset are shown in Tables 1 and 2.
[0122] Table 1: Performance comparison on the baby dataset
[0123]
[0124] Table 2: Performance comparison on video dataset
[0125]
[0126] As shown in Table 2, the multimodal recommendation method based on intra-modal and inter-modal comparative learning in this embodiment is superior to general recommendation methods and recent multimodal recommendation methods in terms of recommendation performance.
[0127] In summary, the multimodal recommendation method based on intra-modal and inter-modal comparative learning in this embodiment first preprocesses the data set, and after data cleaning, extracts image features and text features through pre-trained ResNet50 and BERT models. With users and items as nodes, multimodal information is used to construct the user-item graph (UI graph) of the corresponding modality, and then the high-order features of each modality are captured through the graph convolutional network. Then, the high-order image feature vectors are With text feature vector Fusion is performed to obtain the fused feature vector . Then, an intra-modal contrastive learning task is introduced. By adding different random noises to the high-order features of the specific modality to construct a contrast view, contrastive learning is used to narrow the gap between the two noise contrast views to enhance the feature representation of the specific modality. In addition, an intra-modal contrastive learning task is introduced to enhance the semantic consistency of different modalities by bringing the feature representations of the same project under different modalities closer. Finally, the Bayesian personalized ranking loss function is used as the basic optimization function, and the final total loss is calculated by combining the intra-modal contrast loss, inter-modal contrast loss and regularization loss. Backpropagation is performed to update the parameters, and a mirror gradient optimization algorithm is used during the process to enhance the robustness of the model. The method of this embodiment can use multimodal information as auxiliary information to enrich the feature representation of users and projects, and enhance the multimodal feature representation through inter-modal contrastive learning tasks, and improve the semantic consistency between different modalities through intra-modal contrastive learning tasks, thereby improving the accuracy of personalized recommendations. It can be widely used in various multimodal recommendation scenarios.
[0128] In addition, this embodiment also provides a multimodal recommendation system based on intra-modal and inter-modal comparative learning, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the multimodal recommendation method based on intra-modal and inter-modal comparative learning.
[0129] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal recommendation method based on intra-modal and inter-modal comparative learning through a processor.
[0130] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal recommendation method based on intra-modality and inter-modality comparative learning through a processor.
[0131] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0132] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.
Claims
1. A multimodal recommendation method based on intra-modal and inter-modal contrastive learning, characterized in that: The method comprises using a pre-trained multimodal recommendation network model to obtain an output user's recommendation list by using an image, a description text and an ID. The multimodal recommendation network model generates a user's recommendation list including the following steps: S1, obtains the input image, description text and ID, and extracts image features and text features from the image and description text; S2, embed the ID to obtain the ID embedding feature, and use the ID embedding feature and image feature to construct a user-item graph in the visual mode , embedding ID features and text features to construct a user-item graph in text mode ; For user-item graphs in visual mode and the user-item graph in text mode , respectively use graph convolutional networks to capture high-order features to obtain ID feature vectors , image feature vector and text feature vector ; S3, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector ; S4, the user feature vector With the item feature vector Perform inner product calculation to obtain the user's predicted score for the project, sort the predicted scores, and select the set of top K projects as the user's recommendation list output; When a graph convolutional network is used to capture high-order features in step S2, the graph convolutional network includes L graph convolutional network layers and a summation module, wherein the L graph convolutional network layers are cascaded in sequence, and the output ends of the L graph convolutional network layers are connected to the input ends of the summation module, and any The function expression for calculating the feature vector of the graph convolutional network layer is: ， In the above formula, and Mode No. Layer and The feature vector output by the graph convolutional network layer, and Respectively represent user node u and item node i in the user-item graph One-hop neighbors in ; modality , Respectively represent visual, text and ID modes; the summation module counts the modes High-level features The function expression is: In the above formula, L is the number of graph convolutional network layers, and the modality High-level features Represents the final ID feature vector , image feature vector Or a text feature vector , and there are: ， ， ， In the above formula, for The final embedding of user IDs, for The final embedding of items, In visual mode The final embedding of the user, In visual mode The final embedding of items, In text mode The final embedding of the user, In text mode The final embedding of items, and are the number of users and the number of items respectively; In step S3, the image feature vector and text feature vector Combine the ID feature vectors Perform intra-modal feature enhancement and inter-modal feature alignment to obtain user feature vector With the item feature vector include: S3.1, according to the image feature vector and text feature vector Construct the final embedding vector shown below : ， In the above formula, is the final embedding vector, For the visual mode The final embedding of the user, For text mode The final embedding of the user, For the visual mode The final embedding of items, For text mode The final embedding of the items; S3.2, generate the user feature vector according to the following formula With the item feature vector : ， ； In step S4, the user feature vector With the item feature vector The function expression of the user's predicted score for the project obtained by inner product calculation is: ， In the above formula, Rating the user's predictions for the item, for The final embedding of user IDs, for The final embedding of the items; Before the image, description text and ID are obtained by using the pre-trained multimodal recommendation network model to output the user's recommendation list, the method further includes training and optimizing the network parameters of the multimodal recommendation network model by using the back propagation algorithm and the mirror gradient optimization algorithm, and the loss function used when training and optimizing the network parameters of the multimodal recommendation network model is: ， In the above formula, is the loss function, , and are weight parameters, is the loss of feature alignment between modalities, is the loss for intra-modal feature enhancement, is the regularization loss, is the Bayesian personalized ranking loss; The calculation function expression of the loss of the inter-modal feature alignment is: ， In the above formula, N is the total number of samples, B is the batch size when calculating the loss, is the similarity calculation function, is the image feature of the i-th sample, is the text feature of the i-th sample, is the image feature of the jth sample, is the text feature of the jth sample; The calculation function expression of the loss of the intra-modal feature enhancement is: ， ， ， ， ， In the above formula, , , and is the intermediate variable, are the items in item set I, is a user in the user set U, is the hyperparameter used to balance the loss function, is the cosine similarity function, and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium User The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the image feature vector Medium Items The image features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium User The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features. and is the text feature vector Medium Items The text features are obtained by adding different random noises to the image features; The calculation function expression of the regularization loss is: ， In the above formula, is the image feature vector, is the text feature vector; The calculation function expression of the Bayesian personalized ranking loss is: ， In the above formula, is the user u, item p and item in the training set D The triplet formed by p and n respectively represents the items that have interacted with user u and the items that have not interacted with user u. is the sigmoid activation function, and are user u for item p and item The prediction score of .
2. The multimodal recommendation method based on intra-modal and inter-modal contrastive learning according to claim 1, characterized in that: In step S1, extracting image features and text features from the image and description text respectively includes inputting the image into a pre-trained ResNet50 network model to extract image features, and inputting the description text into a pre-trained BERT model to extract text features.
3. A multimodal recommendation system based on intra-modal and inter-modal contrastive learning, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multimodal recommendation method based on intra-modality and inter-modality comparative learning as described in claim 1 or 2.
4. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the multimodal recommendation method based on intra-modality and inter-modality comparative learning as claimed in claim 1 or 2 through a processor.
5. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the multimodal recommendation method based on intra-modality and inter-modality comparative learning as claimed in claim 1 or 2 through a processor.