Multimodal recommendation method, system and device based on counterfactual contrast pre-training

Abstract

The present invention relates to the field of data mining technology, and provides a multimodal recommendation method, system and device based on counterfactual comparison pre-training. User data, project data and interaction data between user projects are obtained to obtain a user-project interaction graph, and a counterfactual interaction graph is constructed based on the user-project interaction graph; purified project modal features are obtained based on the original multimodal features of the project; user modal features are obtained based on the counterfactual interaction graph; pre-trained multimodal representations are obtained based on collaborative features and modal features; based on a collaborative filtering model, the pre-trained multimodal representation is fine-tuned on the user-project interaction graph to obtain a final user representation and a final project representation, and the user's preference score for each project is calculated based on the final representation, and a recommendation list is obtained by sorting according to the preference scores. The present invention effectively captures the user's potential interest preferences by making full use of the multimodal content of non-interacted projects, and can significantly improve the accuracy of recommendations.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining, and provides a multimodal recommendation method, system and device based on counterfactual comparison pre-training. Background Art

[0002] Recommender systems, as a core technology for internet applications, are widely used in fields such as e-commerce, streaming platforms, and social networks. By recommending information or items that match users' interests, they significantly improve user experience and platform operational efficiency. Collaborative filtering (CF), a classic method in this field, has achieved remarkable results by analyzing historical user-item interaction data to uncover users' latent preferences. However, traditional collaborative filtering techniques still exhibit significant limitations when faced with data sparsity and cold start problems. In particular, when user or item interaction records are insufficient, this method struggles to effectively capture latent interests, thus affecting the accuracy and diversity of recommendations.

[0003] With the rapid growth of multimedia information on the Internet, multimodal data such as text, images, audio, and video have become an important bridge for the interaction between users and projects. This data not only enriches users' understanding of projects, but also provides new solutions to alleviate the long-standing problems of recommendation systems. Multimodal recommendation systems have thus emerged. Their core idea is to fully utilize the multimodal characteristics of projects to capture more fine-grained information about user preferences, thereby improving the accuracy and diversity of recommendations based on traditional collaborative filtering methods and providing users with a better experience. In recent years, with the rise of graph neural networks (GNNs), researchers have leveraged their ability to model high-order node relationships to construct user-project interaction graphs to more deeply characterize user preferences. Multimodal recommendation methods based on user-project graphs can capture complex, high-order interactive relationships between users and projects, and have demonstrated superior performance in both theoretical research and practical applications.

[0004] Although existing multimodal recommendation technologies have made some progress, they still face two core challenges. First, current multimodal recommendation methods usually rely on the modal information of items that users have interacted with to model preferences, but this approach ignores potential items of interest that users may not have observed during actual interactions. Due to the exposure of items or the attention limitations of users, some potentially interesting items are not selected by users. Relying only on the information of interacted items may lead to a one-sided understanding of user interests. Second, the modal features extracted by the pre-trained model are general representations and may not be fully adapted to the recommendation task. For example, item images may contain background, decoration, and other information that is irrelevant to user preferences. These redundant features not only cannot provide help for the recommendation task, but may even introduce noise and reduce the accuracy of the recommendation. Summary of the Invention

[0005] The present invention aims to address at least one of the technical problems existing in the related art. To this end, it provides a multimodal recommendation method, system, and device based on counterfactual comparative pre-training. These methods enhance the learning of users' latent preferences, deeply explore the fine-grained interactions between collaborative features and modal features, and effectively eliminate redundant and noisy information in item modal features. This method can be integrated into traditional collaborative filtering recommendation models, significantly enhancing their recommendation performance.

[0006] The present invention provides a multimodal recommendation method based on counterfactual contrast pre-training, comprising:

[0007] S1: Obtain user data, item data, and interaction data between user items, obtain a user-item interaction graph based on the user data, item data, and interaction data between user items, and construct a counterfactual interaction graph based on the user-item interaction graph;

[0008] S2: Randomly initialize to obtain user collaborative features and item collaborative features, and calculate the item modal features based on the item original modal features;

[0009] S3: Inputting the project collaborative features and project modal features into a bilinear interaction purification module to obtain purified project modal features; obtaining user modal features based on the purified project modal features and the counterfactual interaction graph;

[0010] S4: calculating a pre-trained modal representation based on the modal features, calculating a pre-trained collaborative representation based on the collaborative features, performing modal representation fusion based on the pre-trained modal representation, calculating a final user multimodal representation and a final item multimodal representation, calculating a user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculating an item loss function using the pre-trained item collaborative representation and the final item multimodal representation;

[0011] S5: fine-tuning the pre-trained user multimodal representation and item multimodal representation on the user-item interaction graph according to the collaborative filtering model to obtain a final user representation and a final item representation;

[0012] S6: Calculate the user's preference score for each item based on the final user representation and the final item representation, and sort the items based on the user's preference score to obtain a recommendation list.

[0013] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, the user-item interaction graph is represented as a binary interaction matrix. If there is an edge between a user node and an item node in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​1; if there is no edge between the user and the item in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​0.

[0014] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S1 includes:

[0015] S11: Calculate multimodal similarity:

[0016]

[0017] in, For users Interactive Projects and not with the user Projects that interact with each other In modal Multimodal similarity on , are the raw multimodal features of the items that the user interacts with, are the original multimodal features of items that have not been interacted with by the user, represents the transpose of the matrix, Indicates vector modulus;

[0018] S12: The sample with the highest similarity to the user interaction item Uninteracted items are used to establish new edges with user nodes and initialize the weights of the new edges:

[0019]

[0020] in, For users Items that have not been interacted with The new edge weights between To obtain the highest A function of values, is the value quantity parameter;

[0021] S13: Constructing counterfactual interaction graphs:

[0022]

[0023] in, is the counterfactual interaction graph, is the edge set, is the edge set of the user-item interaction graph, is the set of sampled counterfactual interaction edges.

[0024] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S2 includes:

[0025] S21: Randomly initialize to obtain user collaborative features and project collaboration features ;

[0026] S22: Calculate the project modal features based on the original modal features of the project :

[0027]

[0028] in, is the trainable projection matrix, is the bias vector, The original modal characteristics of the project.

[0029] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S3 includes:

[0030] S31: Input the project collaborative features and project modal features into the bilinear interaction purification module to obtain the purified project modal features:

[0031]

[0032] in, is the purified item modal feature, is a learnable bilinear transformation weight matrix;

[0033] S32: Obtaining user modal features based on the purified item modal features and the counterfactual interaction graph:

[0034]

[0035] in, is the user modal feature, For users In the counterfactual interaction diagram The set of neighbors in To find the number of elements in a set.

[0036] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S4 includes:

[0037] S41: Computing pre-trained co-representations and pre-trained modal representations:

[0038]

[0039]

[0040] in, For the Co-representation of layers, is the layer number, , is the total number of layers, Counterfactual interaction diagram The adjacency matrix of for The degree matrix of For the Co-representation of layers, For the The modal representation of the layer, For the Modal representation of layers;

[0041] in, hour, and The calculation formula is as follows:

[0042]

[0043]

[0044] in, is the collaborative representation of layer 0, For the user collection User collaborative features of each user, , For user collection, For project collection Project collaboration characteristics of each project, , For the project collection, is the modal representation of layer 0, For the user collection User modal features of each user, Project Collection The modal characteristics of the items after purification;

[0045] S42: Calculate the final modal representation of the pre-training phase according to the modal representation, and calculate the final collaborative representation of the pre-training phase according to the collaborative representation:

[0046]

[0047]

[0048] in, is the final modality representation in the pre-training stage, The final collaborative representation for the pre-training stage;

[0049] S43: Perform modal representation fusion based on the pre-trained modal representation to calculate the final user multimodal representation and the final project multimodal representation:

[0050]

[0051]

[0052] in, Multimodal representation for end users, For visual mode, For text mode, is the user-learnable modality importance weight, For The corresponding user modality representation extracted, Final project modal characterization, is the modal importance weight that can be learned by the project, For The corresponding item modality representations extracted;

[0053] S44: Calculate the contrast loss of the final user collaborative representation and the final user multimodal representation, and calculate the contrast loss of the final project collaborative representation and the final project multimodal representation until the loss value is convergence:

[0054]

[0055]

[0056]

[0057] in, is the user loss function, To compare users, is the cosine similarity function, For The corresponding user collaborative representation extracted, is the temperature coefficient, For The corresponding contrast user collaborative representations extracted, is the item loss function, For comparison items, For The corresponding project co-representation extracted, For The corresponding contrast item-user co-representations are extracted.

[0058] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S5 includes:

[0059] S51: using the final modal representation of the pre-training stage to initialize the collaborative filtering model, and using a graph convolutional network to propagate information in the user-item interaction graph to obtain user representation and item representation;

[0060] S52: Fine-tuning the loss function through optimization , optimize the collaborative filtering model:

[0061]

[0062] in, is the training data, , are the items in the edge set of the user-item interaction graph, For items that are not in the edge set of the user-item interaction graph, is the sigmoid function, For users About the project The preference score of For users About the project The preference score of is the weight hyperparameter, are the trainable parameters of the model, is the quadratic norm;

[0063] S53: When The user representation output by the collaborative filtering model at convergence is the final user representation. The item representation output by the collaborative filtering model at convergence is the final item representation.

[0064] According to a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention, step S6 includes:

[0065] S61: Performing a dot product operation on the final user representation and the final item representation to calculate the user's preference score for each item;

[0066] S62: Sort all items according to the preference scores and generate a personalized recommendation list for the user.

[0067] The present invention also provides a multimodal recommendation system based on counterfactual comparison pre-training, comprising:

[0068] Data processing module: obtains user data, project data, and interaction data between user projects, obtains a user-project interaction graph based on the user data, project data, and interaction data between user projects, and constructs a counterfactual interaction graph based on the user-project interaction graph; randomly initializes to obtain user collaboration features and project collaboration features, and calculates project modal features based on the original modal features of the projects;

[0069] Bilinear interaction purification module: input the project collaborative features and project modal features into the bilinear interaction purification module to obtain purified project modal features; obtain user modal features based on the purified project modal features and the counterfactual interaction graph;

[0070] Multimodal representation pre-training module: calculates pre-trained modal representation based on the modal features, calculates pre-trained collaborative representation based on the collaborative features, performs modal representation fusion based on the pre-trained modal representation, calculates the final user multimodal representation and the final project multimodal representation, calculates the user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculates the project loss function using the pre-trained project collaborative representation and the final project multimodal representation;

[0071] Multimodal representation fine-tuning module: Based on the collaborative filtering model, the pre-trained user multimodal representation and item multimodal representation are fine-tuned on the user-item interaction graph to obtain the final user representation and the final item representation;

[0072] Recommendation module: Calculates the user's preference score for each item based on the final user representation and the final item representation, and sorts the items according to the user's preference score to obtain a recommendation list.

[0073] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a multimodal recommendation method based on counterfactual comparison pre-training as described above are implemented.

[0074] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0075] The present invention provides a multimodal recommendation method, system and device based on counterfactual contrast pre-training. By constructing a counterfactual interaction graph and introducing the multimodal content of non-interacted items, it is possible to explore the user's potential interests in a wider possible space and enhance the learning of the user's potential preferences. By designing a bilinear interaction purification module, the fine-grained interaction between collaborative features and modal features is deeply explored, and the redundant and noisy information in the modal features of the items is effectively eliminated. By adopting the self-supervised objective of contrastive learning in the pre-training stage and aligning and optimizing the modal features with the collaborative features, the adaptability and expressiveness of the modal features in the recommendation task can be improved. By fine-tuning the pre-trained modal features in combination with the existing collaborative filtering recommendation framework, the user's potential preferences can be accurately captured. The counterfactual interaction sampling method and the bilinear interaction purification module designed by the present invention jointly promote the effect of multimodal recommendation, can be integrated into the traditional collaborative filtering recommendation model, and can significantly enhance the recommendation performance of the traditional collaborative filtering model.

[0076] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 This is a flowchart of a multimodal recommendation method based on counterfactual contrast pre-training provided by the present invention.

[0079] Figure 2 This is a structural block diagram of a multimodal recommendation device based on counterfactual comparative pre-training provided by the present invention.

[0080] Figure 3 This is a comparison of the recall rate and normalized discounted cumulative gain results of a multimodal recommendation method based on counterfactual comparative pre-training provided by the present invention in an ablation experiment.

[0081] Figure 4 The present invention provides a multimodal recommendation method based on counterfactual comparison pre-training, which uses different important parameters in parameter analysis experiments to compare results.

[0082] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention.

[0083] Reference numerals:

[0084] 101. Data processing module; 102. Bilinear interactive purification module; 103. Multimodal representation pre-training module; 104. Multimodal representation fine-tuning module; 105. Recommendation module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. DETAILED DESCRIPTION

[0085] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0086] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0087] The following combination Figures 1 to 5 The present invention is described.

[0088] Example

[0089] like Figure 1 As shown, Figure 1 The present invention provides a flowchart of a multimodal recommendation method based on counterfactual contrast pre-training, which specifically includes the following steps:

[0090] S1: Obtain user data, item data, and interaction data between user items, obtain a user-item interaction graph based on the user data, item data, and interaction data between user items, and construct a counterfactual interaction graph based on the user-item interaction graph;

[0091] The user-item interaction graph is represented as a binary interaction matrix. If there is an edge between a user node and an item node in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​1; if there is no edge between the user and the item in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​0.

[0092] S2: Randomly initialize to obtain user collaborative features and item collaborative features, and calculate the item modal features based on the item original modal features;

[0093] S3: Inputting the project collaborative features and project modal features into a bilinear interaction purification module to obtain purified project modal features; obtaining user modal features based on the purified project modal features and the counterfactual interaction graph;

[0094] S4: calculating a pre-trained modal representation based on the modal features, calculating a pre-trained collaborative representation based on the collaborative features, performing modal representation fusion based on the pre-trained modal representation, calculating a final user multimodal representation and a final item multimodal representation, calculating a user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculating an item loss function using the pre-trained item collaborative representation and the final item multimodal representation;

[0095] S5: fine-tuning the pre-trained user multimodal representation and item multimodal representation on the user-item interaction graph according to the collaborative filtering model to obtain a final user representation and a final item representation;

[0096] S6: Calculate the user's preference score for each item based on the final user representation and the final item representation, and sort the items based on the user's preference score to obtain a recommendation list.

[0097] Specifically, step S1 includes:

[0098] S11: Calculate multimodal similarity:

[0099]

[0100] in, For users Interactive Projects and not with the user Projects that interact with each other In modal Multimodal similarity on , are the raw multimodal features of the items that the user interacts with, are the original multimodal features of items that have not been interacted with by the user, represents the transpose of the matrix, Indicates vector modulus;

[0101] S12: The sample with the highest similarity to the user interaction item Uninteracted items are used to establish new edges with user nodes and initialize the weights of the new edges:

[0102]

[0103] in, For users Items that have not been interacted with The new edge weights between To obtain the highest A function of values, is the value quantity parameter;

[0104] S13: Constructing counterfactual interaction graphs:

[0105]

[0106] in, is the counterfactual interaction graph, is the edge set, is the edge set of the user-item interaction graph, is the set of sampled counterfactual interaction edges.

[0107] Specifically, step S2 includes:

[0108] S21: Randomly initialize to obtain user collaborative features and project collaboration features ;

[0109] S22: Calculate the project modal features based on the original modal features of the project :

[0110]

[0111] in, is the trainable projection matrix, is the bias vector, The original modal characteristics of the project.

[0112] Specifically, step S3 includes:

[0113] S31: Input the project collaborative features and project modal features into the bilinear interaction purification module to obtain the purified project modal features:

[0114]

[0115] in, is the purified item modal feature, is a learnable bilinear transformation weight matrix;

[0116] S32: Obtaining user modal features based on the purified item modal features and the counterfactual interaction graph:

[0117]

[0118] in, is the user modal feature, For users In the counterfactual interaction diagram The set of neighbors in To find the number of elements in a set.

[0119] Specifically, step S4 includes:

[0120] S41: Computing pre-trained co-representations and pre-trained modal representations:

[0121]

[0122]

[0123] in, For the Co-representation of layers, is the layer number, , is the total number of layers, Counterfactual interaction diagram The adjacency matrix of for The degree matrix of For the Co-representation of layers, For the The modal representation of the layer, For the Modal representation of layers;

[0124] in, hour, and The calculation formula is as follows:

[0125]

[0126]

[0127] in, is the collaborative representation of layer 0, For the user collection User collaborative features of each user, , For user collection, For project collection Project collaboration characteristics of each project, , For the project collection, is the modal representation of layer 0, For the user collection User modal features of each user, Project Collection The modal characteristics of the items after purification;

[0128] S42: Calculate the final modal representation of the pre-training phase according to the modal representation, and calculate the final collaborative representation of the pre-training phase according to the collaborative representation:

[0129]

[0130]

[0131] in, is the final modality representation in the pre-training stage, The final collaborative representation for the pre-training stage;

[0132] S43: Perform modal representation fusion based on the pre-trained modal representation to calculate the final user multimodal representation and the final project multimodal representation:

[0133]

[0134]

[0135] in, Multimodal representation for end users, For visual mode, For text mode, is the user-learnable modality importance weight, For The corresponding user modality representation extracted, Final project modal characterization, is the modal importance weight that can be learned by the project, For The corresponding item modality representations extracted;

[0136] S44: Calculate the contrast loss of the final user collaborative representation and the final user multimodal representation, and calculate the contrast loss of the final project collaborative representation and the final project multimodal representation until the loss value is convergence:

[0137]

[0138]

[0139]

[0140] in, is the user loss function, To compare users, is the cosine similarity function, For The corresponding user collaborative representation extracted, is the temperature coefficient, For The corresponding contrast user collaborative representations extracted, is the item loss function, For comparison items, For The corresponding project co-representation extracted, For The corresponding contrast item-user co-representations are extracted.

[0141] Specifically, step S5 includes:

[0142] S51: using the final modal representation of the pre-training stage to initialize the collaborative filtering model, and using a graph convolutional network to propagate information in the user-item interaction graph to obtain user representation and item representation;

[0143] S52: Fine-tuning the loss function through optimization , optimize the collaborative filtering model:

[0144]

[0145] in, is the training data, , are the items in the edge set of the user-item interaction graph, For items that are not in the edge set of the user-item interaction graph, is the sigmoid function, For users About the project The preference score of For users About the project The preference score of is the weight hyperparameter, are the trainable parameters of the model, is the quadratic norm;

[0146] S53: When The user representation output by the collaborative filtering model at convergence is the final user representation. The item representation output by the collaborative filtering model at convergence is the final item representation.

[0147] Specifically, step S6 includes:

[0148] S61: Performing a dot product operation on the final user representation and the final item representation to calculate the user's preference score for each item;

[0149] S62: Sort all items according to the preference scores and generate a personalized recommendation list for the user.

[0150] This paper verifies the multimodal recommendation method based on counterfactual contrast pre-training:

[0151] This paper conducts experiments on three publicly available Amazon multimodal recommendation datasets: the Baby dataset, the Sports dataset, and the Clothing dataset. The experiments generate preference scores for each user's items, rank them from high to low, and output a set of uninteracted items as the recommended results. Each experiment compares 15 multimodal recommendation models. Experiment 1 groups the prediction results from the three datasets and evaluates them using two metrics: Recall@K (R@K) and Normalized Discounted Cumulative Gain (NDCG@K, N@K). R@10 represents the proportion of the top 10 recommended items that users actually interacted with, while R@20 represents the proportion of the top 20 recommended items that users actually interacted with. N@10 represents the result when 10 items are recommended to a user, and N@20 represents the result when 20 items are recommended to a user.

[0152] The results of Experiment 1 are shown in Tables 1, 2, and 3. Table 1 shows the comparison of the results of the present invention and other methods on the multimodal recommendation task on the Baby dataset, Table 2 shows the comparison of the results of the present invention and other methods on the multimodal recommendation task on the Sports dataset, and Table 3 shows the comparison of the results of the present invention and other methods on the multimodal recommendation task on the Clothing dataset. CoCoMRec is the model using the method proposed in this invention.

[0153] Table 1 Comparison of the results of the present invention and other methods on the multimodal recommendation task on the Baby dataset

[0154]

[0155] Table 2 Comparison of the results of the present invention and other methods on the multimodal recommendation task on the Sports dataset

[0156]

[0157] Table 3 Comparison of the results of the present invention and other methods on the multimodal recommendation task on the Clothing dataset

[0158]

[0159] Among them, BPR (Bayesian Personalized Ranking), LightGCN (Simplifying and powering graph convolution network), VBPR (Visual Bayesian Personalized Ranking), MMGCN (Multi-modal Graph Convolution Network), GRCN (Graph-Refined Convolutional Network), SLMRec (Self-Supervised Learning for Multimedia Recommendation), LATTICE (Latent structure mining method for multimodal recommendation), BM3 (Bootstrap Multi-Modal Model), MICRO (Latent structure mining with Contrastive modality fusion for multimedia), MCLN (Multimodal Counterfactual Learning Network), FREEDOM (Graph structures freezing) and denoising multimodalmodel for recommendation graph structure freezing and denoising multimodal recommendation model), MGCN (Multi-viewGraph Convolutional Network), PromptMM (Multi-Modal KnowledgeDistillation for Recommendation with Prompt-Tuning), LGMRec (Local and Global Graph Learning for Multimodal Recommendation),DRAGON (A framework that learns dual representations of both users and items via constructing homogeneous graphs for multimodal recommendation) is another ranking method.

[0160] The results of Experiment 1 show that the proposed method outperforms previous methods on all three datasets and achieves the best results in both evaluation indicators.

[0161] First, it can be seen that the multimodal recommendation models studied in this paper outperform conventional collaborative filtering models. This advantage is primarily due to the rich knowledge contained in item text and visual information, which enables the models to more comprehensively and accurately model user preferences. By learning modality-specific user preferences and effectively integrating modal features, these models can better understand user interests, significantly improve predictive capabilities, and provide accurate and personalized recommendations.

[0162] Furthermore, the proposed method significantly outperforms existing state-of-the-art multimodal recommendation models. Specifically, CoCoMRec improves R@10, R@20, and N@10, N@20 by an average of 13.35%, 6.96%, 20.86%, and 16.11%, respectively, on three datasets compared to the next best method.

[0163] Experiment 2 demonstrates the performance improvement of the proposed method on two traditional collaborative filtering models in multimodal recommendation tasks. Table 4 shows the performance analysis of these two different collaborative filtering recommendation models enhanced by the proposed method.

[0164] Table 4 Performance improvement results of the present invention on the traditional collaborative filtering model in multimodal recommendation tasks

[0165]

[0166] Among them, BPR † LightGCN is a new model obtained by combining the method proposed in this invention with the BPR model. † The results of Experiment 2 show that the performance of both models has been significantly improved by more than 40% after the multimodal representation pre-trained using the method proposed in this invention. † It performs competitively with models designed specifically for multimodal recommendation tasks, while LightGCN† These results demonstrate the effectiveness of the proposed method in learning high-quality robust multimodal representations.

[0167] like Figure 3 As shown, Figure 3 (a) and Figure 3 Panel (b) compares the recall and normalized discounted cumulative return of a complete method that incorporates all modules utilized by this invention, compared to models without counterfactual interaction sampling and bilinear interaction purification. As can be seen, removing each condition leads to varying degrees of decline in the model's predictive performance, demonstrating that both counterfactual interaction sampling and bilinear interaction purification contribute positively to improved recommendation performance. These results fully demonstrate that the proposed method achieves desirable results in multimodal recommendation.

[0168] Figure 4 The experimental results of the method proposed in this invention under different important parameter values ​​are shown below. Figure 4 (a) Figure 4 (b) Figure 4 (c) verifies the number of counterfactual neighbors , the performance impact of the method proposed in this invention on the multimodal recommendation task; Figure 4 Middle (d), Figure 4 Middle (e), Figure 4 (f) The number of graph neural network layers used to propagate collaborative features L 1. The performance impact of the proposed method on multimodal recommendation tasks; Figure 4 Middle (g), Figure 4 Middle (h), Figure 4 (i) Number of graph neural network layers used to propagate collaborative features L 2. The performance of the method proposed in this invention on the multimodal recommendation task. From the results, it can be seen that with As increases, more counterfactual neighbors provide meaningful information, enhancing the model’s ability to capture user preferences and thus improving recommendation performance. As increases, the performance starts to degrade because the noise introduced by irrelevant neighbors may distort the model’s predictions. The choice of the value needs to strike a balance between information richness and noise control to effectively utilize counterfactual interactions. Figure 4 Middle (d), Figure 4 Middle (e), Figure 4 As shown in (f), L Setting 1 to 2 or 3 enables the model to fully utilize the intrinsic relationship between user and item collaborative features, thereby mining more accurate user preference information. However, further increasingL 1 may lead to overfitting, thus affecting the accuracy of recommendations. L 2. Increasing the number of layers can enhance the model's ability to integrate and utilize modal features. Figure 4 Middle (g), Figure 4 Middle (h), Figure 4 As shown in (i), when L When 2 is set to 3 or 4, the model can more effectively capture the correlation between different modalities, thereby improving the recommendation performance. L 1 and L 2 is crucial for optimizing the performance of the method proposed in this invention.

[0169] like Figure 2 As shown, the present invention provides a multimodal recommendation system based on counterfactual comparison pre-training, comprising:

[0170] Data processing module 101: obtains user data, item data, and interaction data between user items, obtains a user-item interaction graph based on the user data, item data, and interaction data between user items, and constructs a counterfactual interaction graph based on the user-item interaction graph; randomly initializes to obtain user collaboration features and item collaboration features, and calculates item modal features based on item original modal features;

[0171] Bilinear interaction purification module 102: inputs the project collaborative features and project modal features into the bilinear interaction purification module to obtain purified project modal features; obtains user modal features based on the purified project modal features and the counterfactual interaction graph;

[0172] Multimodal representation pre-training module 103: calculates a pre-trained modal representation based on the modal features, calculates a pre-trained collaborative representation based on the collaborative features, performs modal representation fusion based on the pre-trained modal representation, calculates a final user multimodal representation and a final project multimodal representation, calculates a user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculates a project loss function using the pre-trained project collaborative representation and the final project multimodal representation;

[0173] Multimodal representation fine-tuning module 104: fine-tuning the pre-trained user multimodal representation and item multimodal representation on the user-item interaction graph according to the collaborative filtering model to obtain a final user representation and a final item representation;

[0174] Recommendation module 105: Calculates the user's preference score for each item based on the final user representation and the final item representation, and sorts the items based on the user's preference score to obtain a recommendation list.

[0175] Figure 5An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a multimodal recommendation method based on counterfactual comparison pre-training, which includes:

[0176] S1: Obtain user data, item data, and interaction data between user items, obtain a user-item interaction graph based on the user data, item data, and interaction data between user items, and construct a counterfactual interaction graph based on the user-item interaction graph;

[0177] S2: Randomly initialize to obtain user collaborative features and item collaborative features, and calculate the item modal features based on the item original modal features;

[0178] S3: Inputting the project collaborative features and project modal features into a bilinear interaction purification module to obtain purified project modal features; obtaining user modal features based on the purified project modal features and the counterfactual interaction graph;

[0179] S4: calculating a pre-trained modal representation based on the modal features, calculating a pre-trained collaborative representation based on the collaborative features, performing modal representation fusion based on the pre-trained modal representation, calculating a final user multimodal representation and a final item multimodal representation, calculating a user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculating an item loss function using the pre-trained item collaborative representation and the final item multimodal representation;

[0180] S5: fine-tuning the pre-trained user multimodal representation and item multimodal representation on the user-item interaction graph according to the collaborative filtering model to obtain a final user representation and a final item representation;

[0181] S6: Calculate the user's preference score for each item based on the final user representation and the final item representation, and sort the items based on the user's preference score to obtain a recommendation list.

[0182] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal recommendation method based on counterfactual contrast pre-training, characterized in that: include: S1: Obtain user data, item data, and interaction data between user items, obtain a user-item interaction graph based on the user data, item data, and interaction data between user items, and construct a counterfactual interaction graph based on the user-item interaction graph; S2: Randomly initialize to obtain user collaborative features and item collaborative features, and calculate the item modal features based on the item original modal features; S3: Inputting the project collaborative features and project modal features into a bilinear interaction purification module to obtain purified project modal features; obtaining user modal features based on the purified project modal features and the counterfactual interaction graph; S4: calculating a pre-trained modal representation based on the modal features, calculating a pre-trained collaborative representation based on the collaborative features, performing modal representation fusion based on the pre-trained modal representation, calculating a final user multimodal representation and a final item multimodal representation, calculating a user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculating an item loss function using the pre-trained item collaborative representation and the final item multimodal representation; S5: fine-tuning the pre-trained user multimodal representation and item multimodal representation on the user-item interaction graph according to the collaborative filtering model to obtain a final user representation and a final item representation; S6: Calculate the user's preference score for each item based on the final user representation and the final item representation, and sort the items based on the user's preference score to obtain a recommendation list.

2. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 1, characterized in that: The user-item interaction graph is represented as a binary interaction matrix. If there is an edge between a user node and an item node in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​1; if there is no edge between the user and the item in the user-item interaction graph, the corresponding position of the binary interaction matrix is ​​0.

3. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 1, characterized in that: Step S1 includes: S11: Calculate multimodal similarity: in, For users Interactive Projects and not with the user Projects that interact with each other In modal Multimodal similarity on , are the raw multimodal features of the items that the user interacts with, are the original multimodal features of items that have not been interacted with by the user, represents the transpose of the matrix, Indicates vector modulus; S12: The sample with the highest similarity to the user interaction item Uninteracted items are used to establish new edges with user nodes and initialize the weights of the new edges: in, For users Items that have not been interacted with The new edge weights between To obtain the highest A function of values, is the value quantity parameter; S13: Constructing counterfactual interaction graphs: in, is the counterfactual interaction graph, is the edge set, is the edge set of the user-item interaction graph, is the set of sampled counterfactual interaction edges.

4. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 3, characterized in that: Step S2 includes: S21: Randomly initialize to obtain user collaborative features and project collaboration features ; S22: Calculate the project modal features based on the original modal features of the project : in, is the trainable projection matrix, is the bias vector, The original modal characteristics of the project.

5. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 4, characterized in that: Step S3 includes: S31: Input the project collaborative features and project modal features into the bilinear interaction purification module to obtain the purified project modal features: in, is the purified item modal feature, is a learnable bilinear transformation weight matrix; S32: Obtaining user modal features based on the purified item modal features and the counterfactual interaction graph: in, is the user modal feature, For users In the counterfactual interaction diagram The set of neighbors in To find the number of elements in a set.

6. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 5, characterized in that: Step S4 includes: S41: Computing pre-trained co-representations and pre-trained modal representations: in, For the Co-representation of layers, is the layer number, , is the total number of layers, Counterfactual interaction diagram The adjacency matrix of for The degree matrix of For the Co-representation of layers, For the The modal representation of the layer, For the Modal representation of layers; in, hour, and The calculation formula is as follows: in, is the collaborative representation of layer 0, For the user collection User collaborative features of each user, , For user collection, For project collection Project collaboration characteristics of each project, , For the project collection, is the modal representation of layer 0, For the user collection User modal features of each user, Project Collection The modal characteristics of the items after purification; S42: Calculate the final modal representation of the pre-training phase according to the modal representation, and calculate the final collaborative representation of the pre-training phase according to the collaborative representation: in, is the final modality representation in the pre-training stage, The final collaborative representation for the pre-training stage; S43: Perform modal representation fusion based on the pre-trained modal representation to calculate the final user multimodal representation and the final project multimodal representation: in, Multimodal representation for end users, For visual mode, For text mode, is the user-learnable modality importance weight, For The corresponding user modality representation extracted, Final project modal characterization, is the modal importance weight that can be learned by the project, For The corresponding item modality representations extracted; S44: Calculate the contrast loss between the final user collaborative representation and the final user multimodal representation, and calculate the contrast loss between the final item collaborative representation and the final item multimodal representation until the loss value is convergence: in, is the user loss function, To compare users, is the cosine similarity function, For The corresponding user collaborative representation extracted, is the temperature coefficient, For The corresponding contrast user collaborative representations extracted, is the item loss function, For comparison items, For The corresponding project co-representation extracted, For The corresponding contrast item-user co-representations are extracted.

7. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 6, characterized in that: Step S5 includes: S51: using the final modal representation of the pre-training stage to initialize the collaborative filtering model, and using a graph convolutional network to propagate information in the user-item interaction graph to obtain user representation and item representation; S52: Fine-tuning the loss function through optimization , optimize the collaborative filtering model: in, is the training data, , are the items in the edge set of the user-item interaction graph, For items that are not in the edge set of the user-item interaction graph, is the sigmoid function, For users About the project The preference score of For users About the project The preference score of is the weight hyperparameter, are the trainable parameters of the model, is the quadratic norm; S53: When The user representation output by the collaborative filtering model at convergence is the final user representation. The item representation output by the collaborative filtering model at convergence is the final item representation.

8. The multimodal recommendation method based on counterfactual contrast pre-training according to claim 7, characterized in that: Step S6 includes: S61: Performing a dot product operation on the final user representation and the final item representation to calculate the user's preference score for each item; S62: Sort all items according to the preference scores and generate a personalized recommendation list for the user.

9. A multimodal recommendation system based on counterfactual contrast pre-training, for executing the multimodal recommendation method based on counterfactual contrast pre-training according to any one of claims 1 to 8, characterized in that: include: Data processing module: obtains user data, project data, and interaction data between user projects, obtains a user-project interaction graph based on the user data, project data, and interaction data between user projects, and constructs a counterfactual interaction graph based on the user-project interaction graph; randomly initializes to obtain user collaboration features and project collaboration features, and calculates project modal features based on the original modal features of the projects; Bilinear interaction purification module: input the project collaborative features and project modal features into the bilinear interaction purification module to obtain purified project modal features; obtain user modal features based on the purified project modal features and the counterfactual interaction graph; Multimodal representation pre-training module: calculates pre-trained modal representation based on the modal features, calculates pre-trained collaborative representation based on the collaborative features, performs modal representation fusion based on the pre-trained modal representation, calculates the final user multimodal representation and the final project multimodal representation, calculates the user loss function using the pre-trained user collaborative representation and the final user multimodal representation, and calculates the project loss function using the pre-trained project collaborative representation and the final project multimodal representation; Multimodal representation fine-tuning module: Based on the collaborative filtering model, the pre-trained user multimodal representation and item multimodal representation are fine-tuned on the user-item interaction graph to obtain the final user representation and the final item representation; Recommendation module: Calculates the user's preference score for each item based on the final user representation and the final item representation, and sorts the items according to the user's preference score to obtain a recommendation list.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the multimodal recommendation method based on counterfactual contrast pre-training as described in any one of claims 1 to 8 are implemented.

Images