Self-supervised learning multi-modal recommendation method and system based on diffusion model

Abstract

The invention discloses a self-supervised learning multi-modal recommendation method and system based on a diffusion model. The method comprises the steps that loss functions adopted during training of a multi-modal recommendation model comprise the steps that a normalized interaction matrix of user articles is used for generating a user article collaboration diagram through a diffusion model, ID guiding is carried out, and modal semantic generality is extracted so as to obtain an ID-guided collaboration diagram and collaboration diagrams guided by all modals; and respectively calculating the loss represented by the ID guide mode and the loss for enhancing semantic consistency, and summing to obtain the self-supervised loss. The invention aims to generate interaction data by using a diffusion model so as to inject modal information and ID information, compare the injected data to meet modal alignment and ID-guided feature representation, and inject modal perception cooperation signals in self-supervised learning. Therefore, the recommendation accuracy of the multi-modal recommendation system is improved by obtaining effective self-supervision loss.

Description

Technical Field

[0001] The present invention relates to an item recommendation technology in the field of data processing technology, and in particular to a self-supervised learning multimodal recommendation method and system based on a diffusion model. Background Art

[0002] With the rapid development of the Internet, people's lifestyles have undergone tremendous changes. More and more users choose to conduct shopping, entertainment, learning and other activities through online platforms. This has led to the accumulation of massive amounts of user data and item data on various e-commerce platforms, content platforms, etc. How to accurately recommend items that meet users' interests and needs from these massive data has become a key issue in improving user experience and platform competitiveness. Item recommendation technology has emerged as the times require.

[0003] In recent years, commonly used data generation techniques in the field of recommendation systems include generative adversarial networks, variational autoencoders, and diffusion models. These methods can generate high-fidelity image, text, audio and other data by learning the potential distribution of data through neural networks. In the recommendation system, since there is less historical interaction data between users and items, their potential interests can be simulated through generation technology, thereby improving the accuracy of recommendations. In addition, data generation technology can also be used to generate feature data for long-tail products or unpopular content, making up for the problem of uneven data distribution in the recommendation system and improving the ability to recommend unpopular products. Data generation technology enriches the historical interaction data between users and items, alleviates the data sparsity problem to a certain extent, and effectively improves the performance of the recommendation model. Using generated data to compare with original data to generate self-supervisory signals is a common strategy. However, if the modal perception signal cannot be injected into the self-supervised learning task, it cannot reflect the different preferences of users for items, and the extracted self-supervisory signal will lose its effect. Summary of the invention

[0004] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a self-supervised learning multimodal recommendation method and system based on a diffusion model is provided. The present invention aims to use a diffusion model to generate interactive data to inject modal information and ID information, compare the injected data to meet the requirements of modal alignment and ID-guided feature representation, and inject modal-aware collaborative signals into self-supervised learning, thereby obtaining an effective self-supervised loss to improve the recommendation accuracy of the multimodal recommendation system.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A self-supervised learning multimodal recommendation method based on a diffusion model includes the following steps: embedding the input user original ID into 、The original ID of the item is embedded And the modal features of the two modalities of the item image and text A multimodal recommendation model is used to obtain a list of recommended items, wherein the loss function used in the training of the multimodal recommendation model includes a self-supervised loss , and the self-supervised loss The calculation includes: normalizing the interaction matrix of user items Generating user-item collaboration graph using diffusion model , and the user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram , respectively calculate the loss of ID-guided modality representation and the loss of enhanced semantic consistency and sum them as the self-supervised loss ,in The value is or , and Represent image modality and text modality respectively.

[0006] Optionally, the normalized interaction matrix of the user item Generating user-item collaboration graph using diffusion model When the normalized interaction matrix The calculation function expression is: , in, is the original interaction matrix of user items, is a normalized degree value matrix; the diffusion model is based on the normalized interaction matrix of the input user items , the user-item collaboration graph is generated through multiple iterations using the reverse generation process shown in the following formula : , in, is the time step The data distribution, is the time step The data distribution, Depend on The reverse prediction is: represents a normal distribution, is the time step The noise, is the time step The diffusion coefficient, is the time step The cumulative diffusion coefficient, is the time step The cumulative diffusion coefficient.

[0007] Optionally, the user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram The function expression is: , , in, is the interaction matrix of user items, For splicing operation, and For users The modal features of and the modal features of items, and The original ID of the user and the original ID of the item are embedded respectively, so that the ID-guided collaboration graph and each mode Guided Collaboration Diagram Both consist of two embedding parts representing users and items.

[0008] Optionally, the loss of ID-guided modality representation and the loss of enhanced semantic consistency are calculated separately and summed as the self-supervised loss The function expression is: , , , , , in, is the loss of ID-guided modality representation, To enhance the loss of semantic consistency; and is the intermediate variable, is the mode set, For this batch of data, is the similarity function, is the temperature coefficient, They are respectively ID-guided collaboration diagrams and each mode Guided Collaboration Diagram Medium User The embedded part of They are respectively ID-guided collaboration diagrams and each mode Guided Collaboration Diagram Medium Items The embedded part of For modal Guided Collaboration Diagram Medium User The embedded part, For modal Guided Collaboration Diagram Medium Items The embedded part of the user and users The dataset for this batch Users, items in and items The dataset for this batch Items in Mode Collaboration diagram for image mode and text mode Users in The embedded part of For modal Modal when it is text modal Guided Collaboration Diagram Medium User The embedded part, The modal Collaboration diagram for image mode and text mode Medium Items The embedded part, For modal Modal when it is text modal Guided Collaboration Diagram Medium Items The embedded part.

[0009] Optionally, the loss function used in training the multimodal recommendation model is: , in, is the loss function used in training the multimodal recommendation model. is the Bayesian loss, is the self-supervised loss coefficient, is the self-supervision loss.

[0010] Optionally, the input user original ID is embedded in 、The original ID of the item is embedded And the modal features of the two modalities of the item image and text Using the multimodal recommendation model to obtain the item recommendation list includes: embedding the user's original ID separately Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID , embed the updated user ID into Embedded with item ID Collaborative view embedding via graph convolution ; The modal features of each mode After projection, user preferences are calculated to obtain user The modal characteristics of , modal enhancement to obtain the modal features of items , the user The modal characteristics of , modal characteristics of items Obtaining each modality preference view embedding through graph convolution ; Embed each modal preference view into Regularized and collaborative view embedding Fusion gets view fusion embedding , embedding the view fusion Split to get the final user representation With item representation , the final user represents With item representation Perform the inner product to get each user For each item Rating , and for each user Select Rating The top K items are output as the resulting item recommendation list.

[0011] Optionally, the original ID of the user is embedded in Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID The function expression is: The user's original ID is embedded Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID The function expression is: , , , , in, and Respectively represent the user node similarity graph and the item node similarity graph, is the normalized interaction matrix of user items, represents the transposition operation; the modal features under each mode are After projection, user preferences are calculated to obtain user The modal characteristics of , modal enhancement to obtain the modal features of items The function expression is: , , , in, For users The neighbor set on the original interaction matrix of user-items, Neighborhood Set The size of For items The embedding representation after combining items with similar modality features, is the modal features of the items projected into a unified embedding space, A modal semantic graph used to establish associations between items with similar modal characteristics. is the transformation matrix.

[0012] In addition, the present invention also provides a self-supervised learning multimodal recommendation system based on a diffusion model, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model.

[0013] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model through a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or an instruction, wherein the computer program or the instruction is programmed or configured to execute the diffusion model-based self-supervised learning multimodal recommendation method through a processor.

[0015] Compared with the prior art, the present invention has the following advantages: the loss function used in the training of the multimodal recommendation model of the present invention includes self-supervised loss , and the self-supervised loss The calculation includes: normalizing the interaction matrix of user items Generating user-item collaboration graph using diffusion model , and the user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram , respectively calculate the loss of ID-guided modality representation and the loss of enhanced semantic consistency and sum them as the self-supervised loss , the present invention generates interactive data by using a diffusion model to inject modal information and ID information, compares the injected data to meet modal alignment, ID-guided feature representation, and injects modality-aware collaborative signals in self-supervised learning, thereby obtaining an effective self-supervised loss to improve the recommendation accuracy of the multimodal recommendation system. The present invention effectively improves the prediction accuracy of the recommendation system by combining the self-supervised signal generated by the diffusion model with multimodal information. This technology can provide users with more accurate personalized recommendations and is suitable for various multimodal recommendation application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the calculation process of the self-supervision loss in an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of the principle of multimodal recommendation model and self-supervisory loss calculation in an embodiment of the present invention.

[0018] Figure 3 The figure is a schematic diagram of the workflow of the multimodal recommendation model in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0020] like Figure 1 and Figure 2 As shown, the self-supervised learning multimodal recommendation method based on the diffusion model in this embodiment includes the following steps: embedding the input user original ID into 、The original ID of the item is embedded And the modal features of the two modalities of the item image and text A multimodal recommendation model is used to obtain a list of recommended items, wherein the loss function used in the training of the multimodal recommendation model includes a self-supervised loss , and the self-supervised loss The calculation includes: normalizing the interaction matrix of user items Generating user-item collaboration graph using diffusion model , and the user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram , respectively calculate the loss of ID-guided modality representation and the loss of enhanced semantic consistency and sum them as the self-supervised loss ,in The value is or , and Represent image modality and text modality respectively.

[0021] In this embodiment, the normalized interaction matrix of the user item Generating user-item collaboration graph using diffusion model When the normalized interaction matrix The calculation function expression is: , in, is the original interaction matrix of user items, is the normalized degree matrix.

[0022] The forward propagation process of the diffusion model adds noise to the original interaction graph and finally transforms it into an approximate Gaussian distribution, which can be expressed as: , in represents the original interaction graph, Indicates Add t The result after step noise, is the noise weight coefficient, Represents Gaussian noise ; is the time step In this embodiment, the diffusion model is based on the normalized interaction matrix of the input user items. , the user-item collaboration graph is generated through multiple iterations using the reverse generation process shown in the following formula : , in, is the time step The data distribution, is the time step The data distribution, Depend on The reverse prediction is: represents a normal distribution, is the time step The noise, is the time step The diffusion coefficient, is the time step The cumulative diffusion coefficient, is the time step The cumulative diffusion coefficient.

[0023] In this embodiment, the user-item collaboration map Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram The function expression is: , , in, is the interaction matrix of user items, For splicing operation, and For users The modal features of and the modal features of items, and The original ID of the user and the original ID of the item are embedded respectively, so that the ID-guided collaboration graph and each mode Guided Collaboration Diagram Both consist of two embedding parts representing users and items.

[0024] In this embodiment, the loss of ID-guided modality representation and the loss of enhanced semantic consistency are calculated separately and summed as the self-supervised loss The function expression is: , , , , , in, is the loss of ID-guided modality representation, To enhance the loss of semantic consistency; and is the intermediate variable, is the mode set, For this batch of data, is the similarity function, is the temperature coefficient, They are respectively ID-guided collaboration diagrams and each mode Guided Collaboration Diagram Medium User The embedded part of They are respectively ID-guided collaboration diagrams and each mode Guided Collaboration Diagram Medium Items The embedded part of For modal Guided Collaboration Diagram Medium User The embedded part of For modal Guided Collaboration Diagram Medium Items The embedded part of the user and users The dataset for this batch Users, items in and items The dataset for this batch Items in The modal Collaboration diagram for image mode and text mode Users in The embedded part of For modal Modal when it is text modal Guided Collaboration Diagram Medium User The embedded part of The modal Collaboration diagram for image mode and text mode Medium Items The embedded part of For modal Modal when it is text modal Guided Collaboration Diagram Medium Items The embedded part.

[0025] In this embodiment, the loss function used in the training of the multimodal recommendation model is: , in, is the loss function used in training the multimodal recommendation model. is the Bayesian loss, is the self-supervised loss coefficient, is the self-supervision loss. The self-supervision loss coefficient reflects the degree to which the corresponding part affects the optimization during the model optimization process, and the required value can be selected according to the needs. The above loss is combined with the Bayesian loss, and the model is optimized as the optimization loss function. The trained model can use the user's historical interaction behavior and multimodal information to make accurate personalized recommendations to users. The calculation function expression of Bayesian loss is: , in, and Is a user About Project , The prediction score of and Respectively represent and user Interacted with and did not interact with the user Interacted projects, is the activation function.

[0026] Figure 3 FIG. 1 is a schematic diagram of the workflow of the multimodal recommendation model in this embodiment. Figure 3 As shown, in this embodiment, the input user's original ID is embedded in 、The original ID of the item is embedded And the modal features of the two modalities of the item image and text Using a multimodal recommendation model to obtain a list of recommended items includes: Multimodal data preprocessing: extracting user original ID embedding Embedded with the original ID of the item And the modal characteristics under each mode ; Embed the user's original ID Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID , the modal features under each mode After projection, user preferences are calculated to obtain user The modal characteristics of , modal enhancement to obtain the modal features of items ; Multimodal information representation: embedding the updated user ID Embedded with item ID Collaborative view embedding via graph convolution ; The user The modal characteristics of , modal characteristics of items Obtaining each modality preference view embedding through graph convolution ; Multimodal feature fusion: embedding each modality preference view Regularized and collaborative view embedding Fusion gets view fusion embedding ; Recommendation result prediction: embedding view fusion Split to get the final user representation With item representation , the final user represents With item representation Perform the inner product to get each user For each item Rating , and for each user Select Rating The top K items are output as the resulting item recommendation list.

[0027] In this embodiment, the user's original ID is embedded Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID The function expression is: , , , , in, and Respectively represent the user node similarity graph and the item node similarity graph, is the normalized interaction matrix of user items, represents a transposition operation; as an optional implementation, in this embodiment, the user's original ID is embedded in Embedded with the original ID of the item The normalized interaction matrix of users and items is extracted from the original data set containing the interaction between users and items using the built-in tool class of MMRec. It is used to represent the interaction relationship between users and items. It is a two-dimensional matrix, in which rows represent users and columns represent items. Each element in the matrix represents whether there is an interaction between the user and the item. Each element takes a value of 1 or 0, where 1 indicates interaction (such as browsing, clicking, etc.) and 0 indicates no interaction. In addition, a more specific quantitative interaction method can also be used. For example, an evaluation of 1 to 5 points can be quantified into a value between 0 and 1.

[0028] In this embodiment, the modal features under each mode are After projection, the user preferences are calculated separately to obtain the modal features of user u , modal enhancement to obtain the modal features of items The function expression is: , , , in, For users The neighbor set on the original interaction matrix of user-items, Neighborhood Set The size of For items The embedding representation after combining items with similar modality features, is the modal features of the items projected into a unified embedding space, A modal semantic graph used to establish associations between items with similar modal characteristics. is the transformation matrix. In this embodiment, the modal features under each mode are Including modality features under image modality (visual modality) and the modal features under text modal , which are extracted from the original dataset based on the pre-trained Transformer model and ResNet model respectively. First, the modal features can be projected into a unified embedding space by the following formula: , Then, the modal semantic graph is updated to embed the modal characteristics of the item through the following formula: : , Finally, the modal features of the items are aggregated to obtain the user's modal preference: , So that users can get The modal characteristics of Modal semantic graph The existing method is to obtain by constructing an association graph between items with similar modal features, and then performing weighted summation of association graphs of multiple modalities.

[0029] In this embodiment, the updated user ID is embedded in Embedded with item ID Collaborative view embedding via graph convolution , the user The modal characteristics of , modal characteristics of items Obtaining each modality preference view embedding through graph convolution The graph convolution used is an existing method, which includes It uses lightweight graph propagation to learn high-level information of ID embedding or modality.

[0030] In this embodiment, the updated user ID is embedded in Embedded with item ID Collaborative view embedding via graph convolution At any time The function expression of the high-order information of the graph propagation learning ID embedding of the graph convolution layer is: , , in, and Respectively The user ID embedding and item ID embedding output by the graph convolutional layer, and Respectively The user ID embedding and item ID embedding output by the graph convolutional layer, and Respectively represent users With items Normalized interaction matrix of user-items The set of one-hop neighbors on the graph is a symmetric normalization coefficient to avoid the increase in scale during graph convolution operations. , and the initial user ID embedding and item ID embedding are , .go through After the message passing of the layers, the representation of each graph convolutional layer is integrated by summing, and then combined to obtain the representation of the collaborative view embedding, which can be expressed as: , , , in, and They are user ID embedding and item ID embedding respectively. To perform row-by-row concatenation, To embed the updated user ID into Embedded with item ID Collaborative view embedding via graph convolution.

[0031] In this embodiment, the user The modal characteristics of , modal characteristics of items Obtaining each modality preference view embedding through graph convolution At any time The function expression of the high-order information of the graph propagation learning mode of the graph convolution layer is: , , in, and Respectively The modality-unified embedding of the output of the graph convolutional layer, and Respectively The modality-unified embedding of the output of the graph convolutional layer, and Respectively represent users With items Normalized interaction matrix of user-items The set of one-hop neighbors on the graph is a symmetric normalization coefficient to avoid the increase in scale during graph convolution operations. , and the initial user ID embedding and item ID embedding are , .go through After the message passing of the layers, the representation of each graph convolutional layer is integrated by summing, and then combined to obtain the representation of the collaborative view embedding, which can be expressed as: , , , in, and They are user modality unified embedding and item modality unified embedding, For users The modal characteristics of , modal characteristics of items Modality obtained through graph convolution A modal preference view embed.

[0032] In this embodiment, each modal preference view is embedded in Regularized and collaborative view embedding Fusion gets view fusion embedding It can be expressed as: , in, is a normalization function used to achieve regularization, which is used to alleviate the difference in value scales between modes.

[0033] In this embodiment, the view fusion is embedded in Split to get the final user representation With item representation , the final user represents With item representation Perform the inner product to get each user For each item Rating , which can be expressed as: , Among them, the superscript represents the transposition operation. Finally, for each user Select Rating The top K items are output as the resulting item recommendation list.

[0034] In order to verify the performance of the self-supervised learning multimodal recommendation method based on the diffusion model in this embodiment, the datasets used in this embodiment are the comment datasets of Amazon_Baby (referred to as the Baby dataset) and Amazon_Sports (referred to as the Sports dataset). The existing recommendation methods compared with the method of this embodiment (SSLDM) include: LATTICE, DualGNN, MICRO, DiffMM and BM3. The evaluation indicators used are recall rate (Recall) and normalized discounted cumulative gain (NDCG), and experiments are carried out for different numbers of items K recommended to users. The results are shown in Tables 1 and 2.

[0035] Table 1: Comparative experimental results on the Baby dataset

[0036] Table 2: Comparative experimental results on the Sports dataset

[0037] In Tables 1 and 2, R@10 and R@20 are respectively the recall rates (Recall) when the number of items K recommended to the user is 10 and 20, and N@10 and N@20 are respectively the normalized discounted cumulative gain (NDCG) when the number of items K recommended to the user is 10 and 20. As shown in Tables 1 and 2, the method of this embodiment (SSLDM) is superior to the existing recommendation methods in R@10, R@20, N@10, and N@20.

[0038] In order to verify the effectiveness of each part of the self-supervised learning multimodal recommendation method based on the diffusion model in this embodiment, the data sets used are the Baby data set and the Sports data set. R@20 and N@20 are used as evaluation indicators to analyze the impact of each part on the overall model of the method (SSLDM) in this embodiment. The results are shown in Table 3.

[0039] Table 3: Ablation experiment results on Baby dataset and Sports dataset

[0040] In Table 3, w / o It means that the user node similarity graph and the item node similarity graph ( and ) after the variant, w / o Indicates that the modal semantic graph is removed based on the method of this embodiment (SSLDM) The resulting variant, w / o Indicates that the self-supervision loss is removed based on the method in this embodiment (SSLDM) As can be seen from Table 3, the node similarity graph, item semantic graph and self-supervised learning part used in the method (SSLDM) of this embodiment have different degrees of influence on SSLDM, which fully proves the effectiveness of each part of the method (SSLDM) of this embodiment.

[0041] It can be seen that this embodiment uses the diffusion model to generate interactive data to inject modal information and ID information, compares the injected data to meet the requirements of modal alignment and ID-guided feature representation, and injects modality-aware collaborative signals in self-supervised learning, thereby obtaining an effective self-supervised loss to improve the recommendation accuracy of the multimodal recommendation system. This embodiment effectively improves the prediction accuracy of the recommendation system by combining the self-supervised signal generated by the diffusion model with multimodal information. This technology can provide users with more accurate personalized recommendations and is suitable for various multimodal recommendation application scenarios.

[0042] In addition, this embodiment also provides a self-supervised learning multimodal recommendation system based on a diffusion model, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model.

[0043] In addition, this embodiment also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model through a processor.

[0044] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the diffusion model-based self-supervised learning multimodal recommendation method through a processor.

[0045] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present invention may be in the form of methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of a computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0046] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A self-supervised learning multimodal recommendation method based on a diffusion model, characterized in that: The steps include: embedding the input user's original ID into 、Item original ID embed And the modal features of the two modalities of the item image and text A multimodal recommendation model is used to obtain a list of recommended items, wherein the loss function used in the training of the multimodal recommendation model includes a self-supervised loss , and the self-supervised loss The calculation includes: normalizing the interaction matrix of user items Generating user-item collaboration graph using diffusion model , and the user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram , respectively calculate the loss of ID-guided modality representation and the loss of enhanced semantic consistency and sum them as the self-supervised loss ,in The value is or , and Represent image modality and text modality respectively.

2. The self-supervised learning multimodal recommendation method based on the diffusion model according to claim 1, characterized in that: The normalized interaction matrix of the user item Generating user-item collaboration graph using diffusion model When the normalized interaction matrix The calculation function expression is: ， in, is the original interaction matrix of user items, is a normalized degree value matrix; the diffusion model is based on the normalized interaction matrix of the input user items , the user-item collaboration graph is generated through multiple iterations using the reverse generation process shown in the following formula : ， in, is the time step The data distribution, is the time step The data distribution, Depend on The reverse prediction is: represents a normal distribution, is the time step The noise, is the time step The diffusion coefficient, is the time step The cumulative diffusion coefficient, is the time step The cumulative diffusion coefficient.

3. The self-supervised learning multimodal recommendation method based on the diffusion model according to claim 1, characterized in that: The user-item collaboration graph Perform ID guidance and extract modal semantic commonality to obtain ID-guided collaboration graph and each mode Guided Collaboration Diagram The function expression is: ， ， in, is the interaction matrix of user items, For splicing operation, and For users The modal features of and the modal features of items, and The original ID of the user and the original ID of the item are embedded respectively, so that the ID-guided collaboration graph and each mode Guided Collaboration Diagram Both consist of two embedding parts representing users and items.

4. The self-supervised learning multimodal recommendation method based on the diffusion model according to claim 1, characterized in that: The loss of ID-guided modality representation and the loss of enhanced semantic consistency are calculated separately and summed as the self-supervised loss The function expression is: ， ， ， ， ， in, is the loss of ID-guided modality representation, To enhance the loss of semantic consistency; and is the intermediate variable, is the mode set, For this batch of data, is the similarity function, is the temperature coefficient, The collaboration diagrams guided by ID and each mode Guided Collaboration Diagram Medium User The embedded part of The collaboration diagrams guided by ID and each mode Guided Collaboration Diagram Medium Items The embedded part of For modal Guided Collaboration Diagram Medium User The embedded part of For modal Guided Collaboration Diagram Medium Items The embedded part of the user and users The dataset for this batch Users, items in and items The dataset for this batch Items in The modal Collaboration diagram for image mode and text mode Users in The embedded part of For modal Modal when it is text modal Guided Collaboration Diagram Medium User The embedded part of Mode Collaboration diagram for image mode and text mode Medium Items The embedded part of For modal Modal when it is text modal Guided Collaboration Diagram Medium Items The embedded part.

5. The self-supervised learning multimodal recommendation method based on diffusion model according to claim 1, characterized in that: The loss function used in the training of the multimodal recommendation model is: ， in, is the loss function used in training the multimodal recommendation model. is the Bayesian loss, is the self-supervised loss coefficient, is the self-supervision loss.

6. The self-supervised learning multimodal recommendation method based on diffusion model according to claim 1, characterized in that: The input user's original ID is embedded in 、Item original ID embed And the modal features of the two modalities of the item image and text Using the multimodal recommendation model to obtain the item recommendation list includes: embedding the user's original ID separately Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID , embed the updated user ID into Embedded with item ID Collaborative view embedding via graph convolution ; The modal features of each mode After projection, user preferences are calculated to obtain user The modal characteristics of , modal enhancement to obtain the modal features of items , the user The modal characteristics of , modal characteristics of items Obtaining each modality preference view embedding through graph convolution ; Embed each modal preference view into Regularized and collaborative view embedding Fusion gets view fusion embedding , embedding the view fusion Split to get the final user representation With item representation , the final user represents With item representation Perform the inner product to get each user For each item Rating , and for each user Select Rating The top K items are output as the resulting item recommendation list.

7. The self-supervised learning multimodal recommendation method based on diffusion model according to claim 6, characterized in that: The user's original ID is embedded Embedded with the original ID of the item Perform similarity graph augmentation to obtain updated user ID embeddings Embedded with item ID The function expression is: ， ， ， ， in, and Respectively represent the user node similarity graph and the item node similarity graph, is the normalized interaction matrix of user items, represents the transposition operation; the modal features under each mode are After projection, user preferences are calculated to obtain user The modal characteristics of , modal enhancement to obtain the modal features of items The function expression is: ， ， ， in, For users The neighbor set on the original interaction matrix of user-items, Neighborhood Set The size of For items The embedding representation after combining items with similar modality features, is the modal features of the items projected into a unified embedding space, A modal semantic graph used to establish associations between items with similar modal characteristics. is the transformation matrix.

8. A self-supervised learning multimodal recommendation system based on a diffusion model, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model as described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the self-supervised learning multimodal recommendation method based on the diffusion model as described in any one of claims 1 to 7 through a processor.

Images