A multi-modal robust recommendation method and system based on self-supervised learning

Abstract

The application discloses a kind of multi-modal robust recommendation method and system based on self-supervised learning, the application includes extracting the modal feature of each modality of user and article from original data as initial embedding vector, and constructs the user-article heterogeneous graph under each modality, and article-article homogenous graph;For the initial embedding vector of each modality, multi-modal feature coding is carried out, and the final embedding vector of each modality after enhancement is fused with multi-modal feature;Multi-modal feature alignment is carried out to extract final user embedding vector and final article embedding vector, to calculate the final predicted score of user to article and sort, select specified top K article as the recommendation list recommended to user and output.The application aims at eliminating the influence of inherent noise in multi-modal data, and improving the recommendation effect of the whole multi-modal recommendation system.

Description

Technical Field

[0001] This invention belongs to the field of data processing and recommendation system technology, specifically relating to a multimodal robust recommendation method and system based on self-supervised learning. Background Technology

[0002] Early recommender systems typically used collaborative filtering, modeling user preferences based on user-item interaction records. By calculating the similarity between users (items), they recommended similar items to users or recommended items to similar users. This method considered only user interaction behavior, ignoring multimodal information about items, thus having significant limitations. With the development of multimodal technology, integrating multimodal information into recommender systems has become a common method to improve their performance. Multimodal recommender systems model user preferences based on behavioral and item multimodal information. A common approach is multimodal representation learning, using historical interactions to create user and item ID embeddings, and then using multimodal data as auxiliary information for items to enrich their ID embeddings. While the introduction of multimodal information has helped improve recommendation performance, several problems remain. These include inherent issues in recommender systems such as data sparsity and the cold start problem. In addition to these inherent problems, multimodal recommender systems also face challenges in fusing and aligning various modal features, which to some extent affect the performance of the recommender system. Meanwhile, the introduction of multimodal information increases the risk of multimodal noise affecting recommendation performance, thus improving system robustness is also a crucial area for research. In summary, real-world multimodal recommendation scenarios involve a large amount of noise, including inherent noise in the multimodal information itself and noise generated during modality fusion and alignment. This noise affects the overall performance of the multimodal recommendation system, leading to unsatisfactory recommendation results. Therefore, eliminating inherent noise in the multimodal information and noise generated during modality fusion and alignment, and improving the system robustness of multimodal recommendation systems, has become a critical technical problem that urgently needs to be solved. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a multimodal robust recommendation method and system based on self-supervised learning, which aims to eliminate the influence of inherent noise in multimodal data and improve the recommendation effect of the entire multimodal recommendation system.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A multimodal robust recommendation method based on self-supervised learning includes obtaining a recommendation list for users from the original data using a self-supervised learning-based recommendation model:

[0006] S1. Extract modal features of users and items in each modality from the original data as initial embedding vectors, and construct user-item heterogeneous graphs and item-item isomorphic graphs in each modality.

[0007] S2, perform multimodal feature encoding on the initial embedding vectors of each modality, including: using graph convolutional networks to capture high-order features based on the initial embedding vectors and user-item heterogeneous graphs to obtain the final embedding vectors, and extracting semantic relationships between items based on item-item isomorphic graphs to enhance the final embedding vectors.

[0008] S3, perform multimodal feature fusion on the final embedded vectors after modal enhancement;

[0009] S4, perform multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector. and final item embedding vector ;

[0010] S5, based on the end-user embedding vector and final item embedding vector Calculate and sort the user's final predicted rating for the items, select the top K items as the recommendation list to recommend to the user, and output the list.

[0011] Optionally, in step S2, when capturing high-order features using a graph convolutional network based on the initial embedding vector and the user-item heterogeneous graph to obtain the final embedding vector, the graph convolutional network used includes: The nth graph convolutional network layer, any nth graph convolutional network layer, The functional expression for the embedding vector obtained from the layer is:

[0012] ,

[0013] In the above formula, For the first The embedding vector obtained from the layer, and Let u be a single-hop neighbor of item i in the user-item heterogeneity graph. For the first The embedding vector obtained from the layer, where m is the modality, takes values ​​id, v, and t to represent the ID modality, image modality, and text modality, respectively; and the functional expression for the final embedding vector obtained by capturing higher-order features is:

[0014] ,

[0015] In the above formula, Let be the final embedding vector under mode m, and we have:

[0016] ,

[0017] In the above formula, and The final embedding vectors under mode m are respectively User embedding vectors and item embedding vectors;

[0018] Optionally, step S2, which involves extracting semantic relationships between items based on item-item isomorphism graphs to enhance the final embedding vector, includes:

[0019] S2.1, calculate the similarity score between each pair of items in the item-item isomorphism graph according to the following formula:

[0020] ,

[0021] In the above formula, For modality Lowering items and items Similarity scores between them and Modal Lowering items and items The original modal features, where m is the modality, and the modality m takes the values ​​id, v, and t to represent the ID modality, image modality, and text modality, respectively;

[0022] S2.2, use the KNN algorithm to cluster the similarity scores between pairs of items, and retain the similarity scores of the K highest-scoring neighbors according to the following formula. The value is 1, and the similarity score of the other neighbors. The value is 0;

[0023] S2.3, Calculate arbitrary modes using sparse matrix multiplication aggregation according to the following formula. The semantic relationships between the items obtained below:

[0024] ,

[0025] In the above formula, For modality The embedding vector of item i. For item i, a single-hop neighbor For modality Lowering items Embedding vectors of items For the single-hop neighbors of item i Items in the;

[0026] S2.4, enhance the final embedding vector of the corresponding modality by using the semantic relationships between items.

[0027] Optionally, the function expression for enhancing the final embedding vector of the corresponding modality by using the semantic relationships between items in step S2.4 is as follows:

[0028] ,

[0029] In the above formula, For modality The final embedding vector after enhancement and The final embedding vectors under mode m are respectively User embedding vectors and item embedding vectors, "For connection operation, For modality The embedding vector of item i.

[0030] Optionally, the function expression for multimodal feature fusion of the final embedded vectors after modal enhancement in step S3 is as follows:

[0031] ,

[0032] In the above formula, This is the final embedding vector for the fused modality. These are the weighting coefficients. and Modal The final embedding vector after enhancement when the values ​​are image modality v and text modality t. And there are:

[0033] ,

[0034] ,

[0035] ,

[0036] In the above formula, and These are the final embedding vectors after enhancement under image modality v. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement under text modality t. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement in the fusion mode. User embedding vectors and item embedding vectors.

[0037] Optionally, in step S4, the final embedding vectors of each modality and the fused final embedding vector are aligned with multimodal features to extract the final user embedding vector. and final item embedding vector The function expression is:

[0038] S4.1, construct the final embedding vector shown in the following formula based on the final embedding vectors of each modality and the fused final embedding vector. :

[0039] ,

[0040] In the above formula, and These are the final embedding vectors after enhancement under image modality v. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement under text modality t. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement in the fusion mode. User embedding vectors and item embedding vectors;

[0041] S4.2, based on the final embedding vector The final user embedding vector is generated according to the following formula. and final item embedding vector :

[0042] , .

[0043] Optionally, in step S5, based on the final user embedding vector and final item embedding vector The function expression for calculating the user's final predicted rating for an item is:

[0044] ,

[0045] In the above formula, For users For items The final predicted score.

[0046] Optionally, before obtaining the recommendation list for the user from the original data using a self-supervised learning-based recommendation model, the process further includes training the recommendation model. The loss function used in training the recommendation model is a weighted sum of the Bayesian personalized ranking optimization loss and the alignment loss. Furthermore, a mirror gradient optimization algorithm is used in conjunction with the loss function to optimize the network parameters of the recommendation model. The expression for the alignment loss is:

[0047] ,

[0048] In the above formula, For alignment loss, These are the hyperparameters used to balance the loss. For alignment loss directly guided by ID, For alignment loss indirectly guided by ID, For modality-guided alignment loss, Let be the alignment loss indirectly guided by modality, and we have:

[0049] ,

[0050] + ,

[0051] ,

[0052] ,

[0053] + ,

[0054] ,

[0055] ,

[0056] ,

[0057] In the above formula, For alignment loss directly guided by ID, and Modal The final augmented embedding vector when the value is the modality id The mean and standard deviation obtained after parameterizing the Gaussian distribution. and The final embedding vectors of the fused modalities are respectively The mean and standard deviation obtained after parameterizing the Gaussian distribution; and The alignment losses are for image and text modalities indirectly guided by ID, respectively. and Modal The final embedding vector after enhancement when the value is the image modality v. The mean and standard deviation obtained after parameterizing the Gaussian distribution. and Modal The final embedding vector after enhancement when the value is the text modality t. The mean and standard deviation obtained after parameterizing the Gaussian distribution; and These represent the alignment losses for modality-guided image and text modalities, respectively.

[0058] Furthermore, the present invention also provides a multimodal robust recommendation system based on self-supervised learning, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning.

[0059] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the self-supervised learning-based multimodal robust recommendation method by a processor.

[0060] Furthermore, the present invention also provides a computer program product, including a computer program or instructions that are programmed or configured to execute the self-supervised learning-based multimodal robust recommendation method via a processor.

[0061] Compared with the prior art, the present invention has the following main advantages:

[0062] 1. This invention includes extracting modal features of users and items from raw data as initial embedding vectors, and constructing user-item heterogeneous graphs and item-item isomorphic graphs under each modality. By utilizing multiple modalities and user-item heterogeneous graphs and item-item isomorphic graphs under multiple modalities, it is possible to capture high-order modal information and semantic relationships between items, thereby improving the accuracy of recommendations.

[0063] 2. This invention includes performing multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vectors to extract the final user embedding vector. and final item embedding vector Aligning modal features through self-supervised tasks can better enhance the historical interaction features between each modality.

[0064] In summary, the multimodal robust recommendation method based on self-supervised learning proposed in this invention can eliminate the influence of inherent noise in multimodal data and improve the recommendation performance of the entire multimodal recommendation system. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram of the feature extraction, fusion, and alignment process in an embodiment of the present invention.

[0067] Figure 3 This is a schematic diagram of the mirror gradient optimization algorithm in step S6.2 of this embodiment of the invention.

[0068] Figure 4 This is a comparison chart of recall metrics when K=10 in an embodiment of the present invention.

[0069] Figure 5 This is a comparison chart of recall metrics when K=20 in an embodiment of the present invention.

[0070] Figure 6 This is a comparison chart of NDCG indices when K=10 in an embodiment of the present invention.

[0071] Figure 7 This is a comparison chart of NDCG indices when K=20 in an embodiment of the present invention. Detailed Implementation

[0072] like Figure 1 and Figure 2 As shown, this embodiment of the multimodal robust recommendation method based on self-supervised learning includes obtaining a recommendation list for users from the original data using a recommendation model based on self-supervised learning:

[0073] S1. Extract modal features of users and items from the original data as initial embedding vectors, and construct user-item heterogeneous graphs (UI graphs) and item-item isomorphic graphs (II graphs) for each modality.

[0074] S2, perform multimodal feature encoding on the initial embedding vectors of each modality, including: using graph convolutional networks to capture high-order features based on the initial embedding vectors and user-item heterogeneous graphs to obtain the final embedding vectors, and extracting semantic relationships between items based on item-item isomorphic graphs to enhance the final embedding vectors.

[0075] S3, perform multimodal feature fusion on the final embedded vectors after modal enhancement;

[0076] S4, perform multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector. and final item embedding vector ;

[0077] S5, based on the end-user embedding vector and final item embedding vector Calculate and sort the user's final predicted rating for the items, select the top K items as the recommendation list to recommend to the user, and output the list.

[0078] In step S1 of this embodiment, when extracting modal features of users and items from the raw data, the raw data is cleaned and the dataset is divided. Initial ID, visual, and textual modal features are then extracted using a neural network as initial embedding vectors. It should be noted that using neural networks to extract features is a basic application of neural networks. This embodiment only involves the application of this basic application and does not involve improvements; therefore, its implementation details are not described in detail here.

[0079] In step S1 of this embodiment, when constructing the user-item heterogeneous graph (UI graph) and item-item isomorphic graph (II graph) for each modality, the nodes in the user-item heterogeneous graph (UI graph) include users and items. The connection relationship between users and items is constructed based on the user's behavior association with items (such as clicking, browsing, etc.). The modal features of each modality are associated with the corresponding user nodes and item nodes respectively, forming a multimodal user-item heterogeneous graph (UI graph). In this embodiment, m represents a modality, where m takes values ​​id, v, and t to represent the ID modality, image modality, and text modality, respectively. Therefore, the user-item heterogeneous graph can be represented as: .

[0080] Similarly, the nodes in the item-item isomorphism graph (II graph) include items. The connection relationship between users and items is constructed based on the behavioral association between items (such as clicks, browsing, etc.). The modal features of each modality are associated with the corresponding item nodes to form a multimodal item-item isomorphism graph (II graph).

[0081] User-item heterogeneous diagrams (UI diagrams) A multimodal information encoding component can be used to capture the corresponding high-order modal features. In step S2 of this embodiment, when capturing high-order features based on the initial embedding vector and the user-item heterogeneous graph to obtain the final embedding vector using a graph convolutional network, the graph convolutional network used includes... The nth graph convolutional network layer, any nth graph convolutional network layer, The functional expression for the embedding vector obtained from the layer is:

[0082] ,

[0083] In the above formula, For the first The embedding vector obtained from the layer, and Let u be a single-hop neighbor of item i in the user-item heterogeneity graph. For the first The embedding vector obtained from the layer, where m is the modality, takes values ​​id, v, and t to represent the ID modality, image modality, and text modality, respectively. Each modality can be represented as follows:

[0084] ,

[0085] ,

[0086] ,

[0087] Furthermore, the functional expression for the final embedding vector obtained by capturing higher-order features is:

[0088] ,

[0089] In the above formula, Let be the final embedding vector under mode m, and we have:

[0090] ,

[0091] In the above formula, and The final embedding vectors under mode m are respectively The user embedding vector and item embedding vector can be represented in each modality as follows:

[0092] ,

[0093] ,

[0094] .

[0095] In this embodiment, step S2, which involves extracting semantic relationships between items based on the item-item isomorphism graph to enhance the final embedded vector, includes:

[0096] S2.1, calculate the similarity score between each pair of items in the item-item isomorphism graph according to the following formula:

[0097] ,

[0098] In the above formula, For modality Lowering items and items Similarity scores between them and Modal Lowering items and items The original modal features, where m is the modality, and the modality m takes the values ​​id, v, and t to represent the ID modality, image modality, and text modality, respectively;

[0099] S2.2, use the KNN algorithm to cluster the similarity scores between pairs of items, and retain the similarity scores of the K highest-scoring neighbors according to the following formula. The value is 1, and the similarity score of the other neighbors. The value of 0 can be represented as:

[0100] ,

[0101] In the above formula, For modality Lowering items Similarity scores of the K highest-scoring neighbors ;

[0102] S2.3, Calculate arbitrary modes using sparse matrix multiplication aggregation according to the following formula. The semantic relationships between the items obtained below:

[0103] ,

[0104] In the above formula, For modality The embedding vector of item i. For item i, a single-hop neighbor For modality Lowering items Embedding vectors of items For the single-hop neighbors of item i Items in the;

[0105] S2.4, enhance the final embedding vector of the corresponding modality by using the semantic relationships between items.

[0106] In step S2.4 of this embodiment, the function expression for enhancing the final embedding vector of the corresponding modality by the semantic relationships between items is as follows:

[0107] ,

[0108] In the above formula, For modality The final embedding vector after enhancement and The final embedding vectors under mode m are respectively User embedding vectors and item embedding vectors, "For connection operation, For modality The embedding vector of item i.

[0109] In step S3 of this embodiment, the function expression for multimodal feature fusion of the final embedded vectors after modal enhancement is as follows:

[0110] ,

[0111] In the above formula, This is the final embedding vector for the fused modality. These are the weighting coefficients. and Modal The final embedding vector after enhancement when the values ​​are image modality v and text modality t. And there are:

[0112] ,

[0113] ,

[0114] ,

[0115] In the above formula, and These are the final embedding vectors after enhancement under image modality v. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement under text modality t. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement in the fusion mode. User embedding vectors and item embedding vectors.

[0116] In step S4, the final embedding vectors of each modality and the fused final embedding vector are aligned with multimodal features to extract the final user embedding vector. and final item embedding vector The function expression is:

[0117] S4.1, construct the final embedding vector shown in the following formula based on the final embedding vectors of each modality and the fused final embedding vector. :

[0118] ,

[0119] In the above formula, and These are the final embedding vectors after enhancement under image modality v. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement under text modality t. User embedding vectors and item embedding vectors, and These are the final embedding vectors after enhancement in the fusion mode. User embedding vectors and item embedding vectors;

[0120] S4.2, based on the final embedding vector The final user embedding vector is generated according to the following formula. and final item embedding vector :

[0121] , .

[0122] In step S5 of this embodiment, the final user embedding vector is used... and final item embedding vector The function expression for calculating the user's final predicted rating for an item is:

[0123] ,

[0124] In the above formula, For users For items The final predicted score.

[0125] In this embodiment, before obtaining the recommendation list for users from the original data using a self-supervised learning-based recommendation model, the model is trained. The loss function used in training the recommendation model is a weighted sum of the Bayesian personalized ranking optimization loss and the alignment loss. Furthermore, a mirror gradient optimization algorithm is used in conjunction with the loss function to optimize the network parameters of the recommendation model. The expression for the alignment loss is as follows:

[0126] ,

[0127] In the above formula, For alignment loss, These are the hyperparameters used to balance the loss. For alignment loss directly guided by ID, For alignment loss indirectly guided by ID, For modality-guided alignment loss, Let be the alignment loss indirectly guided by modality, and we have:

[0128] ,

[0129] + ,

[0130] ,

[0131] ,

[0132] + ,

[0133] ,

[0134] ,

[0135] ,

[0136] In the above formula, For alignment loss directly guided by ID, and Modal The final augmented embedding vector when the value is the modality id The mean and standard deviation obtained after parameterizing the Gaussian distribution. and The final embedding vectors of the fused modalities are respectively The mean and standard deviation obtained after parameterizing the Gaussian distribution; and The alignment losses are for image and text modalities indirectly guided by ID, respectively. and Modal The final embedding vector after enhancement when the value is the image modality v. The mean and standard deviation obtained after parameterizing the Gaussian distribution. and Modal The final embedding vector after enhancement when the value is the text modality t. The mean and standard deviation obtained after parameterizing the Gaussian distribution; and These represent the alignment losses for modality-guided image and text modalities, respectively. The final embedding vector after augmentation For example, the parameterization of the Gaussian distribution can be expressed as:

[0137] ,

[0138] In the above formula, For modality The mean of the following, For modality Variance under the following conditions, fusion mode Gaussian distribution parameterization and modes The parameterization of the Gaussian distribution is the same, so it will not be repeated here.

[0139] In this embodiment, step S1 is implemented by the data preprocessing module, step S2 by the high-order feature extraction module, step S3 by the modality fusion module, step S4 by the modality alignment module, and step S5 by the prediction module. These modules constitute the recommendation model. To train the recommendation model, this embodiment uses a weighted loss function obtained by combining the Bayesian personalized ranking optimization loss (a known loss) and the alignment loss. Furthermore, based on this, a mirror gradient optimization algorithm is used in conjunction with the loss function to optimize the network parameters of the recommendation model.

[0140] In step S5 of this embodiment, the final user embedding vector is used... and final item embedding vector The function expression for calculating the user's final predicted rating for an item is:

[0141] ,

[0142] In the above formula, For users For items The final predicted score.

[0143] As an optional implementation, the recommendation model used in this embodiment employs Bayesian personalized ranking optimization loss as the basic optimization function, combined with alignment loss from a self-supervised task to update the representations of users and items; and based on Bayesian personalized ranking optimization loss, a mirror gradient optimization algorithm is used to update the model parameters of the recommendation model to improve its robustness. Figure 3 As shown, in this embodiment, the network parameters of the recommendation model are optimized using the mirror gradient optimization algorithm combined with a loss function, including:

[0144] S101, Initialize the iteration interval β and the preset threshold T, and initialize the number of iterations t to 1;

[0145] S102, Determine if the iteration number t is less than or equal to the preset threshold T. If true, proceed to step S103; then proceed to step S107.

[0146] S103, determine whether the result of the modulo operation of the iteration number t with respect to the iteration interval β is 0. If it is true, jump to step S103; otherwise, jump to step S106.

[0147] S104, Update the model parameters of the recommendation model according to the following formula:

[0148] ,

[0149] ,

[0150] In the above formula, and Let be the model parameters for the t-th and t-1th iterations, respectively. As an intermediate variable, For the gradient of the parameters, and For hyperparameters, For learning rate, The loss function is the weighted sum of the Bayesian personalized ranking optimization loss and the alignment loss.

[0151] S105, increment the iteration number t by 1 (t=t+1), and jump to step S102;

[0152] S106, Update the model parameters of the recommendation model according to the following formula:

[0153] ,

[0154] In the above formula, and Let be the model parameters for the t-th and t-1th iterations, respectively. For the gradient of the parameters, For learning rate, Given the loss function; proceed to step S105;

[0155] S107, the model parameters for the t-th iteration. The optimal model parameters are output as the recommendation model.

[0156] To verify the self-supervised learning-based multimodal robust recommendation method in this embodiment, the dataset used in the experiment is the publicly available Amazon-sports dataset. During the experiment, K items were recommended to each user (K ​​was set to values ​​of 10 and 20 in the experiment). Then, two metrics were calculated: recall and Normalized Discounted Cumulative Gain (NDCG). NDCG@10 and NDCG@20 represent the NDCG values ​​when K is 10 and 20, respectively. Existing methods were compared with the classic multimodal recommendation method VBPR and two new multimodal recommendation methods, MMGCN and BM3. The final results are as follows: Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, SSLRec is the method used in this embodiment. See also Figure 4 , Figure 5 , Figure 6 and Figure 7It can be seen that the method in this embodiment (SSLRec) outperforms the three comparative methods (VBPR, MMGCN and BM3) in both recall and NDCG metrics.

[0157] In summary, this embodiment of the multimodal robust recommendation method based on self-supervised learning first processes the user-item interaction data to obtain the corresponding initial embedding vectors, and constructs user-item heterogeneous graphs (UI graphs) and item-item isomorphic graphs (II graphs) for each modality. Then, a graph convolutional neural network is used to extract specific modal features through the UI graph and II graph to capture the semantic relationships between items. Finally, the visual and textual feature embedding vectors are fused to obtain a new feature vector. A multi-level self-supervised task is used to align the various modalities, thereby enhancing the historical interaction features between each modality; then, the user feature embedding vector is extracted from the final embedding vector. With item feature embedding vector The inner product of the two is used as the user's predicted score for the item, and the items are sorted. The top K items are selected as the set of the recommendation list. Finally, the representations of the user and the item are updated by combining the Bayesian personalized ranking loss with the alignment loss from the self-supervised task. A mirror gradient optimization algorithm is used to update the model parameters and optimize the model. This embodiment of the multimodal robust recommendation method based on self-supervised learning can utilize multimodal information to improve the performance of the recommendation system. It can improve the accuracy of recommending items to users based on visual and textual modal information and user-item interaction information in the original data, and can be widely applied to various multimodal recommendation scenarios. On the one hand, this embodiment of the method aligns the features between different modalities by designing a reasonable self-supervised task and using the stability of ID features to guide the alignment between different modalities. On the other hand, it uses a mirror gradient optimization algorithm to optimize the recommendation model to eliminate the influence of inherent noise in multimodal data and improve the recommendation effect of the entire multimodal recommendation system.

[0158] Furthermore, this embodiment also provides a multimodal robust recommendation system based on self-supervised learning, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning.

[0159] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program or instructions that are programmed or configured to execute the self-supervised learning-based multimodal robust recommendation method by a processor.

[0160] Furthermore, this embodiment also provides a computer program product, including a computer program or instructions that are programmed or configured to execute the self-supervised learning-based multimodal robust recommendation method via a processor.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-modal robust recommendation method based on self-supervised learning, characterized in that, The method comprises obtaining a recommendation list recommended to a user from original data by using a recommendation model based on self-supervised learning: S1, extracting modal features of each modality of users and items from the original data as initial embedding vectors, and constructing a user-item heterogeneous graph under each modality and an item-item homogeneous graph; S2, performing multi-modal feature coding on the initial embedding vectors of each modality, comprising: capturing high-order features by using a graph convolution network based on the initial embedding vectors and the user-item heterogeneous graph to obtain final embedding vectors, and enhancing the final embedding vectors based on semantic relationships between items extracted from the item-item homogeneous graph; S3, performing multi-modal feature fusion on the final embedding vectors of each modality after enhancement; S4, performing multi-modal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract a final user embedding vector and a final item embedding vector ; S5, according to the final user embedding vector and the final item embedding vector Calculate the final predicted score of the user to the item and sort, select the specified top K items as the recommended list recommended to the user and output; In step S2, when the final embedding vector is obtained based on the initial embedding vector and the user-item heterogeneous graph by using the graph convolution network, the graph convolution network comprises graph convolution network layers, and any The function expression of the embedding vector obtained by the i-th layer is: ， In the above formula, is the embedding vector obtained by the first layer, is the embedding vector obtained by the first layer, and is the single-hop neighbor of the user u and the item i in the user-item heterogeneous graph, is the embedding vector obtained by the first layer, is the embedding vector obtained by the first layer, m is a modality, the values of the modality m are id, v and t respectively represent the ID modality, the image modality and the text modality; and the functional expression of the final embedding vector obtained by capturing high-order features is: ， In the above formula, is the final embedding vector under modality m, and has: ， In the above formula, and are the user embedding vector and the item embedding vector of the final embedding vector under the mode m, respectively; The step S2 of enhancing the final embedding vectors based on the semantic relationships between items in the item-item homogeneous graph comprises: S2.1, calculating the similarity score between each pair of items in the item-item homogeneous graph according to the following formula: ， In the above formula, is the modality The lower item And the similarity score between the items And is the modality The lower item And the item The original modality feature, m is the modality, and the modality m takes values id, v and t respectively represent the ID modality, image modality and text modality.​ S2.2, cluster the similarity scores between each pair of items using the KNN algorithm and retain the similarity scores of the K highest scoring neighbors according to the following formula where the similarity score of the Kth highest scoring neighbor has a value of 1 and the similarity scores of the remaining neighbors have a value of 0. where the similarity score of the Kth highest scoring neighbor has a value of 1 and the similarity scores of the remaining neighbors have a value of 0. S2.3, aggregate the computation of any modality using sparse matrix multiplication according to the following formula The semantic relationships between the resulting items are: ， In the above formula, is the modality The embedding vector of the item i, is the one-hop neighbor of item i, is the modality The embedding vector of the item The embedding vector of the item is the item in the one-hop neighbor of item i; S2.4, enhancing the final embedding vectors of the corresponding modality based on the semantic relationships between items; The function expression of the step S2.4 of enhancing the final embedding vectors of the corresponding modality based on the semantic relationships between items is: ， In the above formula, is the modality The final enhanced embedding vector, and is the final embedding vector of the user embedding vector and the item embedding vector, is the connection operation, is the modality of the item i. 2.The multi-modal robust recommendation method based on self-supervised learning according to claim 1, wherein, The function expression of the step S3 of performing multi-modal feature fusion on the final embedding vectors of each modality after enhancement is: ， In the above formula, is the final embedding vector of the fusion modal, is the weight coefficient, and are the modal is the enhanced final embedding vector when the value of the image modal v and the text modal t is , and has: ， ， ， In the above formulae, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the image modality v, respectively, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the text modality t, respectively, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the fusion modality, respectively. 3.The multi-modal robust recommendation method based on self-supervised learning according to claim 2, wherein, In step S4, the final embedding vectors of each modality, the fused final embedding vector are subjected to multi-modal feature alignment to extract the final user embedding vector and the final item embedding vector The functional expression is as follows: S4.1, constructing a final embedding vector shown in the following formula according to the final embedding vectors of each modality and the fused final embedding vector : ， In the above formulae, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the image modality v, respectively, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the text modality t, respectively, and are the user embedding vector and the item embedding vector of the final embedding vector enhanced under the fusion modality, respectively. S4.2, based on the final embedding vectors The final user embedding vectors are generated according to the following formula and the final item embedding vectors : ， 。 4.The self-supervised learning based multi-modal robust recommendation method of claim 1, wherein, The final user embedding vector in step S5 and the final item embedding vector The function expression for calculating the final predicted rating of the user for the item is: ， In the above formula, for the user to the item the final predicted rating. 5.The self-supervised learning based multi-modal robust recommendation method of claim 3, wherein, The method further comprises training the recommendation model before obtaining the recommendation list recommended to the user from the original data by using the recommendation model based on self-supervised learning, and the loss function used for training the recommendation model is a loss function obtained by weighting a Bayesian personalized ranking optimization loss and an alignment loss, and a mirror gradient optimization algorithm is used in combination with the loss function to optimize the network parameters of the recommendation model; and the calculation function expression of the alignment loss is: ， In the above formula, is the alignment loss, is a hyperparameter for the balancing loss, is the alignment loss directly guided by the ID, is the alignment loss indirectly guided by the ID, is the alignment loss guided by the modality, is the alignment loss indirectly guided by the modality, and has: ， + ， ， ， + ， ， ， ， In the above formula, is the alignment loss guided by ID directly, and are the final embedding vectors of modal is the enhanced final embedding vector when the value of ID modal id is are the mean and standard deviation obtained after Gaussian distribution parameterization, and are the final embedding vectors of fusion modal are the mean and standard deviation obtained after Gaussian distribution parameterization. and are the alignment losses of image and text modal guided by ID indirectly, and are the final embedding vectors of modal is the enhanced final embedding vector when the value of image modal v is are the mean and standard deviation obtained after Gaussian distribution parameterization, and are the final embedding vectors of modal is the enhanced final embedding vector when the value of text modal t is are the mean and standard deviation obtained after Gaussian distribution parameterization. and are the alignment losses of image and text modal guided by modal.

6. A multi-modal robust recommendation system based on self-supervised learning comprising microprocessors and memories interconnected, characterized in that, The microprocessor is programmed or configured to perform the multi-modal robust recommendation method based on self-supervised learning according to any one of claims 1-5.

7. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the multi-modal robust recommendation method based on self-supervised learning according to any one of claims 1-5 by the processor.

8. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to perform the multi-modal robust recommendation method based on self-supervised learning according to any one of claims 1-5 by the processor.

Images