Multi-modal robust recommendation method and system based on self-supervised learning

Abstract

The invention discloses a multi-modal robust recommendation method and system based on self-supervised learning, and the method comprises the steps: extracting the modal features of each modal of a user and an article from original data as initial embedded vectors, and constructing a user-article heterogeneous graph and an article-article isomerism graph under each modal; performing multi-modal feature coding on the initial embedded vector of each modal, and performing multi-modal feature fusion on the enhanced final embedded vector of each modal; and performing multi-modal feature alignment to extract a final user embedded vector # imgabs0 # and a final article embedded vector # imgabs1 #, calculating to obtain a final prediction score of the user on the articles, sorting the final prediction score, selecting specified first K articles as a recommendation list recommended to the user, and outputting the recommendation list. The invention aims to eliminate the influence of inherent noise in the multi-modal data and improve the recommendation effect of the whole multi-modal recommendation system.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and recommendation systems, and specifically relates to a multimodal robust recommendation method and system based on self-supervised learning. Background Art

[0002] Early recommendation systems generally used collaborative filtering methods to model user preferences based on the interaction records between users and items. By calculating the similarity between users (items), similar items are recommended to users or items are recommended to similar users. This method considers relatively simple information, only considering the user's interactive behavior, ignoring the multimodal information of items, so it has major defects. With the development of multimodal technology, integrating multimodal information into recommendation systems has become a common means to improve the performance of recommendation systems. Multimodal recommendation systems model user preferences based on behavioral information and multimodal information of items. The commonly used method is multimodal representation learning, using historical interactions to create ID embeddings for users and items, and then treating multimodal data as auxiliary information for items to enrich their ID embeddings. Although the introduction of multimodal information has helped recommendation systems improve recommendation performance, there are still some problems in multimodal recommendation systems. For example, the inherent problems of recommendation systems: data sparsity and cold start problems. In addition to these inherent problems, multimodal recommendation systems also have problems with how to fuse and align various modal features, which to a certain extent affect the performance of recommendation systems. At the same time, due to the introduction of multimodal information, the risk of multimodal noise information affecting recommendation performance increases, so improving system robustness is also an aspect that needs to be studied. In summary, in actual multimodal recommendation scenarios, there is a large amount of noise information, including the inherent noise of multimodal information and the noise generated by the modal fusion stage and modal alignment. These noises will affect the overall performance of the multimodal recommendation system, resulting in unsatisfactory recommendation results. How to eliminate the inherent noise of multimodal information and the noise generated by the modal fusion stage and modal alignment, and improve the system robustness of the multimodal recommendation system, has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multimodal robust recommendation method and system based on self-supervised learning are provided. The present invention aims to eliminate the influence of inherent noise in multimodal data and improve the recommendation effect of the entire multimodal recommendation system.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A multimodal robust recommendation method based on self-supervised learning includes obtaining a recommendation list recommended to a user from raw data using a recommendation model based on self-supervised learning: S1, extract the modal features of each modality of users and items from the original data as the initial embedding vector, and construct the user-item heterogeneous graph and item-item isomorphic graph under each modality; S2, multimodal feature encoding is performed on the initial embedding vectors of each modality, including: using a graph convolutional network to capture high-order features based on the initial embedding vectors and the user-item heterogeneous graph to obtain the final embedding vector, and extracting the semantic relationship between items based on the item-item homogeneous graph to enhance the final embedding vector; S3, multi-modal feature fusion of the final embedding vector after each modality enhancement; S4: Perform multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector and the final item embedding vector ; S5, based on the end-user embedding vector and the final item embedding vector The final predicted scores of the user for the items are calculated and sorted, and the top K items are selected as the recommendation list recommended to the user and output.

[0005] Optionally, in step S2, when a graph convolutional network is used to capture high-order features based on the initial embedding vector and the user-item heterogeneous graph to obtain a final embedding vector, the graph convolutional network used includes Graph convolutional network layers, any The function expression of the embedded vector obtained by the layer is: , In the above formula, For the The embedding vector obtained by the layer, and is the one-hop neighbor between user u and item i in the user-item heterogeneous graph, For the The embedding vector obtained by the layer, m is the modality, and the modality m takes the values ​​of id, v and t to represent ID modality, image modality and text modality respectively; and the function expression of the final embedding vector obtained by capturing high-order features is: , In the above formula, is the final embedding vector under mode m, and: , In the above formula, and They are the final embedding vectors under mode m. User embedding vector and item embedding vector of Optionally, in step S2, extracting semantic relationships between items based on the item-item isomorphic graph to enhance the final embedding vector includes: S2.1, calculate the similarity scores between each pair of items in the item-item isomorphism graph according to the following formula: , In the above formula, For modal Next item and items The similarity score between and Mode Next item and items The original modality feature, m is the modality, and the modality m takes the values ​​of id, v and t, which represent ID modality, image modality and text modality respectively; S2.2, use the KNN algorithm to cluster the similarity scores between each pair of items, and retain the similarity scores of the K neighbors with the highest scores according to the following formula The value is 1, and the similarity scores of the other neighbors The value of is 0; S2.3, sparse matrix multiplication is used to aggregate and calculate any mode according to the following formula The semantic relationship between the items is obtained as follows: , In the above formula, For modal The embedding vector of item i is: is the one-hop neighbor of item i, For modal Next item The embedding vector of the item is the one-hop neighbor of item i Items in S2.4, the semantic relationship between items is used to enhance the final embedding vector of the corresponding modality.

[0006] Optionally, in step S2.4, the function expression of enhancing the final embedding vector of the corresponding modality by using the semantic relationship between items is: , In the above formula, For modal The final embedding vector after enhancement is and They are the final embedding vectors under mode m. The user embedding vector and item embedding vector of " is the connection operation, For modal The embedding vector of item i.

[0007] Optionally, in step S3, the function expression for performing multimodal feature fusion on the final embedding vector after each modality enhancement is: , In the above formula, is the final embedding vector of the fused modality, is the weight coefficient, and Mode The final embedding vector after enhancement when the values ​​are image modality v and text modality t , and there are: , , , In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. The user embedding vector and item embedding vector of .

[0008] Optionally, in step S4, the final embedding vectors of each modality and the fused final embedding vector are subjected to multimodal feature alignment to extract the final user embedding vector and the final item embedding vector The function expression is: S4.1, construct the final embedding vector shown in the following formula based on the final embedding vector of each modality and the fused final embedding vector : , In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. User embedding vector and item embedding vector of S4.2, based on the final embedding vector The final user embedding vector is generated according to the following formula and the final item embedding vector : , .

[0009] Optionally, in step S5, according to the final user embedding vector and the final item embedding vector The function expression for calculating the user's final predicted score for the item is: , In the above formula, For users For items The final prediction score.

[0010] Optionally, before obtaining the recommendation list recommended to the user from the original data using the recommendation model based on self-supervised learning, the method further includes training the recommendation model, and the loss function used in the training recommendation model is a loss function obtained by weighting the Bayesian personalized ranking optimization loss and the alignment loss, and the network parameters of the recommendation model are optimized by using the mirror gradient optimization algorithm combined with the loss function; the calculation function expression of the alignment loss is: , In the above formula, is the alignment loss, is the hyperparameter used to balance the loss, is the alignment loss guided directly by ID, is the alignment loss indirectly guided by ID, is the modality-guided alignment loss, is the alignment loss indirectly guided by the modality, and: , + , , , + , , , , In the above formula, is the alignment loss guided directly by ID, and Mode The enhanced final embedding vector when the value is ID modality id The mean and standard deviation obtained after parameterization of the Gaussian distribution, and are the final embedding vectors of the fused modalities respectively. The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the alignment losses for image and text modalities indirectly guided by ID, and Mode The final embedding vector after enhancement when the value is image modality v The mean and standard deviation obtained after parameterization of the Gaussian distribution, and Mode The final embedding vector after enhancement when the value is text mode t The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the modality-guided alignment losses for image and text modalities, respectively.

[0011] In addition, the present invention also provides a multimodal robust recommendation system based on self-supervised learning, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning.

[0012] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning through a processor.

[0013] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning through a processor.

[0014] Compared with the prior art, the present invention mainly has the following advantages: 1. The present invention includes extracting modal features of each modality of users and items from the original data as the initial embedding vector, and constructing a user-item heterogeneous graph under each modality, as well as an item-item isomorphic graph. By using multiple modalities, and user-item heterogeneous graphs under multiple modalities, as well as item-item isomorphic graphs, the semantic connection between high-order modal information and items can be captured, which can improve the accuracy of recommendations.

[0015] 2. The present invention includes performing multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector and the final item embedding vector , aligning modal features through self-supervision tasks can better enhance the historical interaction features between each modality.

[0016] In summary, the multimodal robust recommendation method based on self-supervised learning in the present invention can eliminate the influence of inherent noise in multimodal data and improve the recommendation effect of the entire multimodal recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0018] Figure 2 Schematic diagram of the feature extraction, fusion and alignment process in an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the mirror gradient optimization algorithm flow in step S6.2 in an embodiment of the present invention.

[0020] Figure 4 This is a comparison chart of recall rate indicators when K=10 in an embodiment of the present invention.

[0021] Figure 5 This is a comparison chart of recall rate indicators when K=20 in an embodiment of the present invention.

[0022] Figure 6 This is a comparison chart of NDCG indicators when K=10 in an embodiment of the present invention.

[0023] Figure 7 This is a comparison chart of NDCG indicators when K=20 in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 and Figure 2 As shown, the multimodal robust recommendation method based on self-supervised learning in this embodiment includes obtaining a recommendation list recommended to the user from the original data using a recommendation model based on self-supervised learning: S1, extract the modal features of each modality of users and items from the original data as the initial embedding vector, and construct the user-item heterogeneous graph (UI graph) and item-item isomorphic graph (II graph) under each modality; S2, multimodal feature encoding is performed on the initial embedding vectors of each modality, including: using a graph convolutional network to capture high-order features based on the initial embedding vectors and the user-item heterogeneous graph to obtain the final embedding vector, and extracting the semantic relationship between items based on the item-item homogeneous graph to enhance the final embedding vector; S3, multi-modal feature fusion of the final embedding vector after each modality enhancement; S4: Perform multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector and the final item embedding vector ; S5, based on the end-user embedding vector and the final item embedding vector The final predicted scores of the user for the items are calculated and sorted, and the top K items are selected as the recommendation list recommended to the user and output.

[0025] In step S1 of this embodiment, when extracting modal features of each modality of users and items from the original data, specifically, the original data is cleaned and the data set is divided, and the initial ID, visual and text modal features are extracted using a neural network as the initial embedding vector. It should be noted that using a neural network to extract features is the basic purpose of a neural network. This embodiment only involves the application of this basic purpose without involving improvements, so its implementation details are not described in detail here.

[0026] In step S1 of this embodiment, when constructing the user-item heterogeneous graph (UI graph) and the item-item isomorphic graph (II graph) under each modality, the nodes in the user-item heterogeneous graph (UI graph) include users and items. Based on the user's behavioral association with the item (such as click, browse, etc.), the connection relationship between the user and the item is constructed, and the modal features of each modality are respectively associated with the corresponding user node and item node to form a multi-modal user-item heterogeneous graph (UI graph). In this embodiment, m is the modality, and the modality m takes the values ​​of id, v and t, which represent the ID modality, image modality and text modality respectively. Therefore, the user-item heterogeneous graph can be expressed as: .

[0027] Similarly, the nodes in the item-item isomorphism graph (Figure II) include items. The connection relationship between users and items is constructed based on the behavioral association between items (such as clicks, browsing, etc.), and the modal features of each modality are associated with the corresponding item nodes, forming a multimodal item-item isomorphism graph (Figure II).

[0028] For user-item heterogeneous graph (UI graph) , a multimodal information encoding component can be used to capture the corresponding high-order modal features. In step S2 of this embodiment, when the graph convolution network is used to capture high-order features based on the initial embedding vector and the user-item heterogeneous graph to obtain the final embedding vector, the graph convolution network used includes Graph convolutional network layers, any The function expression of the embedded vector obtained by the layer is: , In the above formula, For the The embedding vector obtained by the layer, and is the one-hop neighbor between user u and item i in the user-item heterogeneous graph, For the The embedding vector obtained by the layer, m is the modality, and the modality m takes the values ​​of id, v and t, which represent the ID modality, image modality and text modality respectively. Each modality can be expressed as: , , , And the function expression of the final embedding vector obtained by capturing high-order features is: , In the above formula, is the final embedding vector under mode m, and: , In the above formula, and They are the final embedding vectors under mode m. The user embedding vector and item embedding vector of can be expressed as follows in each mode: , , .

[0029] In step S2 of this embodiment, extracting the semantic relationship between items based on the item-item isomorphic graph to enhance the final embedding vector includes: S2.1, calculate the similarity scores between each pair of items in the item-item isomorphism graph according to the following formula: , In the above formula, For modal Next item and items The similarity score between and Mode Next item and items The original modality feature, m is the modality, and the modality m takes the values ​​of id, v and t, which represent ID modality, image modality and text modality respectively; S2.2, use the KNN algorithm to cluster the similarity scores between each pair of items, and retain the similarity scores of the K neighbors with the highest scores according to the following formula The value is 1, and the similarity scores of the other neighbors The value of is 0, which can be expressed as: , In the above formula, For modal Next item The similarity scores of the K highest scoring neighbors ; S2.3, sparse matrix multiplication is used to aggregate and calculate any mode according to the following formula The semantic relationship between the items is obtained as follows: , In the above formula, For modal The embedding vector of item i is: is the one-hop neighbor of item i, For modal Next item The embedding vector of the item is the one-hop neighbor of item i Items in S2.4, the semantic relationship between items is used to enhance the final embedding vector of the corresponding modality.

[0030] In step S2.4 of this embodiment, the function expression of enhancing the final embedding vector of the corresponding modality by using the semantic relationship between items is: , In the above formula, For modal The final embedding vector after enhancement is and They are the final embedding vectors under mode m. The user embedding vector and item embedding vector of " is the connection operation, For modal The embedding vector of item i.

[0031] In step S3 of this embodiment, the function expression for multimodal feature fusion of the final embedding vector after each modality enhancement is: , In the above formula, is the final embedding vector of the fused modality, is the weight coefficient, and Mode The final embedding vector after enhancement when the values ​​are image modality v and text modality t , and there are: , , , In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. The user embedding vector and item embedding vector of .

[0032] In step S4, the final embedding vectors of each modality and the fused final embedding vector are subjected to multimodal feature alignment to extract the final user embedding vector and the final item embedding vector The function expression is: S4.1, construct the final embedding vector shown in the following formula based on the final embedding vector of each modality and the fused final embedding vector : , In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. User embedding vector and item embedding vector of S4.2, based on the final embedding vector The final user embedding vector is generated according to the following formula and the final item embedding vector : , .

[0033] In step S5 of this embodiment, the final user embedding vector and the final item embedding vector The function expression for calculating the user's final predicted score for the item is: , In the above formula, For users For items The final prediction score.

[0034] In this embodiment, before obtaining the recommendation list recommended to the user from the original data using the recommendation model based on self-supervised learning, the recommendation model is also trained, and the loss function used in the training recommendation model is a loss function obtained by weighting the Bayesian personalized ranking optimization loss and the alignment loss, and the network parameters of the recommendation model are optimized by combining the mirror gradient optimization algorithm with the loss function; the calculation function expression of the alignment loss is: , In the above formula, is the alignment loss, is the hyperparameter used to balance the loss, is the alignment loss guided directly by ID, is the alignment loss indirectly guided by ID, is the modality-guided alignment loss, is the alignment loss indirectly guided by the modality, and: , + , , , + , , , , In the above formula, is the alignment loss guided directly by ID, and Mode The enhanced final embedding vector when the value is ID modality id The mean and standard deviation obtained after parameterization of the Gaussian distribution, and are the final embedding vectors of the fused modalities respectively. The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the alignment losses for image and text modalities indirectly guided by ID, and Mode The final embedding vector after enhancement when the value is image modality v The mean and standard deviation obtained after parameterization of the Gaussian distribution, and Mode The final embedding vector after enhancement when the value is text mode t The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the alignment losses for the modality-guided image and text modalities, respectively. The final embedding vector after enhancement For example, the parameterization of Gaussian distribution can be expressed as: , In the above formula, For modal The mean value under For modal Variance under , fusion mode Gaussian distribution parameterization and modes The Gaussian distribution parameterization of is the same, so it will not be repeated here.

[0035] In this embodiment, step S1 is implemented by a data preprocessing module, step S2 is a high-order feature extraction module, step S3 is a modal fusion module, step S4 is a modal alignment module, and step S5 is a prediction module. The above modules constitute a recommendation model. In order to train the recommendation model, the loss function used in this embodiment is a weighted loss function obtained by adding the Bayesian personalized ranking optimization loss (known loss) and the alignment loss. On the basis of the above, the mirror gradient optimization algorithm is combined with the loss function to optimize the network parameters of the recommendation model. In step S5 of this embodiment, the final user embedding vector and the final item embedding vector The function expression for calculating the user's final predicted score for the item is: , In the above formula, For users For items The final prediction score.

[0036] As an optional implementation, for the recommendation model adopted by the method of this embodiment, the Bayesian personalized ranking optimization loss is used as the basic optimization function, combined with the alignment loss in the self-supervised task to update the representation of users and items; and on the basis of the Bayesian personalized ranking optimization loss, the mirror gradient optimization algorithm is used to update the model parameters of the recommendation model to improve the robustness of the recommendation model. Figure 3 As shown, in this embodiment, the network parameters of the recommendation model are optimized by using the mirror gradient optimization algorithm combined with the loss function, including: S101, initializing the iteration interval β and the preset threshold T, and initializing the number of iterations t to 1; S102, determine whether the number of iterations t is less than or equal to a preset threshold T, if true, jump to step S103; jump to step S107; S103, determining whether the result of the modulo operation (mod) of the number of iterations t on the iteration interval β is 0 is true, if so, jump to step S103; otherwise, jump to step S106; S104, updating the model parameters of the recommendation model according to the following formula: , , In the above formula, and are the model parameters for the tth and t-1th times, is the intermediate variable, is the parameter gradient, and is a hyperparameter, is the learning rate, is the loss function, i.e., the loss function obtained by weighting the Bayesian personalized ranking optimization loss and the alignment loss; S105, increase the number of iterations t by 1 (t=t+1), and jump to step S102; S106, updating the model parameters of the recommendation model according to the following formula: , In the above formula, and are the model parameters for the tth and t-1th times, is the parameter gradient, is the learning rate, is the loss function; jump to step S105; S107, the t-th model parameter The optimal model parameters are output as the recommended model.

[0037] In order to verify the multimodal robust recommendation method based on self-supervised learning in this embodiment, the dataset used in the experiment is a public dataset Amazon-sports. During the experiment, K items are recommended to each user (the value of K is 10 and 20 in the experiment), and then the recall rate and the normalized discounted cumulative gain NDCG (Normalized Discounted Cumulative Gain) are calculated. NDCG@10 and NDCG@20 are the normalized discounted cumulative gain NDCG when the value of K is 10 and 20 respectively. The existing methods compared include the classic multimodal recommendation method VBPR and two new multimodal recommendation methods MMGCN and BM3. The final results are as follows: Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, SSLRec is the method of this embodiment. Figure 4 , Figure 5 , Figure 6 and Figure 7 It can be seen that the method in this embodiment (SSLRec) is superior to the three compared methods (VBPR, MMGCN and BM3) in both recall rate and NDCG indicators.

[0038] In summary, the multimodal robust recommendation method based on self-supervised learning in this embodiment first processes the interaction data between users and items to obtain the corresponding initial embedding vectors, and constructs user-item heterogeneous graphs (UI graphs) and item-item isomorphic graphs (II graphs) under each modality; then, a graph convolutional neural network is used to extract specific modal features through the UI graph and the II graph to capture the semantic connection between items; then, the feature embedding vectors of vision and text are fused to obtain a new feature vector , use a multi-level self-supervised task to achieve alignment between the various modalities to enhance the historical interaction features between each modality; then separate the user feature embedding vector from the final embedding vector and item feature embedding vector , use the inner product of the two as the predicted score of the user for the item and sort them, select the first K items as the set of recommendation lists; finally, update the representation of users and items through the Bayesian personalized ranking loss combined with the alignment loss in the self-supervised task, and use a mirror gradient optimization algorithm to update the parameters of the model to optimize the model. The multimodal robust recommendation method based on self-supervised learning in this embodiment can use multimodal information to improve the performance of the recommendation system, and realize the accuracy of recommending items to users based on the visual and textual modal information in the original data and the interactive information between users and items. It can be widely used in various multimodal recommendation scenarios. On the one hand, the method of this embodiment aligns the features between different modalities by designing reasonable self-supervised tasks, and uses the stability of ID features to guide the alignment between different modalities. On the other hand, it uses a mirror gradient optimization algorithm to optimize the recommendation model to eliminate the influence of inherent noise in multimodal data and improve the recommendation effect of the entire multimodal recommendation system.

[0039] In addition, this embodiment also provides a multimodal robust recommendation system based on self-supervised learning, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning.

[0040] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning through a processor.

[0041] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning through a processor.

[0042] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0043] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal robust recommendation method based on self-supervised learning, characterized in that: Including using a recommendation model based on self-supervised learning to obtain a recommendation list recommended to the user from the original data: S1, extract the modal features of each modality of users and items from the original data as the initial embedding vector, and construct the user-item heterogeneous graph and item-item isomorphic graph under each modality; S2, multimodal feature encoding for the initial embedding vector of each modality, including: using a graph convolutional network to capture high-order features based on the initial embedding vector and the user-item heterogeneous graph to obtain the final embedding vector, and extracting the semantic relationship between items based on the item-item homogeneous graph to enhance the final embedding vector; S3, multi-modal feature fusion of the final embedding vector after each modality enhancement; S4: Perform multimodal feature alignment on the final embedding vectors of each modality and the fused final embedding vector to extract the final user embedding vector and the final item embedding vector ; S5, based on the end user embedding vector and the final item embedding vector The final predicted scores of the user for the items are calculated and sorted, and the top K items are selected as the recommendation list recommended to the user and output.

2. The multimodal robust recommendation method based on self-supervised learning according to claim 1, characterized in that: In step S2, when the graph convolutional network is used to capture high-order features based on the initial embedding vector and the user-item heterogeneous graph to obtain the final embedding vector, the graph convolutional network used includes Graph convolutional network layers, any The function expression of the embedded vector obtained by the layer is: ， In the above formula, For the The embedding vector obtained by the layer, and is the one-hop neighbor between user u and item i in the user-item heterogeneous graph, For the The embedding vector obtained by the layer, m is the modality, and the modality m takes the values ​​of id, v and t to represent ID modality, image modality and text modality respectively; and the function expression of the final embedding vector obtained by capturing high-order features is: ， In the above formula, is the final embedding vector under mode m, and: ， In the above formula, and They are the final embedding vectors under mode m. User embedding vector and item embedding vector of In step S2, the semantic relationship between items is extracted based on the item-item isomorphic graph to enhance the final embedding vector, including: S2.1, calculate the similarity scores between each pair of items in the item-item isomorphism graph according to the following formula: ， In the above formula, For modal Next item and items The similarity score between and Mode Next item and items The original modality feature, m is the modality, and the modality m takes the values ​​of id, v and t, which represent ID modality, image modality and text modality respectively; S2.2, use the KNN algorithm to cluster the similarity scores between each pair of items, and retain the similarity scores of the K neighbors with the highest scores according to the following formula The value is 1, and the similarity scores of the other neighbors The value of is 0; S2.3, sparse matrix multiplication is used to aggregate and calculate any mode according to the following formula The semantic relationship between the items is obtained as follows: ， In the above formula, For modal The embedding vector of item i is: is the one-hop neighbor of item i, For modal Next item The embedding vector of the item is the one-hop neighbor of item i Items in S2.4, the semantic relationship between items is used to enhance the final embedding vector of the corresponding modality.

3. The multimodal robust recommendation method based on self-supervised learning according to claim 2, characterized in that: In step S2.4, the function expression of the final embedding vector of the corresponding modality is enhanced by the semantic relationship between items: ， In the above formula, For modal The final embedding vector after enhancement is and They are the final embedding vectors under mode m. The user embedding vector and item embedding vector of " " is the connection operation, For modal The embedding vector of item i.

4. The multimodal robust recommendation method based on self-supervised learning according to claim 3, characterized in that: The function expression for multimodal feature fusion of the final embedding vector after each modality enhancement in step S3 is: ， In the above formula, is the final embedding vector of the fused modality, is the weight coefficient, and Mode The final embedding vector after enhancement when the values ​​are image modality v and text modality t , and there are: ， ， ， In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. The user embedding vector and item embedding vector of .

5. The multimodal robust recommendation method based on self-supervised learning according to claim 4, characterized in that: In step S4, the final embedding vectors of each modality and the fused final embedding vector are subjected to multimodal feature alignment to extract the final user embedding vector and the final item embedding vector The function expression is: S4.1, construct the final embedding vector shown in the following formula based on the final embedding vector of each modality and the fused final embedding vector : ， In the above formula, and They are the final embedded vectors after enhancement under image modality v. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under text modality t. The user embedding vector and item embedding vector of and They are the final embedded vectors after enhancement under fusion mode. User embedding vector and item embedding vector of S4.2, based on the final embedding vector The final user embedding vector is generated according to the following formula and the final item embedding vector : ， 。 6. The multimodal robust recommendation method based on self-supervised learning according to claim 1, characterized in that: In step S5, the final user embedding vector and the final item embedding vector The function expression for calculating the user's final predicted score for the item is: ， In the above formula, For users For items The final prediction score.

7. The multimodal robust recommendation method based on self-supervised learning according to claim 5, characterized in that: The method of using the recommendation model based on self-supervised learning to obtain the recommendation list recommended to the user from the original data also includes training the recommendation model, and the loss function used in the training recommendation model is a loss function obtained by weighting the Bayesian personalized ranking optimization loss and the alignment loss, and the network parameters of the recommendation model are optimized by using the mirror gradient optimization algorithm combined with the loss function; the calculation function expression of the alignment loss is: ， In the above formula, is the alignment loss, is the hyperparameter used to balance the loss, is the alignment loss guided directly by ID, is the alignment loss indirectly guided by ID, is the modality-guided alignment loss, is the alignment loss indirectly guided by the modality, and: ， + ， ， ， + ， ， ， ， In the above formula, is the alignment loss guided directly by ID, and Mode The enhanced final embedding vector when the value is ID modality id The mean and standard deviation obtained after parameterization of the Gaussian distribution, and are the final embedding vectors of the fused modalities respectively. The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the alignment losses for image and text modalities indirectly guided by ID, and Mode The final embedding vector after enhancement when the value is image modality v The mean and standard deviation obtained after parameterization of the Gaussian distribution, and Mode The final embedding vector after enhancement when the value is text mode t The mean and standard deviation obtained after parameterization of the Gaussian distribution; and are the modality-guided alignment losses for image and text modalities, respectively.

8. A multimodal robust recommendation system based on self-supervised learning, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the multimodal robust recommendation method based on self-supervised learning described in any one of claims 1 to 7 through a processor.

Images