A multimodal recommendation method and device based on self-supervision to enhance user preference

Through self-supervising methods, multimodal data is extracted and integrated, the user-project map is built and the recommendation system is optimized, which solves the problems of insufficient user preference understanding and loss of modal information, and improves the accuracy and user experience of the recommendation system.

CN118690183BActive Publication Date: 2024-11-05HUBEI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411195558.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-05
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The existing multimodal recommendation system fails to fully understand user preferences, resulting in poor accuracy and performance of recommendation results, and the simple modal information fusion method leads to the loss of modal information.

Method used

Using a self-supervision-based method, by obtaining multimodal data of users and projects, extracting features and dynamically fusion, an enhanced user-project map is constructed, and interactive scores are optimized for project recommendation. The specific steps include: obtaining user and project data, extracting text and visual features, enriching user features, building project-project graphs, performing graph convolution optimization, and finally project recommendation through interactive scores.

Benefits of technology

Deeply explore the relationship between user purchase items, enhance user preference feature representation, solve the problem of loss of modal information, and improve recommendation accuracy and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118690183B_ABST
    Figure CN118690183B_ABST
Patent Text Reader

Abstract

The present invention provides a multimodal recommendation method and device for enhancing user preferences based on self-supervision, which relates to the field of multimodal recommendation, including: enriching user features through project text modal features and project visual modal features to obtain enriched user features; constructing a project-project graph through project text modal features and project visual modal features, dynamically fusing modal features of the project-project graph to obtain project fusion features; constructing an enhanced user-project graph through enriched user features and project fusion features, optimizing the enhanced user-project graph, and obtaining an optimized enhanced user-project graph. The present invention constructs an enhanced user-project graph through enriched user features and project fusion features, optimizes the enhanced user-project graph through a self-supervised learning mechanism, and recommends projects through the optimized enhanced user-project graph, which provides an important improvement in the performance of the recommendation system and user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of multimodal recommendation, and in particular to a multimodal recommendation method and device for enhancing user preference based on self-supervision. Background Art

[0002] With the popularization of the Internet and smart devices, the data generated by users has become more diverse, including text, images, audio, video, and behavioral data. The advancement of deep learning technologies (such as CNN and Transformer) has made it possible to extract and fuse high-level features from different modalities. Multimodal recommendation systems integrate this data to provide accurate and personalized recommendations, significantly improving user experience.

[0003] In e-commerce, personalized product recommendations are made by combining users’ browsing history, purchase history, product images, and comment text to improve sales conversion rates. Social media platforms use data such as user content, likes, comments, and sharing to recommend related posts, friends, or topics, thereby enhancing user stickiness and interaction. Video platforms such as YouTube or Netflix provide accurate movie recommendations and enhance viewing experience based on user viewing history, video content (subtitles and video frames), comments, and ratings.

[0004] Multimodal recommendation systems not only improve the relevance and satisfaction of recommendations, but also help companies gain an advantage in market competition. By leveraging rich user and content data, the system can more accurately capture user interests, provide personalized and attractive recommendations, and enhance user loyalty and platform stickiness. With technological advances and application expansion, multimodal recommendation systems will play a more important role in the future.

[0005] However, in previous multimodal recommendation systems, most methods only enriched the representation of items by analyzing the relationship between items purchased by users. This method often ignores the fact that the feature representation of items purchased by users often contains rich user preference information. Secondly, the past methods of fusion of multimodal information of items usually simply spliced ​​or combined, which may lead to the loss of item modal information. Therefore, the recommendation systems in the past failed to fully understand user preferences and did not effectively handle the loss of item modal information, resulting in unsatisfactory accuracy and performance of recommendation results. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a multimodal recommendation method and device based on self-supervision to enhance user preferences, so as to solve the problem of insufficient accuracy of existing recommendation methods.

[0007] The present invention provides a multimodal recommendation method for enhancing user preferences based on self-supervision, comprising:

[0008] S1: Acquire user-related data and project-related data, where the project-related data includes: project text data and project visual data;

[0009] S2: extracting features from user-related data, item text data, and item visual data to obtain user features, item text modal features, and item visual modal features, respectively;

[0010] S3: enriching the user features through the item text modal features and the item visual modal features to obtain the enriched user features;

[0011] S4: Construct a project-project graph through project text modal features and project visual modal features, dynamically fuse the modal features of the project-project graph, and obtain project fusion features;

[0012] S5: construct an enhanced user-item graph through the enriched user features and item fusion features, optimize the enhanced user-item graph, and obtain an optimized enhanced user-item graph;

[0013] S6: The user-item interaction score is calculated through the optimized enhanced user-item graph, and item recommendations are made based on the interaction score.

[0014] Preferably, step S3 specifically comprises:

[0015] S31: converting user features into item preference queries;

[0016] S32: By evaluating the correlation and criticality between the item preference query and the item text modality features, the attention weight of the text modality is calculated;

[0017] S33: By evaluating the correlation and criticality between the item preference query and the item visual modality features, the attention weight of the visual modality is calculated;

[0018] S34: Calculate the enriched user features through item preference query, attention weight of text modality and attention weight of visual modality.

[0019] Preferred:

[0020] The calculation formula of the attention weight of the text modality is:

[0021]

[0022] in, Indicates The text modality features of the sentences, represents the weight matrix of the sentence, represents the first bias term, Indicates The attention weight of the text modality of the sentence, represents the first nonlinear activation function, N is the total number of sentences;

[0023] The calculation formula of the attention weight of the visual modality is:

[0024]

[0025] in, Indicates The visual modality features of the image, represents the weight matrix of the image, represents the second bias term, Indicates The attention weight of the visual modality of the image, represents the second nonlinear activation function, M is the total number of pictures;

[0026] The calculation formula for the enriched user features is:

[0027]

[0028] Among them, p represents the user, The enriched user features, represents the comprehensive function, Represents an item preference query.

[0029] Preferably, step S4 is specifically:

[0030] S41: construct a cosine similarity matrix, and calculate the similarity scores between each item through the cosine similarity matrix;

[0031] S42: converting the cosine similarity matrix into a sparse matrix, normalizing the project text modal features and the project visual modal features of each project through the sparse matrix, and constructing a project-project graph;

[0032] S43: Optimizing the features and structure of the project-project graph to obtain an optimized project-project graph;

[0033] S44: Set a query matrix, a key matrix and a value matrix, calculate the relationship attention weights between the text modality and the visual modality in the optimized item-item graph through the query matrix, the key matrix and the value matrix, and obtain the item fusion features through the relationship attention weights and the value matrix.

[0034] Preferred:

[0035] The query matrix for item q , key matrix Sum Matrix The calculation formula is:

[0036] ,

[0037] ,

[0038]

[0039] Among them, q represents the project, Represents the modal characteristics of the project text, Represents the visual modality characteristics of the project, , and The learnable weight matrices representing the query matrix, key matrix, and value matrix respectively;

[0040] The calculation formula for the relationship attention weight between the textual modality and the visual modality of item q is:

[0041]

[0042] in, Represents the query matrix No. OK, Represents the key matrix No. OK, Representation query and The relationship between the textual modality and the visual modality when represents the scaling factor;

[0043] The calculation formula of the project fusion feature is:

[0044]

[0045] in, represents the item fusion feature of item q, Represents the value matrix OK.

[0046] Preferably, step S5 is specifically:

[0047] S51: construct an initial user-item graph F, ;

[0048] Among them, p represents a user, q represents a project, A represents a user set, and B represents a project set;

[0049] Indicates user The enriched user features, represents the item fusion feature of item q;

[0050] S52: Perform graph convolution on the initial user-item graph through the enriched user features and item fusion features to obtain an enhanced user-item graph;

[0051] S53: Optimizing the enhanced user-item graph by using negative cosine similarity to obtain an optimized enhanced user-item graph.

[0052] Preferably, step S52 is specifically as follows:

[0053] S521: Use the enriched user features and item fusion features as the embedding of the initial user-item graph to obtain the graph convolution step The embedded user features and embedded item features of the layer are calculated as follows:

[0054]

[0055] in, Indicates user In the graph convolution The embedded user features of the layer, Display items In the graph convolution The embedding item features of the layer, Represents user Interactive item sets, Representation and Project The set of users that interact, Indicates user In the graph convolution The user features after layer enrichment, Display items In the graph convolution The project fusion features of the layer; and They are graph convolution The weight matrix of the layer, and For users and Projects The attention weight between

[0056] S522: Repeat step S521 to obtain all embedded user features and embedded project features, combine each embedded user feature and embedded project feature to obtain enhanced user features and enhanced project features, and the calculation formula is:

[0057] ,

[0058]

[0059] in, Indicates user Enhanced user features, represents the strengthening item characteristics of item q, represents the total number of graph convolution layers, and Both are graph convolutions. The feature combination weights of the layer, Represents the concatenation operation of features;

[0060] S523: Constructing an enhanced user-item graph by enhancing user features and enhancing item features.

[0061] Preferably, step S53 is specifically as follows:

[0062] S531: Adjust the parameters of the enhanced user-item graph to calculate the negative cosine similarity between the enhanced user features in the enhanced user-item graph and the enriched user features , the calculation formula is:

[0063]

[0064] in, represents the vector dot product, Indicates user Enhanced user features;

[0065] S532: Repeat step S531 until the negative cosine similarity is smaller than the preset value, and an optimized enhanced user-item graph is obtained.

[0066] A storage medium stores instructions and data for implementing the multimodal recommendation method based on self-supervision to enhance user preferences.

[0067] A multimodal recommendation device based on self-supervision to enhance user preferences comprises: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the multimodal recommendation method based on self-supervision to enhance user preferences.

[0068] The present invention has the following beneficial effects:

[0069] By enriching user features with item text modal features and item visual modal features, the relationship between items purchased by users is deeply mined, and these relationships are used to enhance the feature representation of user preferences, so as to accurately understand user interests. In addition, by dynamically fusing item features of different modalities, item fusion features are obtained to enrich the overall item feature representation, solve the problem of modality loss, and improve recommendation accuracy. Finally, an enhanced user-item graph is constructed through enriched user features and item fusion features, and the enhanced user-item graph is optimized through a self-supervised learning mechanism. Items are recommended through the optimized enhanced user-item graph, which provides an important improvement in the performance and user experience of the recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0071] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention;

[0072] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0073] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0074] Reference Figure 1 The present invention provides a multimodal recommendation method for enhancing user preferences based on self-supervision, comprising:

[0075] S1: Acquire user-related data and project-related data, where the project-related data includes: project text data and project visual data;

[0076] Specifically, the historical scoring records of the user on the purchased items (articles) and the relevant data of the items are obtained from the corresponding data set, wherein the relevant data of the items include the description data (text data) and relevant attribute data (visual data) of the items.

[0077] S2: extracting features from user-related data, item text data, and item visual data to obtain user features, item text modal features, and item visual modal features, respectively;

[0078] Furthermore, step S2 is specifically as follows:

[0079] S21: Learning textual representations of items by using convolutional and personalized attention networks;

[0080] Specifically, we first represent the user embedding as a vector and calculate the user preference. The specific calculation formula is as follows:

[0081]

[0082] in Indicates the user's preference. and Parameters representing the user vector.

[0083] According to the user's preferences, the word representations in the query item text are calculated, and the attention weight of each word is accurately calculated. Then, these weights are used to weight the contextual representation of the word to obtain a comprehensive weighted representation of the entire text, which is calculated as follows:

[0084]

[0085] in Indicates the number of words in the item text, Indicates the first few words, represents the feature representation of each word, Indicates the relationship between words. A feature representation representing the modality of the item text.

[0086] S22: Using pre-trained Object detection models that can accurately extract eye-catching elements in an item Image (visual modality). Further use Models, extracting from these models Feature representation of images, we can effectively obtain the visual modality features of items .

[0087] S3: enriching the user features through the item text modal features and the item visual modal features to obtain the enriched user features;

[0088] Furthermore, step S3 is specifically as follows:

[0089] S31: converting user features into item preference queries;

[0090] Specifically, the user's embedding vector is converted into an item preference query , in order to accurately reflect the user's interest and demand for the project. The calculation formula is:

[0091]

[0092] in, and Parameters representing the relevant user vector;

[0093] S32: By evaluating the correlation and criticality between the item preference query and the item text modality features, the attention weight of the text modality is calculated;

[0094] Furthermore, the calculation formula of the attention weight of the text modality is:

[0095]

[0096] in, Indicates The text modality features of the sentences, represents the weight matrix of the sentence, represents the first bias term, Indicates The attention weight of the text modality of the sentence, represents the first nonlinear activation function, N is the total number of sentences;

[0097] Specifically, Can be used ;

[0098] S33: By evaluating the correlation and criticality between the item preference query and the item visual modality features, the attention weight of the visual modality is calculated;

[0099] Furthermore, the calculation formula of the attention weight of the visual modality is:

[0100]

[0101] in, Indicates The visual modality features of the image, represents the weight matrix of the image, represents the second bias term, Indicates The attention weight of the visual modality of the image, represents the second nonlinear activation function, M is the total number of pictures;

[0102] Specifically, Can be used ;

[0103] S34: Calculate the enriched user features through item preference query, attention weight of text modality and attention weight of visual modality;

[0104] Furthermore, the calculation formula of the enriched user features is:

[0105]

[0106] Among them, p represents the user, The enriched user features, represents the comprehensive function, Represents an item preference query.

[0107] S4: Construct a project-project graph through project text modal features and project visual modal features, dynamically fuse the modal features of the project-project graph, and obtain project fusion features;

[0108] Furthermore, step S4 is specifically as follows:

[0109] S41: construct a cosine similarity matrix, and calculate the similarity scores between each item through the cosine similarity matrix;

[0110] Specifically, the expression of the cosine similarity matrix is:

[0111]

[0112] in Display items and Projects The cosine similarity between is modal Medium Project The original characteristics of

[0113] S42: converting the cosine similarity matrix into a sparse matrix, normalizing the project text modal features and the project visual modal features of each project through the sparse matrix, and constructing a project-project graph;

[0114] Specifically, S42 is:

[0115] S421: In order to convert the obtained cosine similarity matrix into a sparse matrix, you can use The method of sparseness is to set a threshold and replace all similarity values ​​in the matrix that are less than the threshold with zero. In this way, similarity values ​​greater than or equal to the threshold are retained, and the rest become zero, so that the expression of the sparse matrix is:

[0116]

[0117] in Represents the cosine similarity matrix OK, Returns the input vector The set of highest values, Represents the transformed sparse matrix.

[0118] S422: The expression for normalizing the obtained sparse matrix is:

[0119]

[0120] in represents a diagonal matrix, represents the normalized adjacency matrix.

[0121] S433: Combining the normalized matrix of each modality, the expression for constructing the item-item graph is:

[0122]

[0123] in is modal The normalized matrix The weight of Represents the project-project graph constructed;

[0124] S43: Optimizing the features and structure of the project-project graph to obtain an optimized project-project graph;

[0125] Specifically, S43 is as follows:

[0126] S431: Through in-depth analysis and study of the constructed project drawings, the acquired structural knowledge is combined with the initial design to continuously optimize and improve;

[0127] Specifically, by deeply studying and learning the project graph, the initial modal features are transformed into a more advanced feature structure. This process not only enriches the expression of the project graph, but also improves the abstract level of the features, laying the foundation for subsequent analysis and application. The calculation formula is:

[0128]

[0129] in is a trainable transformation matrix, is the bias vector, It is the high-level feature after conversion.

[0130] After that, the previously constructed project-project graph is combined with the learned graph. The calculation formula is:

[0131]

[0132] in, Representing modality The final graph adjacency matrix of the latent structure, is the coefficient of the skip connection.

[0133] S432: After obtaining the modality-aware latent structure, a graph convolution operation is applied to learn the item-item graph to improve the quality of item representation;

[0134] Specifically, a lightweight GCN without feature transformation and feature aggregation is used to perform simple message propagation and aggregation. The calculation formula is:

[0135]

[0136] in, Indicates the mode No. The embedding matrix of the layer, Indicates the number of convolution layers;

[0137] S44: setting a query matrix, a key matrix, and a value matrix, calculating the optimized relationship attention weights between the text modality and the visual modality in the item-item graph through the query matrix, the key matrix, and the value matrix, and calculating the item fusion features through the relationship attention weights and the value matrix;

[0138] Furthermore, the query matrix of item q is , key matrix Sum Matrix The calculation formula is:

[0139] ,

[0140] ,

[0141]

[0142] Among them, q represents the project, Represents the modal characteristics of the project text, Represents the visual modality characteristics of the project, , and The learnable weight matrices representing the query matrix, key matrix, and value matrix respectively;

[0143] The calculation formula for the relationship attention weight between the textual modality and the visual modality of item q is:

[0144]

[0145] in, Represents the query matrix No. OK, Represents the key matrix No. OK, Representation query and The relationship between the textual modality and the visual modality when represents the scaling factor;

[0146] Specifically, Used to prevent the dot product value from being too large;

[0147] The calculation formula of the project fusion feature is:

[0148]

[0149] in, represents the item fusion feature of item q, Represents the value matrix OK.

[0150] S5: construct an enhanced user-item graph through the enriched user features and item fusion features, optimize the enhanced user-item graph, and obtain an optimized enhanced user-item graph;

[0151] Furthermore, step S5 is specifically as follows:

[0152] S51: construct an initial user-item graph F, ;

[0153] Among them, p represents a user, q represents a project, A represents a user set, and B represents a project set;

[0154] Indicates user The enriched user features, represents the item fusion feature of item q;

[0155] Specifically, user p in project The positive feedback on the edge ,otherwise ;

[0156] S52: Perform graph convolution on the initial user-item graph through the enriched user features and item fusion features to obtain an enhanced user-item graph;

[0157] Further, step S52 is specifically as follows:

[0158] S521: Use the enriched user features and item fusion features as the embedding of the initial user-item graph to obtain the graph convolution step The embedded user features and embedded item features of the layer are calculated as follows:

[0159]

[0160] in, Indicates user In the graph convolution The embedded user features of the layer, Display items In the graph convolution The embedding item features of the layer, Represents user Interactive item sets, Representation and Project The set of users that interact, Indicates user In the graph convolution The user features after layer enrichment, Display items In the graph convolution The project fusion features of the layer; and They are graph convolution The weight matrix of the layer, and For users and Projects The attention weight between

[0161] Specifically, and Used to project the feature vector into the new feature space, and Indicates the degree of influence of neighbor nodes on the central node; embedded user features and embedded project features include all information levels from initial features to advanced features, ensuring the diversity and richness of features;

[0162] S522: Repeat step S521 to obtain all embedded user features and embedded project features, combine each embedded user feature and embedded project feature to obtain enhanced user features and enhanced project features, and the calculation formula is:

[0163] ,

[0164]

[0165] in, Indicates user Enhanced user features, represents the strengthening item characteristics of item q, represents the total number of graph convolution layers, and Both are graph convolutions. The feature combination weights of the layer, Represents the concatenation operation of features;

[0166] S523: constructing an enhanced user-item graph by enhancing user features and enhancing item features;

[0167] S53: optimizing the enhanced user-item graph by using negative cosine similarity to obtain an optimized enhanced user-item graph;

[0168] Furthermore, a comparison is made between the user feature representation before convolution and the user feature representation after convolution to form a mutually supervised learning mechanism to ensure that the user's preferences are fully learned. This method enhances the expressiveness of user features through contrastive learning, thereby improving the performance of the model;

[0169] Step S53 is specifically as follows:

[0170] S531: Adjust the parameters of the enhanced user-item graph to calculate the negative cosine similarity between the enhanced user features in the enhanced user-item graph and the enriched user features , the calculation formula is:

[0171]

[0172] in, represents the vector dot product, Indicates user Enhanced user features;

[0173] S532: Repeat step S531 until the negative cosine similarity is smaller than the preset value, and the optimized enhanced user-item graph is obtained;

[0174] Specifically, in order to further optimize the model, the paired Bayesian personalized ranking method can be continued to be used to improve the ranking performance by encouraging the predicted scores of positive user-item pairs to be higher than their negative pairs. The calculation formula is:

[0175]

[0176] in is the training instance set, is the sigmoid activation function.

[0177] S6: The user-item interaction score is calculated through the optimized enhanced user-item graph, and item recommendations are made based on the interaction score.

[0178] Furthermore, step S6 is specifically as follows:

[0179] According to the predicted interaction scores, the candidate items are sorted in descending order, and the top-ranked items are recommended to the user. This process effectively captures the relationship between user and item features, improves the personalized recommendation effect of the recommendation system, and thus enhances user satisfaction and experience. The formula for calculating the interaction score between users and items is:

[0180]

[0181] in Represents the interaction score between the user and the item. The higher the score, the more the user likes the item.

[0182] See also Figure 2 , Figure 2 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a multimodal recommendation device 401 for enhancing user preference based on self-supervision, a processor 402 and a storage medium 403.

[0183] A multimodal recommendation device 401 based on self-supervision to enhance user preference: The multimodal recommendation device 401 based on self-supervision to enhance user preference implements the multimodal recommendation method based on self-supervision to enhance user preference.

[0184] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the multimodal recommendation method based on self-supervision to enhance user preferences.

[0185] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the multimodal recommendation method based on self-supervision to enhance user preferences.

[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0187] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0188] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal recommendation method based on self-supervision to enhance user preferences, characterized in that: include: S1: Acquire user-related data and project-related data, where the project-related data includes: project text data and project visual data; S2: extracting features from user-related data, item text data, and item visual data to obtain user features, item text modal features, and item visual modal features, respectively; S3: enriching the user features through the item text modal features and the item visual modal features to obtain the enriched user features; S4: Construct a project-project graph through project text modal features and project visual modal features, dynamically fuse the modal features of the project-project graph, and obtain project fusion features; S5: construct an enhanced user-item graph through the enriched user features and item fusion features, optimize the enhanced user-item graph, and obtain an optimized enhanced user-item graph; S6: Obtain the user-item interaction score by calculating the optimized enhanced user-item graph, and recommend items based on the interaction score; Step S3 is specifically as follows: S31: converting user features into item preference queries; S32: By evaluating the correlation and criticality between the item preference query and the item text modality features, the attention weight of the text modality is calculated; S33: By evaluating the correlation and criticality between the item preference query and the item visual modality features, the attention weight of the visual modality is calculated; S34: Calculate the enriched user features through item preference query, attention weight of text modality and attention weight of visual modality; The calculation formula of the attention weight of the text modality is: in, Indicates The text modality features of the sentences, represents the weight matrix of the sentence, represents the first bias term, Indicates The attention weight of the text modality of the sentence, represents the first nonlinear activation function, N is the total number of sentences; The calculation formula of the attention weight of the visual modality is: in, Indicates The visual modality features of the image, represents the weight matrix of the image, represents the second bias term, Indicates The attention weight of the visual modality of the image, represents the second nonlinear activation function, M is the total number of pictures; The calculation formula for the enriched user features is: Among them, p represents the user, The enriched user features, represents the comprehensive function, Represents an item preference query; Step S4 is specifically as follows: S41: construct a cosine similarity matrix, and calculate the similarity scores between each item through the cosine similarity matrix; S42: converting the cosine similarity matrix into a sparse matrix, normalizing the project text modal features and the project visual modal features of each project through the sparse matrix, and constructing a project-project graph; S43: Optimizing the features and structure of the project-project graph to obtain an optimized project-project graph; S44: setting a query matrix, a key matrix, and a value matrix, calculating the optimized relationship attention weights between the text modality and the visual modality in the item-item graph through the query matrix, the key matrix, and the value matrix, and calculating the item fusion features through the relationship attention weights and the value matrix; The query matrix for item q , key matrix Sum Matrix The calculation formula is: , , Among them, q represents the project, Represents the modal characteristics of the project text, Represents the visual modality characteristics of the project, , and The learnable weight matrices representing the query matrix, key matrix, and value matrix respectively; The calculation formula for the relationship attention weight between the textual modality and the visual modality of item q is: in, Represents the query matrix No. OK, Represents the key matrix No. OK, Representation query and The relationship between the textual modality and the visual modality when represents the scaling factor; The calculation formula of the project fusion feature is: in, represents the item fusion feature of item q, Represents the value matrix OK; Step S5 is specifically as follows: S51: construct an initial user-item graph F, ; Among them, p represents a user, q represents a project, A represents a user set, and B represents a project set; Indicates user The enriched user features, represents the item fusion feature of item q; S52: Perform graph convolution on the initial user-item graph through the enriched user features and item fusion features to obtain an enhanced user-item graph; S53: Optimizing the enhanced user-item graph by using negative cosine similarity to obtain an optimized enhanced user-item graph.

2. The multimodal recommendation method based on self-supervision to enhance user preferences according to claim 1, characterized in that: Step S52 is specifically as follows: S521: Use the enriched user features and item fusion features as the embedding of the initial user-item graph to obtain the graph convolution step The embedded user features and embedded item features of the layer are calculated as follows: in, Indicates user In the graph convolution The embedded user features of the layer, Display items In the graph convolution The embedding item features of the layer, Represents user Interactive item sets, Representation and Project The set of users that interact, Indicates user In the graph convolution The user features after layer enrichment, Display items In the graph convolution The project fusion features of the layer; and They are graph convolution The weight matrix of the layer, and For users and Projects The attention weight between S522: Repeat step S521 to obtain all embedded user features and embedded project features, combine each embedded user feature and embedded project feature to obtain enhanced user features and enhanced project features, and the calculation formula is: , in, Indicates user Enhanced user features, represents the strengthening item characteristics of item q, represents the total number of graph convolution layers, and Both are graph convolutions. The feature combination weights of the layer, Represents the concatenation operation of features; S523: Constructing an enhanced user-item graph by enhancing user features and enhancing item features.

3. The multimodal recommendation method based on self-supervision to enhance user preferences according to claim 1, characterized in that: Step S53 is specifically as follows: S531: Adjust the parameters of the enhanced user-item graph to calculate the negative cosine similarity between the enhanced user features in the enhanced user-item graph and the enriched user features , the calculation formula is: in, represents the vector dot product, Indicates user Enhanced user features; S532: Repeat step S531 until the negative cosine similarity is smaller than the preset value, and an optimized enhanced user-item graph is obtained.

4. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the multimodal recommendation method based on self-supervision to enhance user preferences as described in any one of claims 1 to 3.

5. A multimodal recommendation device based on self-supervision to enhance user preferences, characterized in that: include: Processor and storage media; The processor loads and executes instructions and data in the storage medium to implement the multimodal recommendation method based on self-supervision to enhance user preferences as described in any one of claims 1 to 3.