Cross-modal feature alignment and fusion method based on comparative learning

Abstract

The invention discloses a cross-modal feature alignment and fusion method, and belongs to the field of multi-modal recommendation. The method comprises the following steps: firstly, capturing a semantic relationship between projects in the same modal by constructing a project semantic graph, and performing graph convolution operation on overall features and modal specific features on the basis to enhance the expression ability of the overall features and the modal specific features in the modal; potential information between features is captured and utilized, and the problem of incomplete matching between modes is relieved. And then, comparing the enhanced overall features with modal features in each modal by using a comparative learning method, and promoting cross-modal information transmission and complementation. And through conjoint analysis of project semantic graphs of different modalities, effective modeling of a complex relationship between the modalities is realized. According to the method, cross-modal semantic consistency is ensured, so that redundancy and noise in a modal fusion process are reduced, and the accuracy of multi-modal recommendation is improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal recommendation and relates to a cross-modal feature alignment and fusion method based on contrastive learning. Contrastive learning technology and item semantic graph are used to ensure cross-modal semantic consistency, thereby reducing redundancy and noise in the modal fusion process and improving the accuracy of multimodal recommendation. Background Art

[0002] Recommendation systems have changed the way users browse through large amounts of information and products. These systems are driven by advanced algorithms that analyze user preferences and item attributes, and typically rely on implicit interactions (such as purchases and clicks) to tailor recommendations. However, human experience is inherently multimodal, covering multiple sensory inputs including audio, text, and images. Therefore, a series of works focus on multimodal recommendations, dedicated to understanding and integrating data from different modalities to provide more accurate and personalized recommendations (Chen, J., Zhang, H., He, X., Nie, L., Liu, W., Chua, T.-S., 2017. Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention.).

[0003] In recent multimodal recommendation research, graph neural networks have been widely used due to their powerful feature expression capabilities and ability to model complex relationships. In addition to being applied to traditional user-item interaction graphs, researchers have also tried to apply them to more graph structures, such as user-user relationship graphs (Wang, Q., Wei, Y., Yin, J., Wu, J., Song, X., Nie, L., 2023. DualGNN: Dual Graph Neural Network for Multimedia Recommendation.) and item-item relationship graphs (Zhou, X., Shen, Z., 2023. A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation.). These graph structures help capture potential relationships and improve recommendation accuracy, but their applications mostly focus on combining features of different modalities in various ways, while ignoring the correspondence between different modalities. This makes it difficult to ensure the consistency of feature expression under different modalities, thereby increasing information redundancy and the presence of noise.

[0004] In existing research, self-supervised learning has been introduced into multimodal recommendation to generate more robust and comprehensive feature representations. (Tao, Z., Liu, X., Xia, Y., Wang, X., Yang, L., Huang, X., Chua, T.-S., 2023. Self-Supervised Learning for Multimedia Recommendation.) Self-supervised learning tasks such as contrastive learning can help align features of different modalities in modal fusion. However, it is easy to introduce some irrelevant features in this process and under-model the complex relationship between modalities. Summary of the invention

[0005] The present invention mainly solves the challenges of feature alignment and fusion between different modal data in a multimodal recommendation system, especially the problem of complex relationship modeling and incomplete feature matching between different modalities. By constructing a cross-modal feature contrast learning model, semantic consistency is ensured, redundancy and noise are reduced, and better modal fusion is achieved. First, the semantic relationship between items under the same modality is captured by constructing a project semantic graph, and on this basis, graph convolution operations are performed on the overall features and modality-specific features to enhance their expressive power under this modality. The potential information between features is captured and utilized to alleviate the problem of incomplete matching between modalities. Then, the contrastive learning method is used to compare the enhanced overall features and modal features in each modality to promote cross-modal information transfer and complementarity. By jointly analyzing the project semantic graphs of different modalities, effective modeling of complex relationships between modalities is achieved.

[0006] The technical solution of the present invention:

[0007] The cross-modal feature alignment and fusion method based on contrastive learning has the following steps:

[0008] Step 100, feature extraction and processing;

[0009] Step 101, extracting original features from the image and text description of the item respectively;

[0010] Visual features are extracted using a pre-trained deep convolutional neural network. Text features are extracted using a sentence transformer on the concatenation of each item’s title, description, category, and brand.

[0011] Step 102, constructing a project semantic graph for each modality;

[0012] In each mode, the cosine similarity is used to calculate the similarity between items based on the extracted original features. The formula is expressed as

[0013]

[0014] Among them, m is the mode, including visual mode and text mode, f i m and f j m is the original feature of item i and item j under mode m, (f i m ) T f j m Yes i m and f j m The dot product of ||f i m || and ||f j m || respectively represent f i m and f j m The Euclidean norm of , is the similarity score between item i and item j under mode m; for each item, find the k most similar neighbors, construct a kNN graph, and use this to construct the item semantic graph The formula is expressed as

[0015]

[0016] in, is the set of similarity scores between item i and other items; then the adjacency matrix Normalize to get the adjacency matrix The formula is expressed as Where D m yes The degree matrix of

[0017] Step 103, initialization and processing of user and project ID embedding;

[0018] The Xavier method is used to initialize ID embedding, and then feature masking is performed to enhance the generalization ability of the model and prevent overfitting. The formula is expressed as

[0019]

[0020] in, is the initial user ID embedded, is the initial item ID embedding, ρ r represents the proportion of elements to be discarded, Bernoulli represents the generation of an independent Bernoulli random variable, and · represents the element-level multiplication. and are the user ID embedding and item ID embedding after feature masking;

[0021] Step 200, introducing a contrastive learning mechanism to align cross-modal features;

[0022] Step 201, projecting multimodal features;

[0023] Multi-layer perceptron MLP is used to project multimodal features into the same latent space, which is expressed as

[0024]

[0025] Among them, W m and b m represents the linear transformation matrix and bias in MLP, f i m represents the original modal features of item i under modality m, is the modal feature obtained after projection;

[0026] Step 202, using the project semantic graph to enhance feature expression;

[0027] Semantic graph of items in each modality Multi-layer graph convolution operations are performed on it to improve the feature expression ability by gradually integrating neighborhood information; the formula is expressed as

[0028]

[0029] Among them, N i represents the one-hop neighbor set of item i, is the semantic graph of items under modality m, represents the enhanced modal features of the item at the lth level under modality m, Represents the enhanced overall features of the project at the lth level under mode m; ii After layer convolution, the final enhanced modal feature f is obtained i,m and enhance the overall feature h i,m ;

[0030] Step 203, performing comparative study on the enhanced overall features and each modal feature;

[0031] ID embedding is used as the overall feature. After multi-layer graph convolution operation on the project semantic graph, the expressive ability of this modality is enhanced. Then it is compared with the enhanced modal features of each modality, and the loss function is compared. The formula is

[0032]

[0033] Among them, h i,m represents the enhancement of overall features, f i,mstands for enhanced modal features, sim(·) measures the similarity between two vectors and applies the cosine similarity function to these vectors, τ is an adjustable hyperparameter representing temperature, and N represents the number of items in the batch;

[0034] Step 300, integrating multimodal information and historical interaction information, generating a final representation and making a recommendation;

[0035] Step 301, fusing multimodal information;

[0036] The semantic graphs of items in all modalities are integrated according to the importance score of each modality. The formula is expressed as

[0037]

[0038] Where S ij is the overall project semantic graph, is the semantic graph of items under modality m, α m is the importance score of modality m, and the importance score of visual modality α v and the importance score α of the text modality t The sum of the two is 1; then, multiple layers of neighbors are aggregated on the overall project semantic graph to enhance the representation ability of the feature, and the formula is expressed as

[0039]

[0040] Among them, N i is the set of one-hop neighbors of node i, S ij is the overall project semantic graph, is the enhanced overall feature of item i in the lth layer under all modalities. When l = 0, The feature masked item ID obtained in step 103 is embedded Initialization; after L ii After layer graph convolution, we get item representations that encode multimodal information

[0041] Step 302, constructing a user-item interaction graph;

[0042] Collect the interaction data between users and projects; represent users and projects as user nodes and project nodes in the user-project interaction graph respectively, and establish edges between user nodes and project nodes based on the interaction data; then, according to the degree of the node, the edges are proportional to ρ. d Prune edges to remove noise;

[0043] Step 303, encoding a user-item interaction graph;

[0044] Use LightGCN to encode the user-item interaction graph, the formula is expressed as

[0045]

[0046] Among them, N u and N i is the set of one-hop neighbors of nodes u and i, is the representation of user u in layer l, The representation of item i in the lth layer, when l = 0, and The feature-masked user ID obtained in step 103 is embedded and project ID embedded initialization, Used to normalize the features learned from each layer; after L ui After the layer graph convolution, the final user and item representations are obtained by element-wise summing the representations of each layer, and the formula is expressed as

[0047]

[0048] in, is the representation of user u in layer l, The representation of item i in layer l, L ui represents the total number of graph convolution layers; h u and h i are the user and item representations obtained after encoding the user-item interaction graph;

[0049] Step 304, integrating all representations;

[0050] The representation that integrates multimodal information and interactive information is fused and the formula is expressed as

[0051] z u =h u

[0052]

[0053] Among them, z u and z i are the final user representation and the final project representation, respectively, and h u and h i are the user and item representations obtained after encoding the user-item interaction graph, is an item representation that encodes multimodal information;

[0054] Step 305, model optimization;

[0055] In order to optimize the model parameters of the recommendation task, Bayesian personalized ranking BPR loss is used as the basic optimization function; a training set D is constructed, including the triples (u, p, n) of each user, where the positive item p is the item that user u has interacted with, and the negative item n is randomly selected from the items that user u has not interacted with; BPR loss The definition is as follows:

[0056]

[0057] in, Calculate z u and z p The inner product of ; σ is the Sigmoid function; then, the BPR loss and contrastive learning loss are combined to update the user and item representations, and the formula is expressed as

[0058]

[0059] Where λ is the weight of contrastive learning loss, is the BPR loss, is the contrastive learning loss, is the total loss.

[0060] Compared with the existing method, the beneficial effects of the present invention are:

[0061] The present invention constructs a multimodal recommendation framework through contrastive learning, and combines the project semantic graph to achieve effective alignment and fusion of cross-modal features, thereby improving the accuracy of multimodal recommendation. The first part is to extract the original modal features from the multimodal data and construct the project semantic graph, as well as generate ID embedding. The second part uses contrastive learning and combines the project semantic graph to deeply explore the relationship between the modality and the modality to achieve effective alignment of cross-modal features. The third part fuses the multimodal information and encodes the interaction information of users and projects to form the final representation of users and projects, and then makes recommendations.

[0062] The present invention applies the project semantic graph to the contrastive learning task, takes ID embedding as the overall feature, highlights the features under each modality by performing graph convolution operations on the project semantic graph of each modality, and then performs contrastive learning with the enhanced modal features under each modality. The graph convolution operation integrates neighbor information and alleviates the problem of incomplete feature matching between modalities. The project semantic graphs of different modalities are jointly analyzed, and the contrastive learning mechanism is introduced to realize information transfer between modalities, which effectively models the complex relationships within and between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart for implementing the cross-modal feature alignment and fusion method based on contrastive learning provided by the present invention;

[0064] Figure 2 is a flowchart for implementing step 100;

[0065] Figure 3 is a flow chart for implementing step 200;

[0066] Figure 4 is a flowchart for implementing step 300. DETAILED DESCRIPTION

[0067] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0068] Figure 1 This is a flow chart of the implementation of the cross-modal feature alignment and fusion method based on contrastive learning provided by the present invention. The specific steps are as follows:

[0069] Step 100, extracting original modal features from multimodal data and constructing an item semantic graph, and generating ID embedding.

[0070] like Figure 2 As shown, step 100 includes steps 101 to 103:

[0071] Step 101, extracting raw features from the image and text description of the item, including extracting visual features using a pre-trained deep convolutional neural network, the dimension of the visual features is 4096; using a sentence transformer on the concatenation of the title, description, category and brand of each item to extract text features, the dimension of the text features is 384;

[0072] Step 102, construct the semantic graph of each modality, calculate the cosine similarity of the original features under each modality, find the k most similar neighbors for each item, construct the semantic graph of the item, and then normalize it, taking k as 10;

[0073] Step 103, initialization and processing of user and project ID embedding: Use the Xavier method to initialize ID embedding, the dimension of ID embedding is 64, and then perform dropout regularization to enhance the generalization ability of the model and prevent overfitting. The dropout parameter (the proportion of elements to be discarded) ρ r The value range is {0.1, 0.2, 0.3};

[0074] Step 200 uses contrastive learning and combines it with the item semantic graph to deeply explore the relationship between the modality and the modality, so as to achieve effective alignment of cross-modal features.

[0075] like Figure 3 As shown, step 200 includes steps 201 to 203:

[0076] Step 201, using a multi-layer perceptron (MLP) to project the multi-modal features into the same latent space, the dimension of the projected modal features is 64;

[0077] Step 202: Enhance feature expression using item semantic graphs. Graph convolution is performed on the image to improve the feature expression capability by gradually integrating neighborhood information. The number of graph convolution layers is 1.

[0078] Step 203, performing comparative learning on the enhanced overall features and each modal feature, wherein the temperature parameter τ of the comparative learning ranges from {0.1, 0.2, 0.3, 0.4};

[0079] Step 300, multimodal information is integrated and the interaction information of users and items is encoded to form the final representation of users and items for recommendation.

[0080] like Figure 4 As shown, step 300 includes steps 301 to 305:

[0081] Step 301, fusing multimodal information, integrating the project semantic graphs of all modalities according to the importance score of each modality (the importance score of the visual modality is determined to be 0.1 and the importance score of the text modality is 0.9 based on experience), and then performing a graph convolution operation on the overall feature on the overall project semantic graph, with the number of graph convolution layers being 1, to enhance the representation capability of the feature;

[0082] Step 302, construct a user-item interaction graph, collect the interaction data between users and items, represent users and items as nodes in the graph, and establish edges between user nodes and item nodes based on the interaction data. Then prune the edges according to a certain proportion based on the degree of the nodes, and the pruning parameter ρ d The value range is {0.6, 0.7, 0.8, 0.9};

[0083] Step 303: Use LightGCN to encode the user-item interaction graph, with 2 graph convolution layers. The final user and item representations are obtained by element-wise summing the representations of each layer.

[0084] Step 304, fusing the representation integrating the multimodal information and the interactive information;

[0085] Step 305, in order to optimize the model parameters of the recommendation task, Bayesian Personalized Ranking (BPR) loss is used as the basic optimization function, and combined with contrastive learning loss to update the user and item representations.

[0086] In combination with the solution of the present invention, experimental analysis was conducted on different data sets, and three widely used Amazon data sets were selected: (1) Baby (2) Sports and Outdoors (abbreviated as Sports) (3) Clothing, Shoes, and Jewelry (abbreviated as Clothing). In order to evaluate the method proposed in the present invention, two evaluation indicators were selected: Recall@K (recall rate@K), which measures the proportion of items that users are actually interested in among the top K recommended items; NDCG@K (normalized discounted cumulative gain@K), which combines the relevance and position of the recommendation results and can comprehensively reflect the effect of the recommendation algorithm. When K=10 and K=20, a series of experiments were carried out to demonstrate the effectiveness of the method proposed in the present invention. The experimental results are as follows:

[0087] Table 1 Recall@K (R@K) and NDCG@K (N@K) results in Top-K recommendation scenario

[0088]

[0089] From the results in Table 1, we can see that on the three data sets, compared with the best performing baseline model, the proposed method (denoted as CLAM in the table) improves the Recall@10 index by 6.70%, 4.18% and 5.09% respectively, and the Recall@20 index by 2.92%, 3.40% and 3.93% respectively, and the NDCG@10 and NDCG@20 indexes also improve by 3% to 8%. This shows that the method of the present invention has a good effect on cross-modal feature alignment and fusion, and ultimately improves the recommendation accuracy.

Claims

1. A cross-modal feature alignment and fusion method based on contrastive learning, characterized in that: Here are the steps: Step 100, feature extraction and processing; Step 101, extracting original features from the image and text description of the item respectively; Visual features are extracted using a pre-trained deep convolutional neural network. Text features are extracted using a sentence transformer on the concatenation of each item’s title, description, category, and brand. Step 102, constructing a project semantic graph for each modality; In each mode, the cosine similarity is used to calculate the similarity between items based on the extracted original features. The formula is expressed as Among them, m is the mode, including visual mode and text mode, f i m and f j m is the original feature of item i and item j under mode m, (f i m ) T f j m Yes i m and f j m The dot product of ||f i m || and ||f j m || respectively represent f i m and f j m The Euclidean norm of , is the similarity score between item i and item j under mode m; for each item, find the k most similar neighbors, construct a kNN graph, and use this to construct the item semantic graph The formula is expressed as in, is the set of similarity scores between item i and other items; then the adjacency matrix Normalize to get the adjacency matrix The formula is expressed as Where D m yes The degree matrix of Step 103, initialization and processing of user and project ID embedding; The Xavier method is used to initialize ID embedding, and then feature masking is performed to enhance the generalization ability of the model and prevent overfitting. The formula is expressed as in, is the initial user ID embedded, is the initial item ID embedding, ρ r represents the proportion of elements to be discarded, Bernoulli represents the generation of an independent Bernoulli random variable, and · represents the element-level multiplication. and are the user ID embedding and item ID embedding after feature masking; Step 200, introducing a contrastive learning mechanism to align cross-modal features; Step 201, projecting multimodal features; Multi-layer perceptron MLP is used to project multimodal features into the same latent space, which is expressed as Among them, W m and b m represents the linear transformation matrix and bias in MLP, f i m represents the original modal features of item i under modality m, is the modal feature obtained after projection; Step 202, using the project semantic graph to enhance feature expression; Semantic graph of items in each modality Multi-layer graph convolution operations are performed on it to improve the feature expression ability by gradually integrating neighborhood information; the formula is expressed as Among them, N i represents the one-hop neighbor set of item i, is the semantic graph of items under modality m, represents the enhanced modal features of the item at the lth level under modality m, Represents the enhanced overall features of the item at the lth layer under mode m; after L ii After layer convolution, the final enhanced modal feature f is obtained i,m and enhance the overall feature h i,m ; Step 203, performing comparative study on the enhanced overall features and each modality feature; ID embedding is used as the overall feature. After multi-layer graph convolution operation on the project semantic graph, the expressive ability of this modality is enhanced. Then it is compared with the enhanced modal features of each modality, and the loss function is compared. The formula is Among them, h i,m represents the enhancement of overall features, f i,m stands for enhanced modal features, sim(·) measures the similarity between two vectors and applies the cosine similarity function to these vectors, τ is an adjustable hyperparameter representing temperature, and N represents the number of items in the batch; Step 300, integrating multimodal information and historical interaction information, generating a final representation and making a recommendation; Step 301, fusing multimodal information; The semantic graphs of items in all modalities are integrated according to the importance score of each modality. The formula is expressed as Where S ij is the overall project semantic graph, is the semantic graph of items under modality m, α m is the importance score of modality m, and the importance score of visual modality α v and the importance score α of the text modality t The sum of the two is 1; then, multi-layer neighbors are aggregated on the overall project semantic graph to enhance the representation ability of the feature, and the formula is expressed as Among them, N i is the set of one-hop neighbors of node i, S ij is the overall project semantic graph, is the enhanced overall feature of item i in the lth layer under all modalities. When l = 0, The feature masked item ID obtained in step 103 is embedded Initialization; after L ii After layer graph convolution, we get item representations that encode multimodal information Step 302, constructing a user-item interaction graph; Collect the interaction data between users and projects; represent users and projects as user nodes and project nodes in the user-project interaction graph respectively, and establish edges between user nodes and project nodes based on the interaction data; then, according to the degree of the node, the edges are proportional to ρ. d Prune edges to remove noise; Step 303, encoding a user-item interaction graph; LightGCN is used to encode the user-item interaction graph, and the formula is expressed as Among them, N u and N i is the set of one-hop neighbors of nodes u and i, is the representation of user u in layer l, The representation of item i in the lth layer, when l = 0, and The feature-masked user ID obtained in step 103 is embedded and project ID embedded initialization, Used to normalize the features learned from each layer; after L ui After the layer graph convolution, the final user and item representations are obtained by element-wise summing the representations of each layer, and the formula is expressed as in, is the representation of user u in layer l, The representation of item i in layer l, L ui represents the total number of graph convolution layers; h u and h i are the user and item representations obtained after encoding the user-item interaction graph; Step 304, integrating all representations; The representation that integrates multimodal information and interactive information is fused and the formula is expressed as z u =h u Among them, z u and z i are the final user representation and the final project representation, respectively, and h u and h i are the user and item representations obtained after encoding the user-item interaction graph, is an item representation that encodes multimodal information; Step 305, model optimization; In order to optimize the model parameters of the recommendation task, Bayesian personalized ranking BPR loss is used as the basic optimization function; a training set is constructed It consists of a triplet (u, p, n) for each user, where the positive item p is the item that user u has interacted with, and the negative item n is randomly selected from the items that user u has not interacted with; BPR loss The definition is as follows: in, Calculate z u and z p The inner product of ; σ is the Sigmoid function; then, the BPR loss and contrastive learning loss are combined to update the user and item representations, and the formula is expressed as Where λ is the weight of contrastive learning loss, is the BPR loss, is the contrastive learning loss, is the total loss.

Images