Recommendation method, electronic device and storage medium based on multimodal graph contrast learning

Abstract

The present invention discloses a recommendation method based on multimodal graph contrastive learning, the steps of which include: 1. data collection and preprocessing; 2. graph convolution layer; 3. constructing contrastive learning layer; 4. constructing loss function; 5. training graph contrastive learning model. When processing the recommendation task of multimodal data, the present invention can enhance the representation of users and items through separate graph learning mode and contrastive learning, and alleviate the problem of multimodal noise pollution.

Description

Technical Field

[0001] The invention relates to a multimedia recommendation method, electronic equipment and storage medium based on multimodal graph contrast learning, and belongs to the field of recommendation systems. Background Art

[0002] Multimedia-based recommendation is a challenging task that requires not only learning collaborative signals from user-item interactions, but also capturing modality-specific user interest cues from complex multimedia content. Although current solutions for multimedia-based recommendation algorithms have made significant progress, they are still limited by multimodal noise pollution. Specifically, a considerable portion of the multimedia content of an item is irrelevant to user preferences, such as background, overall layout, image brightness, word order in the title, and semanticless words. In addition, most of the latest research is conducted through graph learning. This means that the noise spreads into user and item representations as the message propagates, and the pollution effect will be further amplified. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention proposes a multimedia recommendation method, electronic device and storage medium based on multimodal graph contrastive learning, so as to alleviate the problem of multimodal noise pollution when processing the recommendation task of multimodal data, and enhance the representation of users and items through separate graph learning modes and contrastive learning, thereby improving the accuracy and precision of recommendations.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The multimedia recommendation method based on multimodal graph contrast learning of the present invention is characterized in that it comprises the following steps:

[0006] Step 1: Data collection and preprocessing;

[0007] Step 1.1: Construct a set of product items, denoted as I = {i1,i2,…,in,…i |I|}, where in represents the nth item and |I| represents the total number of items;

[0008] Construct a user set, denoted as U = {u1,u2,…,um,…,u |U|}, where um represents the mth user; |U| represents the total number of users;

[0009] Constructing a user-project bipartite graph in, represents the mth user u m Is there an interaction between and the nth item in? If so, let Otherwise, let

[0010] Map the mth user um and the nth item in to user embeddings and project embedding The corresponding embedding vectors of the mth user um in image mode V and text mode T are and

[0011] Step 1.2: Deep feature extraction:

[0012] The image v corresponding to the nth product item in n Input into the pre-trained VGG16 model for processing to obtain image features Thus, the image feature matrix of image modality V is constructed using formula (1): dV is the dimension of image features:

[0013]

[0014] The text t corresponding to the nth product item in n Input into the pre-trained Sentence2Vec model for processing to obtain text features Thus, the text feature matrix of text modality T is constructed using formula (2): dT is the dimension of text features:

[0015]

[0016] Step 2: Construct a multimodal graph contrastive learning model, including graph convolution layer, contrastive learning layer, and prediction layer;

[0017] Step 2.1: Processing of the graph convolutional layer:

[0018] Step 2.2.1, use equation (3) and equation (4) to get the embedding of the mth user um and the nth item in the lth graph convolutional layer respectively:

[0019]

[0020]

[0021] In formula (3) and formula (4), and They represent the neighbor sets of the mth user um and the nth item in respectively. and They represent the number of neighbors of the m-th user um and the number of neighbors of the n-th item in respectively; is the embedding of the nth item in the l-1th graph convolutional layer. When l = 1, let is the embedding of the mth user um in the l-1th graph convolutional layer. When l = 1, let

[0022] Step 2.2.2: Under the image mode V or text mode T, use equations (5) and (6) to obtain the mth user u: m and the embedding of the nth item in the lth graph convolutional layer under multimodal modal

[0023]

[0024]

[0025] In formula (5) and formula (6), α is a hyperparameter, TR represents transposition, modal represents multimodality, and modal = V or T, is the weight transformation matrix of the multimodal modal, d moda l is the dimension of the multimodal feature, d is the embedding size; The features of the multimodal modal representing the nth item, represents the embedding of the l-1th graph convolutional layer of the mth user um in the multimodal modal, and represents the mth user u m The embedding vector of the l-1th layer under the image modality V, represents the mth user u m In the embedding vector of the l-1th layer of the text modality T, when l=1, let represents the embedding of the nth item in the l-1th graph convolutional layer under multimodal modal; and represents the image features of the nth item in, represents the text feature of the nth item in. When l = 1, let

[0026] Step 2.2.3: Use equations (7) and (8) to get the mth user u m and the embedding of the nth item in the l+1th graph convolutional layer under multimodal modal

[0027]

[0028]

[0029] Step 2.2.4: Process according to the process of step 2.2.2-step 2.2.3, so that the Lth layer outputs the features of the mth user um Features of the mth user um in multimodal modal Features of the nth item in

[0030] Step 2.3, processing of the contrastive learning layer:

[0031] Step 2.3.1: Construct the user contrast loss function through formula (9)

[0032]

[0033] In formula (9), represents the feature of the j-th user uj under the multimodal modal of the L-th layer, τ is a hyperparameter; Step 2.3.2, construct the item contrast loss function through formula (10)

[0034]

[0035] In formula (10), Represents the features of the k-th user ik under multimodal modal;

[0036] Step 2.3.3: Construct the contrast loss function using formula (11)

[0037]

[0038] Step 2.4: Processing of the prediction layer:

[0039] Formula (12) is used to calculate the preference score between the mth user um and the nth item in

[0040]

[0041] In formula (12), λ is a hyperparameter;

[0042] Step 3: Construct the loss function of the multimodal graph contrast learning model:

[0043] Step 3.1: Use formula (13) to construct the first loss function

[0044]

[0045] Step 3.2: Use formula (14) to construct the second loss function

[0046]

[0047] Step 3.3: Use formula (15) to construct the total loss function

[0048]

[0049] In formula (13) to formula (15), is the training data, ix represents the xth item, represents the neighbor set of the mth user um, σ is the sigmoid function;

[0050] Step 4: Based on the training data O, the multimodal graph contrast learning model is trained using the gradient descent method, and the total loss function is calculated. When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thereby obtaining the optimal multimodal image contrast learning model for the image feature matrix of the image modality. Text feature matrix for text modality User embed Project Embed Dense vector representation and Process and output each user's score for each item, so as to select the top items for recommendation to each user.

[0051] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the multimedia recommendation method, and the processor is configured to execute the program stored in the memory.

[0052] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the multimedia recommendation method when the computer program is executed by a processor.

[0053] Compared with the existing technology, the present invention has the following beneficial effects:

[0054] 1. The present invention constructs a graph convolutional network module specifically for propagating the embedding of users and items, which alleviates the problem of multimodal noise pollution.

[0055] 2. The present invention enhances the representation of users and items through a separate graph learning model and contrastive learning to better capture collaborative signals and multimodal preferences, and weaken the impact of multimodal noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1It is a schematic diagram of the recommendation method based on multimodal graph contrast learning of the present invention. DETAILED DESCRIPTION

[0057] In this embodiment, a recommendation method based on multimodal graph contrastive learning is to construct a graph convolution module to capture collaborative signals and multimodal user preferences, and then use contrastive learning to eliminate noise pollution in multimodal user preference modeling. Finally, in order to ensure sufficient learning of the model, this embodiment uses an alternating training strategy to optimize them separately. Specifically, Figure 1 As shown, the steps are as follows:

[0058] Step 1: Data collection and preprocessing;

[0059] Step 1.1: Construct a set of product items, denoted as I = {i1,i2,…,in,…i |I|}, where in represents the nth item and |I| represents the total number of items;

[0060] Construct a user set, denoted as U = {u1,u2,…,um,…,u |U|}, where um represents the mth user; |U| represents the total number of users;

[0061] Constructing user-item interaction graph using implicit feedback data in dataset in, Indicates whether there is an interaction between the mth user um and the nth item in. If so, let Otherwise, let

[0062] Map the mth user um and the nth item in to user embeddings and project embedding The corresponding embedding vectors of the mth user um in image mode V and text mode T are and

[0063] Step 1.2: Deep feature extraction:

[0064] The image v corresponding to the nth product item in n Input into the pre-trained VGG16 model for processing to obtain image features Thus, the image feature matrix of image modality V is constructed using formula (1): dV is the dimension of image features:

[0065]

[0066] The text t corresponding to the nth product item in nInput into the pre-trained Sentence2Vec model for processing to obtain text features Thus, the text feature matrix of text modality T is constructed using formula (2): dT is the dimension of text features:

[0067]

[0068] Step 2: Construct a multimodal graph contrastive learning model, including graph convolution layer, contrastive learning layer, and prediction layer;

[0069] Step 2.1: Processing of the graph convolutional layer:

[0070] Step 2.2.1: In order to model clean high-order collaborative signals, the present invention performs a user-item interaction graph without combining multimodal features. Perform graph convolution operations. Using equations (3) and (4), we can get the embeddings of the mth user um and the nth item in at the lth graph convolution layer:

[0071]

[0072]

[0073] In formula (3) and formula (4), and They represent the neighbor sets of the mth user um and the nth item in respectively. and They represent the number of neighbors of the m-th user um and the number of neighbors of the n-th item in respectively; is the embedding of the nth item in the l-1th graph convolutional layer. When l = 1, let is the embedding of the mth user um in the l-1th graph convolutional layer. When l = 1, let

[0074] Step 2.2.2: Under the image mode V or text mode T, use equations (5) and (6) to obtain the mth user u: m and the embedding of the nth item in the lth graph convolutional layer under multimodal modal To incorporate multimodal information of historical interactions into node representations:

[0075]

[0076]

[0077] In formula (5) and formula (6), α is a hyperparameter, TR represents transposition, modal represents multimodality, and modal = V or T, is the weight transformation matrix of the multimodal modal, d moda l is the dimension of the multimodal feature, d is the embedding size; The features of the multimodal modal representing the nth item, represents the embedding of the l-1th graph convolutional layer of the mth user um in the multimodal modal, and represents the mth user u m The embedding vector of the l-1th layer under the image modality V, represents the mth user u m In the embedding vector of the l-1th layer of the text modality T, when l=1, let represents the embedding of the nth item in the l-1th graph convolutional layer under multimodal modal; and represents the image features of the nth item in, represents the text feature of the nth item in. When l = 1, let

[0078] Step 2.2.3: Use equations (7) and (8) to get the mth user u m and the embedding of the nth item in the l+1th graph convolutional layer under multimodal modal

[0079]

[0080]

[0081] Step 2.2.4: Process according to the process of step 2.2.2-step 2.2.3, so that the Lth layer outputs the features of the mth user um Features of the mth user um in multimodal modal Features of the nth item in

[0082] Step 2.3, processing of the contrastive learning layer:

[0083] Step 2.3.1: Construct the user contrast loss function through formula (9)

[0084]

[0085] In formula (9), represents the feature of the j-th user uj under the multimodal modal of the L-th layer, τ is a hyperparameter; Step 2.3.2, construct the item contrast loss function through formula (10)

[0086]

[0087] In formula (10), Represents the features of the k-th user ik under multimodal modal;

[0088] Step 2.3.3: In order to combine the node features of users and items in visual and textual modalities, the contrast loss function is constructed through formula (11):

[0089]

[0090] Step 2.4: Processing of the prediction layer:

[0091] Formula (12) is used to calculate the preference score between the mth user um and the nth item in

[0092]

[0093] In formula (12), λ is a hyperparameter;

[0094] Step 3: In order to optimize the multimodal graph contrastive learning model, the widely used Bayesian personalized ranking loss is used as the basic optimization objective to update the representation of users and items. The Bayesian personalized ranking loss assumes that users prefer historically interacted items rather than untouched items. Construct the loss function of the multimodal graph learning model:

[0095] Step 3.1: Use formula (13) to construct the first loss function

[0096]

[0097] Step 3.2: Use formula (14) to construct the second loss function

[0098]

[0099] Step 3.3: Use formula (15) to construct the total loss function

[0100]

[0101] In formula (13) to formula (15), is the training data, ix represents the xth item, represents the neighbor set of the mth user um, σ is the sigmoid function;

[0102] Step 4: Based on the training data O, the multimodal graph contrast learning model is trained using the gradient descent method, and the total loss function is calculated. When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thereby obtaining the optimal multimodal image contrast learning model for the image feature matrix of the image modality. Text feature matrix for text modality User embed Project Embed Dense vector representation and Process and output each user's score for each item, so as to select the top items for recommendation to each user.

[0103] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above-mentioned multimedia recommendation method. The processor is configured to execute the program stored in the memory.

[0104] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the multimedia recommendation method are executed.

Claims

1. A multimedia recommendation method based on multimodal graph contrast learning, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing; Step 1.1: Build a set of product items, denoted as ,in, represents the nth item, Indicates the total number of items; Construct a user set, denoted as ,in, represents the mth user; Indicates the total number of users; Constructing a user-project bipartite graph ,in, Indicates the mth user and the nth item Is there any interaction between them? If so, let , otherwise, let ; The mth user and the nth item Mapped to user embedding and project embedding , the mth user The corresponding embedding vectors in image mode V and text mode T are and ; Step 1.2: Deep feature extraction: The nth item Corresponding image Input into the pre-trained VGG16 model for processing to obtain image features , and then use formula (1) to construct the image feature matrix of image modality V , is the dimension of image features: (1) The nth item Corresponding text Input into the pre-trained Sentence2Vec model for processing to obtain text features , thus constructing the text feature matrix of text modality T using formula (2) , is the dimension of text features: (2) Step 2: Construct a multimodal graph contrastive learning model, including graph convolution layer, contrastive learning layer, and prediction layer; Step 2.1: Processing of the graph convolution layer: Step 2.2.1: Use equation (3) and equation (4) to get the mth user and the nth item In the Embedding of the graph convolutional layer: (3) (4) In formula (3) and formula (4), and Respectively represent the mth user and the nth item The neighbor set of and Respectively represent the mth user The number of neighbors and the nth item The number of neighbors of It is the nth item In the The embedding of the convolutional layer, when l=1, is ; is the mth user In the The embedding of the convolutional layer, when l=1, is ; Step 2.2.2: Under the image mode V or text mode T, use equations (5) and (6) to obtain the mth user and the nth item In multimodal modal Embedding of layer graph convolutional layers , : (5) (6) In formula (5) and formula (6), is a hyperparameter, TR stands for transpose; represents multimodality, and =V or T, Is multimodal The weight transformation matrix, Is multimodal The dimension of is the embedding size; The features of the multimodal modal representing the nth item, Indicates the mth user In multimodal modal The embedding of the convolutional layer, and ; Indicates the mth user In the image mode V The embedding vector of the layer, Indicates the mth user In the text mode T The embedding vector of the layer, when l=1, let ; = ; Indicates the nth item In multimodal modal The embedding of the graph convolutional layer; and ; Indicates the nth item The image features, Indicates the nth item Text features, when l=1, let ; = ; Step 2.2.3: Use equations (7) and (8) to get the mth user and the nth item In multimodal modal Embedding of layer graph convolutional layers , : (7) (8) Step 2.2.4: Process according to the process of steps 2.2.2 to 2.2.3, so that the Lth layer outputs the mth user Features The mth user Features in multi-modal 、nth item Features ; Step 2.3, processing of the contrastive learning layer: Step 2.3.1: Construct the user contrast loss function through formula (9) : (9) In formula (9), represents the jth user Features under the multimodal modal at layer L, is a hyperparameter; Step 2.3.2: Construct the project comparison loss function through formula (10) : (10) In formula (10), Indicates Users Features under multimodal modal; Step 2.3.3: Construct the contrast loss function using formula (11) : (11) Step 2.4: Processing of the prediction layer: Using formula (12) to calculate the mth user and the nth item Preference score between : (12) In formula (12), is a hyperparameter; Step 3: Construct the loss function of the multimodal graph contrast learning model: Step 3.1: Use formula (13) to construct the first loss function : (13) Step 3.2: Use formula (14) to construct the second loss function : (14) Step 3.3: Use formula (15) to construct the total loss function : (15) In formula (13) to formula (15), is the training data, Indicates projects, Indicates the mth user The neighbor set of is the sigmoid function; Step 4: Based on training data , the multimodal graph contrast learning model is trained using the gradient descent method, and the total loss function is calculated When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thereby obtaining the optimal multimodal image contrast learning model for the image feature matrix of the image modality. , text feature matrix of text modality , User Embed , Project Embedding , dense vector representation and Process and output each user's score for each item, so as to select the top items for recommendation to each user.

2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the multimedia recommendation method according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimedia recommendation method according to claim 1 are executed.

Images