A recommendation method and system based on multi-modal data augmentation contrast learning

Abstract

The present invention discloses a recommendation method and system based on multimodal data-enhanced contrastive learning. The method comprises the following steps: obtaining a recommendation dataset, constructing a user-item adjacency matrix and a user-item knowledge graph, then constructing a set of entity representation vectors and a word vector embedding matrix, performing feature extraction operations on each of these, obtaining a set of feature representation vectors for the items in the entity and a set of representation vectors for the multimodal data corresponding to the items, and then performing dimensionality reduction on each of these. The method also comprises constructing a contrastive learning loss function, performing contrastive learning on the reduced representation vector sets, and obtaining an optimized set of representation vectors for the items in the entity and a set of representation vectors for the users when the contrastive learning loss is less than a preset threshold. The method also comprises calculating a matching score for each user with each item, and recommending items with matching scores greater than the preset threshold to the user. The method not only effectively alleviates the problem of data sparsity but also reduces the cost of data enhancement operations, thereby improving the robustness of the model to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of personalized recommendation, more particularly, to a recommendation method and system based on multi-modal data enhancement contrast learning. BACKGROUND

[0002] Traditional recommendation systems mainly recommend based on user behavior data, but there are problems such as data sparsity, which leads to low recommendation accuracy. Contrast learning is a self-supervised learning and also an unsupervised learning. By judging which data points are similar or not similar through model training, the general features of the data set without labels are extracted. The core idea of contrast learning is some data enhancement methods, which generate noise samples, and through a series of encoding, projection and other operations, construct contrast loss, calculate the distance between samples, and by adjusting the loss, the distance between positive samples is narrowed and the distance between negative samples is enlarged.

[0003] The three main data enhancement methods in this field at present are: sequence-based enhancement, graph-based enhancement and feature-based enhancement. However, most of the existing data enhancement methods use edge discarding, masking and other methods, which may exacerbate data sparsity, and as the number of items increases, the operation cost required by data enhancement becomes more and more high. SUMMARY

[0004] In order to overcome the defects of the prior art that may exacerbate data sparsity and increase operation cost, the present application provides a recommendation method and system based on multi-modal data enhancement contrast learning, which can not only effectively alleviate the problem of data sparsity, but also reduce the cost of data enhancement operation, and to a certain extent, improve the robustness of the model.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows:

[0006] The present application provides a recommendation method based on multi-modal data enhancement contrast learning, which comprises:

[0007] S1: obtaining a recommendation data set;

[0008] S2: preprocessing the recommendation data set, constructing a user-item adjacency matrix and a user-item knowledge graph;

[0009] S3: constructing an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph;

[0010] S4: performing feature extraction operation on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of the items in the entity;

[0011] S5: obtaining a set of representation vectors of the multi-modal data corresponding to the item in the entity by using the word vector embedding matrix;

[0012] S6: dimensionally reducing the set of feature representation vectors of the item in the entity and the set of representation vectors of the multi-modal data corresponding to the item respectively to obtain a set of representation vectors of the item in the entity after dimension reduction and a set of representation vectors of the multi-modal data corresponding to the item after dimension reduction;

[0013] S7: constructing a contrast learning loss function to perform contrast learning on the set of representation vectors of the item in the entity after dimension reduction and the set of representation vectors of the multi-modal data corresponding to the item after dimension reduction, and obtaining an optimized set of representation vectors of the item in the entity when the contrast learning loss is less than a preset threshold; and obtaining an optimized set of representation vectors of the user in the entity according to the optimized set of representation vectors of the item in the entity;

[0014] S8: calculating a matching score of each user to each item according to the optimized set of representation vectors of the user in the entity and the set of representation vectors of the item, and recommending the item with a matching score greater than a preset threshold to the user.

[0015] Preferably, the recommendation data set includes a user set, an item set, an item description information set, a word set of users and items, and a history record of user scoring items, and a positive sample set and a negative sample set are constructed according to the item set and the item description information set.

[0016] The user set is denoted as U={u1,u2,…,u m}, u m m represents the mth user, the item set is denoted as I={i1,i2,…,i n}, i n n represents the nth item, the item description information set is denoted as S={s1,s2,…,s n}, s n n represents the description information of the nth item.

[0017] Preferably, the nth item i n in the item set and the description information s n of the nth item in the item description information set are positive sample pairs, the nth item i n in the item set and other description information except the description information s n of the nth item in the item description information set are negative sample pairs; all positive sample pairs form a positive sample set, and all negative sample pairs form a negative sample set.

[0018] Preferably, in S2, the user-item adjacency matrix is constructed as follows:

[0019] According to the historical records of the user rating items, the user-item interaction matrix is ​​recorded as:

[0020]

[0021] in, The value is 0 or 1. represents the mth user u m and item i n There are interactive relationships between them, such as click, watch, browse and other behaviors; represents the mth user u m and item i n There is no interactive relationship between them;

[0022] The user-item interaction matrix is ​​used to obtain the user-item adjacency matrix:

[0023]

[0024] in,

[0025] Preferably, in S2, constructing a user-item knowledge graph includes:

[0026] G = (E, R, T) represents the user-item knowledge graph, and the entity set E = {e1, e2, ..., e k}, e k Represents the kth entity; constructs a relation set R = {r1, r2, ..., r p}, r p represents the p-th relation; k represents the number of entities in the knowledge graph, p represents the number of relations in the knowledge graph, and T represents the set of triples, which are represented in the form of entity-relationship-entity.

[0027] Preferably, in S3, constructing an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph includes:

[0028] Based on the triple set in the user-item knowledge graph, the entity representation vector set is obtained by training the TransR model:

[0029] X e ={X e1 , X e2 ,…,X ek}={X eu , X ei}

[0030] Among them, X ek represents the representation vector of the k-th entity, a set of representation vectors of users in the entity, a set of representation vectors of items in the entity, a representation vector of the mth user u m in the entity, a representation vector of the nth item i n in the entity;

[0031] Based on the set of triples in the user-item knowledge graph, the word vector embedding matrix is obtained by training the TransR model:

[0032] M emb ={m emb1 , m emb2 , …, m embz}

[0033] Where z represents the number of words in the recommendation data set.

[0034] Preferably, in S4, the user-item adjacency matrix and the set of entity representation vectors are respectively subjected to feature extraction operations to obtain a set of feature representation vectors of items in the entity, including:

[0035] The user-item adjacency matrix and the set of entity representation vectors are respectively input into the LightGCN model, and for each layer of the LightGCN model:

[0036]

[0037]

[0038] Where, represents the representation vector of the mth user u m in the lth layer graph propagation, represents the representation vector of the nth item i n in the lth layer graph propagation, and the size of l can be changed to control the strength of information propagation; represents the set of items interacted with the mth user u m , represents the set of users interacted with the nth item i n , represents the total number of items in the set of items interacted with the mth user u m , represents the total number of users in the set of users interacted with the nth item i n ;

[0039] The set of representation vectors of users in the lth layer graph propagation is The set of representation vectors of items in the lth layer graph propagation is

[0040] After the LightGCN model, the feature representation vector set X of the user in the entity eu ′ and the feature representation vector set X of the item ei ′ are respectively:

[0041]

[0042]

[0043] in, Represents the nth item i in the entity n The feature representation vector of Represents the mth user u in the entity m The feature representation vector of , L is the number of propagation layers of the LightGCN model, which is generally 3 by default;

[0044] The set of all entity feature representation vectors is:

[0045] X e ′={X e1 ′,X e2 ′,…,X ek ′}={X ei ′,X eu ′}

[0046] Among them, X ek ′ represents the feature representation vector of the kth entity.

[0047] Preferably, in S5, obtaining a set of representation vectors corresponding to multimodal data of items in the entity using the word vector embedding matrix includes:

[0048] Sort the words of the user and item word sets in the recommendation dataset to obtain the sorted word set, encode the sorted word set using the One-Hot encoding method, and obtain the encoded word vector set O h ={O h1 , O h2 ,…,O hz},in represents the encoded word vector of the zth word, where z represents the number of words;

[0049] For the nth item i n Description information sequence s n ={s n1 , s n2 ,…,s nr}, s nr Represents sequence s n The rth word in , r represents the length of the description information text;

[0050] In the encoded word vector set O h Find s n The encoded word vector corresponding to each word in composes the nth item i n The set of encoded word vectors O n ={O n1 , O n2 ,…,O nr}, O nr Indicates s n The encoded word vector of the rth word in ;

[0051] Embed the word vector into the matrix M emb With the nth item i n Multiply the encoded word vector corresponding to each word in the encoded word vector set to obtain the corresponding word representation vector:

[0052]

[0053] in, Indicates s n The word representation vector of the rth word in the entity is concatenated with all the corresponding word representation vectors to obtain the nth item i in the entity n The multimodal data representation vector is K n :

[0054]

[0055] Among them, ‖ represents the concatenation operator of vector splicing, and the set of representation vectors corresponding to multimodal data of all items in the entity is K = {K1, K2, ..., K n}.

[0056] Preferably, in S6, the dimension reduction of the feature representation vector set of items in the entity and the representation vector set of multimodal data corresponding to the items is performed respectively, and the obtained dimension-reduced representation vector set of items in the entity and the representation vector set of multimodal data corresponding to the items include:

[0057] Inputting a set of feature representation vectors of items in the entity and a set of representation vectors of multimodal data corresponding to the items into a multilayer perceptron, including:

[0058] X mlp =W2σ(W1X ei ′+b1)+b2

[0059] K mlp =W2σ(W1K+b1)+b2

[0060] Among them, X mlp The set of representation vectors of items in the entity after dimensionality reduction is composed of n representation vectors of items in the entity after dimensionality reduction; K mlpThe set of representation vectors representing the multimodal data corresponding to the items in the entity after dimensionality reduction is composed of n representation vectors representing the multimodal data corresponding to the items in the entity after dimensionality reduction, specifically:

[0061] X mlp ={X mlp1 , X mlp2 ,…,X mlpn}

[0062] K mlp ={K mlp1 , K mlp2 ,…,K mlpn}

[0063] Among them, X mlpn Represents the nth item i in the entity after dimensionality reduction n The representation vector, K mlpn Represents the nth item i in the entity after dimensionality reduction n Corresponding to the multimodal data representation vector, W1, W2, b1 and b2 are trainable parameters, and σ is the nonlinear activation function sigmoid.

[0064] Preferably, in S7, the method for determining the contrastive learning loss function includes:

[0065]

[0066] Among them, L final Represents the contrastive learning loss function of all positive samples in the positive sample set, which is used to promote the separation of positive and negative sample pairs, making the positive sample pairs closer and the negative sample pairs farther and farther away. Θ represents the model parameter set, To prevent the model from overfitting, Θ j represents the jth parameter in the model parameter set, w represents the number of parameters, ||·||2 represents the second-order norm; L i represents the contrastive learning loss function of the i-th sample in the positive sample set, which is calculated as:

[0067]

[0068] Among them, L i represents the contrastive learning model loss function of the i-th sample in the positive sample set, sim(*) represents the cosine similarity calculation function, τ represents the temperature parameter, X mlpi Represents the representation vector of the item in the entity after dimensionality reduction corresponding to the i-th sample in the positive sample set, K mlpi Indicates the multimodal data representation vector corresponding to the item in the entity after dimensionality reduction corresponding to the i-th sample in the positive sample set, X mlpq Represents the representation vector of the item in the entity after dimensionality reduction corresponding to the qth negative sample of the i-th sample in the negative sample set, Kmlpi represents the multi-modal data representation vector of the i-th sample in the negative sample set corresponding to the q-th negative sample in the reduced dimension entity.

[0069] Preferably, there is a connection between the optimized representation vector set of the entity in the item and the representation vector of the user, and the optimized representation vector set of the entity in the item is obtained at the same time as the optimized representation vector set of the user in the entity.

[0070] Preferably, the calculation of the matching score of each user for each item includes:

[0071] For the optimized representation vector set of the user in the entity and the representation vector set of the item The matching score is calculated by using the inner product:

[0072]

[0073] wherein, represents the matching score vector set of the user for the item.

[0074] The application also provides a recommendation system based on multi-modal data enhanced contrast learning, which is used to implement the above-mentioned recommendation method, comprising:

[0075] A data acquisition module is configured to acquire a recommendation dataset.

[0076] A data preprocessing module is configured to preprocess the recommendation dataset, construct a user-item adjacency matrix and a user-item knowledge graph.

[0077] A knowledge graph processing module is configured to construct an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph.

[0078] A feature extraction module is configured to perform feature extraction operations on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of the item in the entity.

[0079] A multi-modal data generation module is configured to obtain a representation vector set of multi-modal data corresponding to the item in the entity using the word vector embedding matrix.

[0080] A dimension reduction module is configured to reduce the dimension of the feature representation vector set of the item in the entity and the representation vector set of multi-modal data corresponding to the item respectively to obtain a reduced dimension representation vector set of the item in the entity and a reduced dimension representation vector set of multi-modal data corresponding to the item.

[0081] a model optimization module, configured to construct a contrastive learning loss function, and perform contrastive learning on the set of representation vectors of the items in the entity after dimension reduction and the set of representation vectors of the multi-modal data corresponding to the items, and obtain the set of representation vectors of the items in the entity after optimization when the contrastive learning loss is less than a preset threshold, and obtain the set of representation vectors of the users in the entity after optimization according to the set of representation vectors of the items in the entity after optimization;

[0082] a recommendation module, configured to calculate a matching score of each user for each item according to the set of representation vectors of the users in the entity after optimization and the set of representation vectors of the items, and recommend the items with the matching score greater than a preset threshold to the users.

[0083] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0084] The recommendation method and system based on multi-modal data enhancement contrastive learning provided by the present application first acquire a recommendation dataset, pre-process the recommendation dataset, construct a user-item adjacency matrix and a user-item knowledge graph, then construct a set of entity representation vectors and a word vector embedding matrix by using the user-item knowledge graph, perform feature extraction operations on the user-item adjacency matrix and the set of entity representation vectors respectively to obtain a set of feature representation vectors of the items in the entity, then obtain a set of representation vectors of the multi-modal data corresponding to the items in the entity by using the word vector embedding matrix, and then respectively reduce the dimension of the set of feature representation vectors of the items in the entity and the set of representation vectors of the multi-modal data corresponding to the items, construct a contrastive learning loss function, and perform contrastive learning on the set of representation vectors of the items in the entity after dimension reduction and the set of representation vectors of the multi-modal data corresponding to the items, and obtain the set of representation vectors of the items in the entity after optimization when the contrastive learning loss is less than a preset threshold, and obtain the set of representation vectors of the users in the entity after optimization according to the set of representation vectors of the items in the entity after optimization. Finally, the matching score of each user for each item is calculated according to the set of representation vectors of the users in the entity after optimization and the set of representation vectors of the items, and the items with the matching score greater than a preset threshold are recommended to the users. The present application can not only effectively alleviate the data sparsity problem, but also reduce the cost of data enhancement operation, and improve the robustness of the model to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 A flowchart of the recommendation method based on multi-modal data enhancement contrastive learning described in Embodiment 1.

[0086] Figure 2 A network architecture diagram of the recommendation method based on multi-modal data enhancement contrastive learning described in Embodiment 2.

[0087] Figure 3A structural schematic diagram of the recommendation system based on multi-modal data enhancement contrast learning described in Embodiment 3. DETAILED DESCRIPTION

[0088] The accompanying drawings are only intended to illustrate, and cannot be understood as a limitation to the patent;

[0089] In order to better illustrate the embodiments, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0090] It can be understood by those skilled in the art that some well-known structures in the drawings and their descriptions may be omitted.

[0091] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0092] Embodiment 1

[0093] The present embodiment provides a recommendation method based on multi-modal data enhancement contrast learning, as shown in Figure 1 , comprising:

[0094] S1: obtaining a recommendation data set;

[0095] S2: preprocessing the recommendation data set, constructing a user-item adjacency matrix and a user-item knowledge graph;

[0096] S3: constructing an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph;

[0097] S4: performing feature extraction operations on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of the items in the entity;

[0098] S5: obtaining a representation vector set of the corresponding multi-modal data of the items in the entity using the word vector embedding matrix;

[0099] S6: dimensionally reducing the feature representation vector set of the items in the entity and the representation vector set of the corresponding multi-modal data of the items respectively to obtain a dimensionally reduced representation vector set of the items in the entity and a dimensionally reduced representation vector set of the corresponding multi-modal data of the items;

[0100] S7: constructing a contrast learning loss function, performing contrast learning on the dimensionally reduced representation vector set of the items in the entity and the dimensionally reduced representation vector set of the corresponding multi-modal data of the items, obtaining an optimized representation vector set of the items in the entity when the contrast learning loss is less than a preset threshold, and obtaining an optimized representation vector set of the users in the entity according to the optimized representation vector set of the items in the entity;

[0101] S8: calculating a matching score of each user to each item according to the set of representation vectors of the users in the entity and the set of representation vectors of the items, and recommending the items with the matching score greater than a preset threshold to the users.

[0102] In the implementation process, the embodiment pre-processes the obtained recommendation dataset, constructs a user-item adjacency matrix and a user-item knowledge graph, and then uses the user-item knowledge graph to construct a set of entity representation vectors and a word vector embedding matrix. The user-item adjacency matrix and the set of entity representation vectors are subjected to feature extraction operations respectively to obtain a set of feature representation vectors of the items, and the word vector embedding matrix is used to obtain a set of representation vectors of the corresponding multi-modal data of the items in the entity. Then, the set of feature representation vectors of the items in the entity and the set of representation vectors of the corresponding multi-modal data of the items are respectively reduced in dimension, a contrastive learning loss function is constructed, and the set of representation vectors of the items in the entity and the set of representation vectors of the corresponding multi-modal data of the items after dimension reduction are subjected to contrastive learning. When the contrastive learning loss is less than a preset threshold, an optimized set of representation vectors of the items in the entity is obtained, and an optimized set of representation vectors of the users in the entity is obtained according to the optimized set of representation vectors of the items in the entity. Finally, according to the set of representation vectors of the users in the entity and the set of representation vectors of the items, the matching score of each user to each item is calculated, and the items with the matching score greater than a preset threshold are recommended to the users. The embodiment can not only effectively alleviate the data sparsity problem, but also reduce the cost of data enhancement operation, and improve the robustness of the model to a certain extent.

[0103] Embodiment 2

[0104] The application provides a recommendation method based on multi-modal data enhancement contrastive learning, which comprises the following steps:

[0105] S1: obtaining a recommendation dataset;

[0106] The recommendation dataset comprises a user set, an item set, an item description information set, a word set of users and items, and a history record of user rating items, and a positive sample set and a negative sample set are constructed according to the item set and the item description information set.

[0107] The user set is denoted as U={u1,u2,…,u m},u m represents the mth user, the item set is denoted as I={i1,i2,…,i n},i n represents the nth item, the item description information set is denoted as S={s1,s2,…,s n},s n represents the description information of the nth item.

[0108] The nth item i in the item set n The description information s of the nth item in the item description information set n The nth item i in the item set n The description information s of the nth item in the item description information set n The other description information except the description information s of the nth item in the item description information set; all positive sample pairs form a positive sample set, and all negative sample pairs form a negative sample set.

[0109] S2: preprocessing the recommendation data set, constructing a user-item adjacency matrix and a user-item knowledge graph;

[0110] The user-item adjacency matrix is constructed as follows:

[0111] According to the historical record of the user scoring items, the user-item interaction matrix is denoted as:

[0112]

[0113] Wherein, The value is 0 or 1, Indicates that there is an interaction relationship between the mth user u m And the nth item i n Such as click, view, browse and other behaviors; Indicates that there is no interaction relationship between the mth user u m And the nth item i n

[0114] The user-item adjacency matrix is obtained by using the user-item interaction matrix:

[0115]

[0116] Wherein,

[0117] The user-item knowledge graph is constructed as follows:

[0118] The user-item knowledge graph is denoted as G=(E,R,T), and the entity set E={e1,e2,…,e k} is constructed according to the user set and the item set, e k Indicates the kth entity; the relationship set R={r1,r2,…,r p} is constructed according to the historical record of the user scoring items, and r p Indicates the pth relationship; k represents the number of entities in the knowledge graph, p represents the number of relationships in the knowledge graph, and T represents the triple set in the form of entity-relation-entity.

[0119] ​S3: constructing an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph;

[0120] Based on the triple set in the user-item knowledge graph, an entity representation vector set is obtained by training using a TransR model:

[0121] X e ={X e1 ,X e2 ,…,X ek}={X eu ,X ei}

[0122] wherein X ek represents the representation vector of the kth entity, represents the representation vector set of the user in the entity, represents the representation vector set of the item in the entity, represents the representation vector of the mth user u m in the entity, represents the representation vector of the nth item i n in the entity;

[0123] Based on the triple set in the user-item knowledge graph, a word vector embedding matrix is obtained by training using a TransR model:

[0124] M emb ={m emb1 ,m emb2 ,…,m embz}

[0125] wherein z represents the number of words in the recommendation data set.

[0126] S4: performing a feature extraction operation on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of the items in the entity;

[0127] The user-item adjacency matrix and the entity representation vector set are respectively input into a LightGCN model, and for each layer of the LightGCN model:

[0128]

[0129]

[0130] wherein, represents the representation vector of the mth user um in the lth layer of graph propagation, represents the representation vector of the nth item i n in the lth layer of graph propagation, and the strength of information propagation can be controlled by changing the size of l; denotes the set of users interacting with the mth user u m denotes the set of items interacting with the mth user u denotes the set of users interacting with the nth item i n denotes the set of items interacting with the nth item i denotes the set of users interacting with the mth user u m denotes the set of items interacting with the mth user u denotes the set of users interacting with the nth item i n denotes the set of users interacting with the nth item i

[0131] The set of representation vectors of the users propagated by the lth layer of the graph is denoted as The set of representation vectors of the items propagated by the lth layer of the graph is denoted as

[0132] After the LightGCN model, the set of feature representation vectors of the users in the entity X eu ′ and the set of feature representation vectors of the items X ei ′ are respectively:

[0133]

[0134]

[0135] wherein, denotes the feature representation vector of the nth item i n in the entity, denotes the feature representation vector of the mth user u m in the entity, and L is the number of propagation layers of the LightGCN model, which is generally 3 by default.

[0136] The set of all entity feature representation vectors is:

[0137] X e ′={X e1 ′,X e2 ′,…,X ek ′}={X ei ′,X eu ′}

[0138] wherein, X ek ′ denotes the feature representation vector of the kth entity.

[0139] S5: obtaining the set of representation vectors of the multi-modal data corresponding to the items in the entity by using the word vector embedding matrix;

[0140] sorting the words of the user and item word set in the recommendation data set to obtain a sorted word set, encoding the sorted word set using One-Hot encoding to obtain an encoded word vector set O h ={O h1 ,Oh2 ,..., O hz}, wherein represents the encoding word vector of the zth word, z represents the number of words;

[0141] For the nth item i n , the description information sequence s n = {s n1 , s n2 ,..., s nr}, s nr represents the rth word in the sequence s n , and r represents the length of the description information text;

[0142] In the encoding word vector set O h , find the encoding word vector corresponding to each word in s n , and form the encoding word vector set O n of the nth item i n = {O n1 , O n2 ,..., O nr}, O nr represents the encoding word vector of the rth word in s n ;

[0143] Multiply the word vector embedding matrix M emb by the encoding word vector corresponding to each word in the encoding word vector set of the nth item i n , to obtain the corresponding word representation vector:

[0144]

[0145] wherein, represents the word representation vector of the rth word in s n , and all corresponding word representation vectors are concatenated to obtain the multi-modal data representation vector corresponding to the nth item i n in the entity, denoted as K n :

[0146]

[0147] wherein, ‖ represents the connection operator of vector concatenation, and the set of multi-modal data representation vectors corresponding to all items in the entity is K = {K1, K2,..., K n}.

[0148] S6: Reduce the feature representation vector set of the items in the entity and the multi-modal data representation vector set corresponding to the items respectively to obtain the reduced representation vector set of the items in the entity and the multi-modal data representation vector set corresponding to the items;

[0149] The feature representation vector set of the items in the entity and the representation vector set of the multi-modal data corresponding to the items are input to a multi-layer perception, specifically:

[0150] X mlp = W2σ(W1X ei +b1) + b2

[0151] K mlp = W2σ(W1K + b1) + b2

[0152] wherein X mlp represents the representation vector set of the items in the entity after dimension reduction, consisting of n representation vectors of the items in the entity after dimension reduction; K mlp represents the representation vector set of the multi-modal data corresponding to the items in the entity after dimension reduction, consisting of n representation vectors of the multi-modal data corresponding to the items in the entity after dimension reduction, specifically:

[0153] X mlp = {X mlp1 , X mlp2 , …, X mlpn}

[0154] K mlp = {K mlp1 , K mlp2 , …, K mlpn}

[0155] wherein X mlpn represents the representation vector of the nth item i n in the entity after dimension reduction, K mlpn represents the representation vector of the multi-modal data corresponding to the nth item i n in the entity after dimension reduction, W1, W2, b1 and b2 are trainable parameters, and σ is a nonlinear activation function sigmoid.

[0156] S7: constructing a contrastive learning loss function, performing contrastive learning on the representation vector set of the items in the entity after dimension reduction and the representation vector set of the multi-modal data corresponding to the items, and obtaining an optimized representation vector set of the items in the entity when the contrastive learning loss is less than a preset threshold; obtaining an optimized representation vector set of the users in the entity according to the optimized representation vector set of the items in the entity;

[0157] The contrastive learning loss function is:

[0158]

[0159] wherein L finalrepresents the contrastive learning loss function of all positive samples in the positive sample set, used to promote the separation of positive and negative sample pairs, so that the positive sample pairs are closer and closer, and the negative sample pairs are farther and farther apart. represents the model parameter set, for preventing model overfitting, Θ j represents the jth parameter in the model parameter set, w represents the number of parameters, and ||·||2 represents the 2-norm; L i represents the contrastive learning loss function of the ith sample in the positive sample set, and the calculation method is:

[0160]

[0161] wherein, L i represents the contrastive learning model loss function of the ith sample in the positive sample set, sim(*) represents the cosine similarity calculation function, τ represents the temperature parameter, X mlpi represents the representation vector of the item in the reduced entity corresponding to the ith sample in the positive sample set, K mlpi represents the multi-modal data representation vector corresponding to the item in the reduced entity corresponding to the ith sample in the positive sample set, X mlpq represents the representation vector of the item in the reduced entity corresponding to the qth negative sample of the ith sample in the negative sample set, K mlpi represents the multi-modal data representation vector corresponding to the item in the reduced entity corresponding to the qth negative sample of the ith sample in the negative sample set.

[0162] The optimized representation vector set of the item in the entity and the representation vector set of the user are related, and the optimized representation vector set of the item in the entity is obtained at the same time. The optimized representation vector set of the user in the entity is also obtained.

[0163] S8: According to the optimized representation vector set of the user in the entity and the representation vector set of the item, the matching score of each user to each item is calculated, and the item with a matching score greater than a preset threshold is recommended to the user.

[0164] For the optimized representation vector set of the user in the entity and the representation vector set of the item The inner product is used to calculate the matching score:

[0165]

[0166] wherein, represents the matching score vector set of the user to the item.

[0167] In the specific implementation process, for example, Figure 2As shown, the network architecture diagram of the recommendation method based on multi-modal data enhancement contrast learning of the embodiment, the entity representation vector set includes m user representation vectors and n item representation vectors, which are input into the LightGCN model for feature extraction to obtain item feature vectors and user feature vectors, then the item feature vectors and the item corresponding multi-modal vectors are input into the multi-layer perception dimension reduction, and then the contrast loss is calculated, the loss is optimized to obtain the best item vector and user vector. Calculate the matching score of each user to each item, and recommend the items with a matching score greater than a preset threshold to the user.

[0168] Embodiment 3

[0169] The embodiment provides a recommendation system based on multi-modal data enhancement contrast learning, which is used to implement the method of embodiments 1 or 2, as shown in Figure 3 As shown, it comprises:

[0170] A data acquisition module is configured to acquire a recommendation dataset.

[0171] A data preprocessing module is configured to preprocess the recommendation dataset, construct a user-item adjacency matrix and a user-item knowledge graph.

[0172] A knowledge graph processing module is configured to construct an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph.

[0173] A feature extraction module is configured to perform feature extraction operations on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of items in the entity.

[0174] A multi-modal data generation module is configured to obtain a representation vector set of item corresponding multi-modal data in the entity using the word vector embedding matrix.

[0175] A dimension reduction module is configured to reduce the dimension of the feature representation vector set of items in the entity and the representation vector set of item corresponding multi-modal data respectively to obtain a reduced representation vector set of items in the entity and a reduced representation vector set of item corresponding multi-modal data.

[0176] A model optimization module is configured to construct a contrast learning loss function, perform contrast learning on the reduced representation vector set of items in the entity and the reduced representation vector set of item corresponding multi-modal data, and obtain an optimized representation vector set of items in the entity when the contrast learning loss is less than a preset threshold, and obtain an optimized representation vector set of users in the entity according to the optimized representation vector set of items in the entity.

[0177] The recommendation module is configured to calculate a matching score of each user to each item according to the optimized set of representation vectors of the users and the set of representation vectors of the items, and recommend the items with the matching score greater than a preset threshold to the user.

[0178] Identical or similar reference numerals can correspond to identical or similar components;

[0179] The terms describing the positional relationship in the drawings are only used for illustrative description, and should not be understood as a limitation on the patent;

[0180] Obviously, the above embodiments of the present application are merely exemplary for clearly illustrating the present application, and are not intended to limit the implementation manners of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and also impossible to enumerate all the implementation manners. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A recommendation method based on multimodal data enhanced contrastive learning, characterized in that: The method comprises: S1: Get the recommended dataset; S2: Preprocess the recommendation dataset to construct a user-item adjacency matrix and a user-item knowledge graph; S3: Utilize the user-item knowledge graph to construct an entity representation vector set and a word vector embedding matrix; S4: performing feature extraction operations on the user-item adjacency matrix and the entity representation vector set respectively to obtain a feature representation vector set of items in the entity; S5: Using the word vector embedding matrix, obtain a set of representation vectors corresponding to the multimodal data of the items in the entity; S6: reducing the dimensions of the feature representation vector set of the items in the entity and the representation vector set of the multimodal data corresponding to the items, respectively, to obtain the representation vector set of the items in the entity and the representation vector set of the multimodal data corresponding to the items after dimension reduction; S7: Construct a contrastive learning loss function to perform contrastive learning on the set of representation vectors of the items in the entity after dimensionality reduction and the set of representation vectors of the multimodal data corresponding to the items. When the contrastive learning loss is less than a preset threshold, an optimized set of representation vectors of the items in the entity is obtained. Based on the optimized set of representation vectors of the items in the entity, an optimized set of representation vectors of the users in the entity is obtained. S8: Calculate the matching score of each user to each item based on the optimized set of user representation vectors and item representation vectors in the entity, and recommend items with matching scores greater than a preset threshold to the user.

2. The recommendation method based on multimodal data enhanced contrastive learning according to claim 1, characterized in that: In S1, the recommendation dataset includes a user set, an item set, an item description information set, a word set of users and items, and a historical record of user rating items. A positive sample set and a negative sample set are constructed based on the item set and the item description information set. The user set is recorded as , Indicates the users, and the item set is recorded as , Indicates the items, and the item description information set is recorded as , Indicates the Description of an item.

3. The recommendation method based on multimodal data enhanced contrastive learning according to claim 2, characterized in that: In S2, the construction of the user-item adjacency matrix is ​​specifically as follows: According to the historical records of the user rating items, the user-item interaction matrix is ​​recorded as: in The value is 0 or 1. Indicates the users Hedi items There is an interactive relationship between them; Indicates the users Hedi items There is no interactive relationship between them; The user-item interaction matrix is ​​used to obtain the user-item adjacency matrix: in .

4. The recommendation method based on multimodal data enhanced contrastive learning according to claim 2, characterized in that: In S2, constructing the user-item knowledge graph includes: by Represents the user-item knowledge graph and constructs an entity set based on the user set and item set , Indicates the Entities; build a relationship set based on the historical records of user-rated items , Indicates the a relationship; represents the number of entities in the knowledge graph, represents the number of relations in the knowledge graph, Represents a set of triples, expressed in the form of entity-relationship-entity.

5. The recommendation method based on multimodal data enhanced contrastive learning according to claim 4, characterized in that: In S3, constructing an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph includes: Based on the triple set in the user-item knowledge graph, the entity representation vector set is obtained by training the TransR model: in, Indicates the The representation vector of an entity, represents the set of representation vectors of users in the entity, A set of representation vectors representing items in the entity, , Indicates the entity users The representation vector of Indicates the entity items The representation vector of Based on the triple set in the user-item knowledge graph, the word vector embedding matrix is ​​obtained by training the TransR model: in, Indicates the number of words in the recommendation dataset.

6. The recommendation method based on multimodal data enhanced contrastive learning according to claim 5, characterized in that: In S4, feature extraction operations are performed on the user-item adjacency matrix and the entity representation vector set respectively, and the feature representation vector set of the items in the entity is obtained, including: The user-item adjacency matrix and entity representation vector set are input into the LightGCN model respectively. For each layer of the LightGCN model: in, Indicates the Layer Graph Propagation users The representation vector of Indicates the Layer Graph Propagation items The representation vector of Indicates the users A collection of interactive items. Indicates the items The set of interacting users, Indicates the users The total number of items in the interacted item set, Indicates the items The total number of users in the interactive user set; No. The representation vector set of layer graph propagation users is , No. The representation vector set of layer graph propagation items is ; After the LightGCN model, the feature representation vector set of items in the entity for: in, , Indicates the entity items The feature representation vector of is the number of propagation layers of the LightGCN model.

7. The recommendation method based on multimodal data enhanced contrastive learning according to claim 5, characterized in that: In S5, obtaining a set of representation vectors corresponding to multimodal data of items in the entity using the word vector embedding matrix includes: Sort the words of the user and item word sets in the recommendation dataset to obtain the sorted word set, encode the sorted word set using the One-Hot encoding method, and obtain the encoded word vector set ,in , indicating the The encoded word vector of each word, Indicates the number of words; For the items Description information sequence , Representation sequence Middle words, Indicates the length of the description information text; In the set of encoded word vectors Search The encoded word vector corresponding to each word in composes the first items The set of encoded word vectors , express Middle The encoded word vector of each word; Embed word vectors into a matrix With the items Multiply the encoded word vector corresponding to each word in the encoded word vector set to obtain the corresponding word representation vector: in, express Middle The word representation vector of each word is concatenated All word representation vectors in the entity are obtained items The corresponding multimodal data representation vector is recorded as : in, The concatenation operator represents vector concatenation. The set of representation vectors for multimodal data corresponding to all items in the entity is .

8. The recommendation method based on multimodal data enhanced contrastive learning according to claim 1, characterized in that: In S6, the dimension of the feature representation vector set of the items in the entity and the representation vector set of the multimodal data corresponding to the items are reduced respectively, and the representation vector set of the items in the entity and the representation vector set of the multimodal data corresponding to the items after dimension reduction are obtained, including: Inputting a set of feature representation vectors of items in the entity and a set of representation vectors of multimodal data corresponding to the items into a multilayer perceptron, including: in, Represents the set of vectors representing items in the entity after dimensionality reduction, represented by The representation vectors of items in the entity after dimensionality reduction; Represents the set of representation vectors corresponding to the multimodal data of the items in the entity after dimensionality reduction, which is represented by The multimodal data representation vectors corresponding to the items in the entity after dimensionality reduction are composed of: in, Represents the entity after dimensionality reduction items The representation vector of Represents the entity after dimensionality reduction items Corresponding to the multimodal data representation vector, 、 、 and are training parameters, is the nonlinear activation function sigmoid.

9. The recommendation method based on multimodal data enhanced contrastive learning according to claim 1, characterized in that: In S7, the method for determining the contrastive learning loss function includes: in, represents the contrastive learning loss function of all positive samples in the positive sample set, represents the model parameter set, , Indicates the first parameter in the model parameters, Indicates the number of parameters, represents the 2nd-order norm; Indicates the first The contrastive learning loss function of samples is calculated as: in, Indicates the first The loss function of the contrastive learning model for samples is, is the cosine similarity calculation function, represents the temperature parameter, Indicates the first The representation vector of the item in the entity after dimensionality reduction corresponding to the sample, Indicates the first The multimodal data representation vectors of items in the entity after dimension reduction corresponding to the samples, express Middle The representation vector of the item in the entity after dimensionality reduction corresponding to the sample, express Middle The multimodal data representation vectors of the items in the entity after dimensionality reduction corresponding to the samples.

10. The recommendation method based on multimodal data enhanced contrastive learning according to claim 9, characterized in that: In S8, calculating the matching score of each user for each item includes: For the optimized user representation vector set in the entity and the set of vectors representing items , use the inner product to calculate its matching score: in, Represents a set of matching score vectors between users and items.

11. A recommendation system based on multimodal data enhanced contrastive learning, used to implement the method according to any one of claims 1 to 10, characterized in that: include: Data acquisition module, used to obtain recommended data sets; A data preprocessing module is used to preprocess the recommendation dataset and construct a user-item adjacency matrix and a user-item knowledge graph; A knowledge graph processing module, configured to construct an entity representation vector set and a word vector embedding matrix using the user-item knowledge graph; A feature extraction module is used to perform feature extraction operations on the user-item adjacency matrix and the entity representation vector set to obtain a feature representation vector set of items in the entity; A multimodal data generation module, configured to use the word vector embedding matrix to obtain a set of representation vectors corresponding to the multimodal data of items in the entity; a dimensionality reduction module, configured to reduce the dimensions of the feature representation vector set of items in the entity and the representation vector set of the multimodal data corresponding to the items, respectively, to obtain the representation vector set of the items in the entity and the representation vector set of the multimodal data corresponding to the items after dimensionality reduction; The model optimization module is used to construct a contrastive learning loss function and perform contrastive learning on the set of representation vectors of the items in the entity after dimensionality reduction and the set of representation vectors of the multimodal data corresponding to the items. When the contrastive learning loss is less than a preset threshold, the optimized set of representation vectors of the items in the entity is obtained. Based on the optimized set of representation vectors of the items in the entity, the optimized set of representation vectors of the users in the entity is obtained. The recommendation module is used to calculate the matching score of each user to each item based on the optimized set of user representation vectors and the set of item representation vectors in the entity, and recommend items with matching scores greater than a preset threshold to the user.

Images