Multi-modal knowledge graph recommendation method and device based on momentum update contrastive learning framework
By using a momentum update contrastive learning framework, a multimodal knowledge graph dataset is constructed and multi-task learning is performed. This solves the problem of inconsistency between visual and textual representations in multimodal data fusion and improves the accuracy and effectiveness of knowledge graph recommendations.
Patent Information
- Application Number
- CN202311162961.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-11
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2043-09-11
AI Technical Summary
Existing technologies suffer from inconsistencies between visual and textual representations when integrating multimodal data into knowledge graphs. Furthermore, existing methods fail to effectively utilize multimodal information to calculate node similarity, resulting in poor recommendation performance.
We adopt a momentum-updated contrastive learning framework, construct a multimodal knowledge graph dataset, introduce a multimodal attention mechanism, utilize the momentum contrastive learning loss function, generate subviews and perform multi-task learning, and combine ordinary contrastive learning signals and momentum contrastive learning signals to optimize the recommendation model.
It effectively solves the inconsistency between visual and textual representations, improves the utilization efficiency of multimodal information, and enhances the accuracy and effectiveness of knowledge graph recommendations.
Smart Images
Figure CN117151199B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, in particular to a multi-modal knowledge graph recommendation method and device based on momentum update contrast learning framework. BACKGROUND
[0002] In recent years, the application of knowledge graph in the field of recommendation system presents a significant growth trend. This increase in interest can be attributed to the ability of knowledge graph to effectively convey the relationship between items through rich attributes. Some researchers have begun to try to integrate multi-modal data, such as text and images, into knowledge graph, although the achievements are still limited. These attempts mainly rely on image models to extract visual representations and text models to extract textual representations. However, this approach raises concerns about potential inconsistencies between the two types of representations. In addition, existing methods for calculating node similarity using multi-modal information have not made significant progress or improvements.
[0003] Inspired by contrast learning techniques in other fields, some research has begun to improve the performance of deep learning models by designing contrast learning loss functions on self-supervised signals. The goal of the contrast learning loss function is to maximize the similarity between positive sample pairs generated by data augmentation of the same sample. However, the introduction of contrast learning techniques into knowledge graph raises a series of new problems, such as how to construct effective positive sample pairs to support data augmentation methods, and how to increase the number of negative samples while avoiding misjudgment of negative samples. SUMMARY
[0004] The present application aims to provide a multi-modal knowledge graph recommendation method and device based on momentum update contrast learning framework to solve the above problems existing in the prior art.
[0005] In a first aspect, a multi-modal knowledge graph recommendation method based on momentum update contrast learning framework is provided, comprising the following steps:
[0006] S1, determine an initial entity, collect text information and picture information associated with the initial entity within a predetermined interval, and construct a multi-modal knowledge graph dataset based on the text information and picture information;
[0007] S2, combine the multi-modal knowledge graph dataset into a connection network composed of a plurality of entity connections, and combine it with the initial entity to obtain second entity data; introduce a multi-modal attention mechanism to represent the conduction coefficient between the head and tail entities in the connection network;
[0008] S3, establish an encoder, import the second entity data into the encoder for dynamic updating to obtain third entity samples, and calculate momentum contrast learning loss based on the third entity samples;
[0009] S4、in the multi-modal knowledge graph dataset on the generation of 2N different sub-views, wherein N is used for ordinary contrast learning, and an ordinary contrast learning signal is output; N is used for momentum contrast learning, and a momentum contrast learning signal is output;
[0010] S5, using the ordinary contrast learning signal and the momentum contrast learning signal, performing multi-task learning.
[0011] In further embodiments of the first aspect, the process of constructing the multi-modal knowledge graph dataset in step S1 comprises:
[0012] extracting the entity name and entity description information of the initial entity, searching the top 10 data related to the initial entity in a search engine using the entity name and entity description information, and obtaining text information and picture information;
[0013] inputting the entity description information and one of the picture information selected at random into a CLIP model, thereby generating a text vector and an image vector of the initial entity;
[0014] passing the text vector and the image vector through a projection function, so that the text vector, the image vector, and the vector generated by the CLIP model have the same dimension;
[0015] wherein the projection function is represented as p(x) = W2(W1x), where x is a feature vector, and W1 and W2 are weight vectors of fully connected layers.
[0016] In further embodiments of the first aspect, in step S2, a multi-modal attention mechanism is introduced to represent the conduction coefficient between the head and tail entities in the connection network, and the multi-modal attention mechanism is represented as:
[0017]
[0018] wherein tanh(*) represents a nonlinear activation function, e hi represents the image vector of the head entity, e ti represents the image vector of the tail entity, e ht represents the text vector of the head entity, e tt represents the text vector of the tail entity, e h represents the head entity vector; e r represents the inter-entity relationship vector; W r represents the transition matrix of the relationship r; represents the transpose of the image vector of the tail entity; represents the transpose of the text vector of the head entity.
[0019] In further embodiments of the first aspect, the multi-modal attention mechanism obtains the attention score by the distance between the head entity and the tail entity, the distance between the tail entity and the multi-modal vector;
[0020] The parameters are regularized by a softmax function:
[0021]
[0022] wherein, represents the connection network between the head entity h and the tail entity t; r represents the inter-entity relationship; r ′ represents the inter-entity relationship after random replacement; t ′ represents the tail entity after random replacement; exp(*) represents an exponential operation with a base of e, the natural logarithm base, performed inside the parentheses;
[0023] Connection network between the head entity h and the tail entity t
[0024] The vector representation of the first layer entity is wherein the function f(*) is a bidirectional connection fusioner, represented as:
[0025]
[0026] wherein LeakyReLU(*) represents an activation function; represents the expression of the head entity h connection network; W1, W2 are weight vectors of the fully connected layer.
[0027] In further embodiments of the first aspect, in a knowledge graph of L layers, the final user u and item i vectors are the concatenation of each layer vector, represented as:
[0028]
[0029] wherein, represents the concatenated user u vector; represents the initial user u vector; represents the Lth layer user u vector; represents the concatenated item i vector; represents the initial item i vector; represents the Lth layer item i vector; ‖…‖ represents the concatenation operation.
[0030] In further embodiments of the first aspect, step S3 further comprises:
[0031] S3-1, two encoders q and k are established, and the parameters in the encoder k are dynamically updated according to the parameters of q and k in the last round of training;
[0032] S3-2. The entity vectors after passing through the k encoder enter a queue of fixed length, and the earliest entity vector in the queue will be dequeued.
[0033] S3-3. When calculating the momentum contrastive learning loss, entities that have passed through two encoders are used as positive samples, and entities in the queue are used as negative samples.
[0034] θ k ←mθ k +(1-m)θ q ;
[0035] In the formula, θ k θ represents the parameters in encoder k; q q represents the parameters in encoder q; m represents the momentum update parameters.
[0036] In a further embodiment of the first aspect, when performing step S3-3, if an incorrect negative sample is introduced, then negative samples in the queue with a similarity greater than a threshold to positive samples are determined to be positive samples:
[0037]
[0038] In the formula, h represents the weight of the negative sample. q This represents a sample in encoder q; φ represents the sample in encoder k; Sim(*) represents obtaining the sample similarity between two encoders through dot product; φ represents the similarity threshold.
[0039] These erroneous negative samples are not considered when calculating the loss function:
[0040]
[0041] In the formula, This represents the momentum comparison learning objective function; τ represents a positive sample; K represents the queue length; τ represents the hyperparameter controlling the temperature coefficient.
[0042] In a further embodiment of the first aspect, the subview generated on the multimodal knowledge graph dataset in step S4 is represented as follows:
[0043]
[0044] In the formula, It is a set of nodes, M∈{0,1} |ε| It is the mask vector on the edge set ε. It is a random operator. This represents the initial view.
[0045] In a further embodiment of the first aspect, step S5 further includes:
[0046] Recommendation objective function Knowledge graph objective function Comparative learning objective function Learning objective function in contrast to momentum Combined; the overall loss function Represented as:
[0047]
[0048] In the formula, λ, λ SSL , λ M It is a hyperparameter used to control the intensity of L2 regularization, ordinary contrastive learning, and momentum contrastive learning; This indicates L2 regularization.
[0049] Secondly, the present invention proposes a knowledge graph recommendation device, the device comprising:
[0050] The data acquisition module is used to determine the initial entity, collect text information and image information whose relevance to the initial entity is within a predetermined range, and construct a multimodal knowledge graph dataset based on the text information and image information;
[0051] A network connection module is used to merge the multimodal knowledge graph dataset into a connection network composed of several entity connections, and combine it with the initial entity to obtain the second entity data; a multimodal attention mechanism is introduced to represent the transmission coefficient between the head and tail entities in the connection network;
[0052] The dynamic update module is used to establish an encoder, import the second entity data into the encoder for dynamic updates, obtain the third entity sample, and calculate the momentum contrast learning loss based on the third entity sample.
[0053] The view generation module is used to generate 2N different subviews on the multimodal knowledge graph dataset, of which N are used for ordinary contrastive learning and output ordinary contrastive learning signals; and N are used for momentum contrastive learning and output momentum contrastive learning signals.
[0054] The learning module performs multi-task learning using the general contrastive learning signal and the momentum contrastive learning signal.
[0055] Beneficial effects: This invention effectively addresses concerns about potential inconsistencies between visual and textual representations, while effectively utilizing multimodal information to calculate node similarity and recommend items to users. The model performs excellently on three datasets. The method is simple and effective, possessing significant theoretical and practical value for improving the performance of multimodal knowledge graph recommendation models. Attached Figure Description
[0056] Figure 1 This is a flowchart of a multimodal knowledge graph recommendation method based on a momentum update contrastive learning framework in an embodiment of the present invention.
[0057] Figure 2 This is a diagram illustrating the overall model framework of the multimodal knowledge graph recommendation method in this embodiment of the invention. Detailed Implementation
[0058] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.
[0059] Research has shown that introducing contrastive learning techniques into knowledge graphs raises a series of new questions, such as how to construct effective positive sample pairs to support data augmentation methods, and how to increase the number of negative samples while avoiding negative samples that lead to misclassification.
[0060] Therefore, this embodiment proposes a multimodal knowledge graph recommendation method based on a momentum update contrastive learning framework, such as... Figure 1 As shown, this method includes five steps: multimodal dataset construction, multimodal attention mechanism, momentum update, knowledge graph comparative learning, and multi-task learning.
[0061] A schematic diagram of the overall model framework of the multimodal knowledge graph recommendation method in the embodiment is shown below. Figure 2 .
[0062] The multimodal knowledge graph recommendation method proposed in this embodiment is as follows:
[0063] I. Constructing a multimodal dataset:
[0064] Building a multimodal dataset involves three main stages: text acquisition, image acquisition, and multimodal vector generation. These stages are crucial for ensuring the comprehensiveness and richness of the final dataset.
[0065] 1) Text Extraction. To extract the English names and descriptions of entities from Freebase, you can use...< / ns / type.object.name> and< / ns / common.topic.description> Two relationships need to be retrieved. It's crucial to ensure that both the retrieved name and description include the `@en` tag, guaranteeing that the retrieved language is English.
[0066] 2) Image Acquisition. The acquired text information was used to search for visual data related to the entities. Using the English name and accompanying description of each entity as search terms, the top 10 images related to each entity were searched on the Bing search engine. Using the entity's name and description as keywords ensured the relevance and specificity of the images obtained.
[0067] 3) Multimodal Vector Generation. After obtaining the text and image information of the entity, the entity description and a randomly selected image are input into the CLIP model to generate text and image vectors for the entity. The CLIP model provides a collaborative method for generating vectors from text and images, ensuring consistency in representation across different modalities. However, the text and image vectors generated by CLIP have a dimension of 512, while the entity vector has a dimension of 64. Therefore, the text and image vectors need to be passed through a projection function to ensure that the vectors have the same dimension. The projection function can be expressed as p(x) = W2(W1x), where x is the feature vector, and W1 and W2 are the weight vectors of the fully connected layer.
[0068] II. Multimodal attention mechanism:
[0069] After obtaining the text and image vectors of the entities, these two vectors need to be merged into the entity's connection network and combined with the initial entity vector to obtain a new vector representing the entity. A multimodal attention mechanism can be used to represent the transmission coefficient between the head and tail entities, expressed as:
[0070]
[0071] In the formula, tanh(*) represents the nonlinear activation function, e hi The image vector representing the head entity, e ti The image vector representing the tail entity, e ht The text vector representing the head entity, e tt This represents the text vector of the tail entity. This multimodal attention mechanism obtains attention scores based on the distances between the head and tail entities, and between the tail entity and the multimodal vector. Finally, the parameters are regularized using the softmax function.
[0072]
[0073] This represents the connection network between the head entity h and the tail entity t; exp(*) indicates that the operation within the parentheses is performed with the natural logarithm base e as the base.
[0074] Therefore, the connection network between the head entity h and the tail entity t can be used To represent, specifically:
[0075]
[0076] Finally, the vector representation of the first-level entity is as follows: The function f(*) is a bidirectional connection fusion mechanism, represented as:
[0077]
[0078] In a knowledge graph with L layers, the final user u and item i vectors are formed by concatenating vectors from each layer, as follows:
[0079]
[0080] III. Momentum Update:
[0081] Two encoders, q and k, are established. The parameters of encoder k are dynamically updated based on the parameters of q and k from the previous training round. Entity vectors processed by encoder k are entered into a fixed-length queue, and the earliest entity vector in the queue is dequeued. When calculating the momentum-contrast learning loss, entities processed by both encoders are treated as positive samples, and entities in the queue are treated as negative samples.
[0082] θ k ←mθ k +(1-m)θ q ;
[0083] Since the queue length is much larger than the batch length, it is inevitable that erroneous negative samples will be introduced. We classify negative samples in the queue whose similarity to positive samples is greater than a threshold as positive samples:
[0084]
[0085] These erroneous negative samples are not considered when calculating the loss function:
[0086]
[0087] Based on the above principle, an encoder q and an encoder k are established for each user and item entity, and parameter initialization and updates are performed independently.
[0088] IV. Comparative Learning with Knowledge Graphs:
[0089] Using a random edge selection method, four distinct subviews are generated on the existing knowledge graph structure: two for traditional contrastive learning and two for momentum-based contrastive learning. The generated views can be represented as follows:
[0090]
[0091] In the formula, It is a set of nodes, M∈{0,1} |ε|It is the mask vector on the edge set ε. It is a random operator.
[0092] V. Multi-task learning:
[0093] To utilize both ordinary contrastive learning signals and momentum contrastive learning signals, this invention employs a multi-task learning technique to optimize the recommendation objective function. Knowledge graph objective function Comparative learning objective function Learning objective function in contrast to momentum Combined; the overall loss function Represented as:
[0094]
[0095] In the formula, λ, λ SSL , λ M It is a hyperparameter used to control the intensity of L2 regularization, ordinary contrastive learning, and momentum contrastive learning.
[0096]
[0097] O={(u,i,j)|(u,i)∈R + ,(u,j)∈R -} represents the training set, R + R represents the observed interactions between user items. - This represents unobserved interactions between user items.
[0098]
[0099] (h,r,t ′ ) is the triple after randomly replacing the tail entity, and σ(*) is the sigmoid function.
[0100]
[0101] Sim(*) represents the similarity between two enhanced subviews obtained through dot product. Represents user / object expression, It is a 0 and 1 symbolic function.
[0102]
[0103] In the formula, This represents the momentum comparison learning objective function; τ represents a positive sample; K represents the queue length; τ represents the hyperparameter controlling the temperature coefficient.
[0104] The experimental results are shown in Table 1 below:
[0105] Table 1: Comparison of this invention with six other models on three datasets
[0106]
[0107] In Table 1, the bolded results represent the results of this invention, the underlined results represent the second-best model results, and the improvement is relative to the second-best model.
[0108] To implement the multimodal knowledge graph recommendation method proposed in the above embodiments, the following embodiments further propose a knowledge graph recommendation device, which includes five components: a data acquisition module, a network connection module, a dynamic update module, a view generation module, and a learning module. The data acquisition module is used to determine initial entities, collect text and image information whose relevance to the initial entities is within a predetermined range, and construct a multimodal knowledge graph dataset based on the text and image information. The network connection module is used to merge the multimodal knowledge graph dataset into a connection network composed of several entity connections, and combine it with the initial entities to obtain second entity data; a multimodal attention mechanism is introduced to represent the transmission coefficient between the head and tail entities in the connection network. The dynamic update module is used to build an encoder, import the second entity data into the encoder for dynamic updates to obtain third entity samples, and calculate the momentum contrastive learning loss based on the third entity samples. The view generation module is used to generate 2N different sub-views on the multimodal knowledge graph dataset, where N are used for ordinary contrastive learning, outputting ordinary contrastive learning signals; and N are used for momentum contrastive learning, outputting momentum contrastive learning signals. The learning module utilizes the general contrastive learning signal and the momentum contrastive learning signal to perform multi-task learning.
[0109] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.
Claims
1. A multimodal knowledge graph recommendation method based on a momentum update contrastive learning framework, characterized in that, Includes the following steps: S1. Determine the initial entity, collect text information and image information with a correlation degree with the initial entity within a predetermined range, and construct a multimodal knowledge graph dataset based on the text information and image information; S2. Merge the multimodal knowledge graph dataset into a connection network composed of several entity connections, and combine it with the initial entity to obtain the second entity data; introduce a multimodal attention mechanism to represent the transmission coefficient between the head and tail entities in the connection network; S3. Establish an encoder, import the second entity data into the encoder for dynamic updating, obtain the third entity sample, and calculate the momentum contrast learning loss based on the third entity sample. S3-1. Establish two encoders q and k. The parameters in encoder k are dynamically updated based on the parameters of q and k during the previous training round. S3-2. The entity vectors after passing through the k encoder enter a queue of fixed length, and the earliest entity vector in the queue will be dequeued. S3-3. When calculating the momentum contrastive learning loss, entities that have passed through two encoders are used as positive samples, and entities in the queue are used as negative samples. ； In the formula, This represents the parameters in encoder k; The parameter q represents the encoder parameter; m represents the momentum update parameter. If an incorrect negative sample is introduced during step S3-3, then negative samples in the queue with a similarity greater than a threshold to positive samples will be classified as positive samples. ； In the formula, Indicates the weight of negative samples; This represents a sample in encoder q; This represents a sample in encoder k; Sim(*) indicates that the similarity between samples in two encoders is obtained through dot product. Indicates the similarity threshold; These erroneous negative samples are not considered when calculating the loss function: ； In the formula, This represents the momentum comparison learning objective function; Indicates a positive sample; K represents the queue length; This represents the hyperparameter controlling the temperature coefficient; S4. Generate 2N different sub-views on the multimodal knowledge graph dataset, of which N are used for ordinary contrastive learning and output ordinary contrastive learning signals; and N are used for momentum contrastive learning and output momentum contrastive learning signals. S5. Perform multi-task learning using the ordinary contrastive learning signal and the momentum contrastive learning signal.
2. The multimodal knowledge graph recommendation method according to claim 1, characterized in that, The process of constructing the multimodal knowledge graph dataset in step S1 includes: Extract the entity name and entity description information of the initial entity, and use the entity name and entity description information to search for a predetermined amount of data related to the initial entity in a search engine to obtain text information and image information; The entity description information and one of the randomly selected image information are input into the CLIP model to generate the text vector and image vector of the initial entity; The text vector and image vector are passed through a projection function so that the text vector, image vector, and vector generated by the CLIP model have the same dimension. Wherein, the projection function is expressed as In the formula, x is the eigenvector. , It is the weight vector of the fully connected layer.
3. The multimodal knowledge graph recommendation method according to claim 1, characterized in that, In step S2, a multimodal attention mechanism is introduced to represent the transmission coefficient between the head and tail entities in the connection network. The multimodal attention mechanism is expressed as follows: ； In the formula, Represents a non-linear activation function. The image vector representing the head entity. The image vector representing the tail entity. The text vector representing the header entity. The text vector representing the tail entity. Represents the head entity vector; Represents a vector of relationships between entities; Represents the transition matrix of relation r; Represents the transpose of the image vector of the tail entity; This represents the transpose of the text vector of the header entity.
4. The multimodal knowledge graph recommendation method according to claim 3, characterized in that, The multimodal attention mechanism obtains attention scores by measuring the distances between head and tail entities, and between the tail entity and the multimodal vector. Regularize the parameters using the softmax function: ； In the formula, The r represents the connection network between the head entity h and the tail entity t; r represents the relationship between entities. This represents the relationship between entities after random replacement; This represents the tail entity after random replacement; exp(*) indicates that the exponentiation operation with the natural logarithm base e is performed on the part within the parentheses; Connection network between head entity h and tail entity t ; The vector representation of the first-level entity is as follows , where the function It is a bidirectional fusion type, represented as: ； In the formula, Indicates the activation function; This represents the representation of the network connecting the head entity h; , It is the weight vector of the fully connected layer.
5. The multimodal knowledge graph recommendation method according to claim 4, characterized in that, In a knowledge graph with L layers, the final user u and item i vectors are formed by concatenating vectors from each layer, as follows: ； In the formula, This represents the concatenated user u vector; Represents the initial user vector u; Represents the vector u of the user at layer L; Represents the concatenated vector of item i; Represents the initial item vector i; Represents the vector of item i in the Lth layer; This indicates a splicing operation.
6. The multimodal knowledge graph recommendation method according to claim 1, characterized in that, The subview generated on the multimodal knowledge graph dataset in step S4 is represented as follows: ； In the formula, It is a set of nodes. It is in the edge set The mask vector on, It is a random operator. This represents the initial view.
7. The multimodal knowledge graph recommendation method according to claim 1, characterized in that, Step S5 further includes: Recommendation objective function Knowledge graph objective function Comparative learning objective function Learning objective function in contrast to momentum Combined; the overall loss function Represented as: ； In the formula, , , It is a hyperparameter used to control the intensity of L2 regularization, ordinary contrastive learning, and momentum contrastive learning; This indicates L2 regularization.
8. A knowledge graph recommendation apparatus, used to execute the multimodal knowledge graph recommendation method according to any one of claims 1 to 7, characterized in that, include: The data acquisition module is used to determine the initial entity, collect text information and image information whose relevance to the initial entity is within a predetermined range, and construct a multimodal knowledge graph dataset based on the text information and image information; A network connection module is used to merge the multimodal knowledge graph dataset into a connection network composed of several entity connections, and combine it with the initial entity to obtain the second entity data; a multimodal attention mechanism is introduced to represent the transmission coefficient between the head and tail entities in the connection network; The dynamic update module is used to establish an encoder, import the second entity data into the encoder for dynamic updates, obtain the third entity sample, and calculate the momentum contrast learning loss based on the third entity sample. The view generation module is used to generate 2N different subviews on the multimodal knowledge graph dataset, of which N are used for ordinary contrastive learning and output ordinary contrastive learning signals; and N are used for momentum contrastive learning and output momentum contrastive learning signals. The learning module performs multi-task learning using the general contrastive learning signal and the momentum contrastive learning signal.