Attention mechanism-based multi-modal knowledge graph recommendation method and device, and medium

Abstract

The application relates to a multimodal knowledge graph recommendation method and device based on an attention mechanism and a medium. The method comprises the following steps: obtaining behavior data of a user clicking an item; converting the behavior of the user clicking the item into a user feature vector, an item feature vector, an ID feature vector and an interaction matrix vector of the user and the item; performing intramodule attention mechanism processing and intermodule attention mechanism processing on the user feature vector and the item feature vector to obtain a first candidate recommendation feature vector; integrating entity information weights in a knowledge graph into the first candidate recommendation feature vector and the interaction matrix vector of the user and the item to obtain a second candidate recommendation feature vector; extracting a predicted behavior probability corresponding to the behavior data of the user clicking the item based on the second candidate recommendation feature vector to obtain a target recommendation. The application can fuse multimodal information and an attention mechanism, accurately recommend items to the user according to the clicking behavior, and thus improves the recommendation efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and more specifically, to a multimodal knowledge graph recommendation method, apparatus, and medium based on an attention mechanism. Background Technology

[0002] With the development of internet technology, recommender systems, as one of the methods to solve information overload, have become an indispensable system in internet products. In recent years, with the continuous development of graph neural networks, recommender algorithms that combine graph neural network technology with knowledge graphs have achieved very good results on various public datasets. However, most existing recommender algorithms that combine graph neural networks and knowledge graphs are unimodal and do not take into account the multimodal information of items, resulting in information loss and affecting the accuracy of the recommender model.

[0003] Human understanding of the world is multimodal, encompassing visual, textual, and auditory information. This multimodal information facilitates more accurate prediction of user preferences. For instance, in movie recommendation scenarios, users can learn about a movie not only from its poster and cover image but also from its textual synopsis or audio descriptions of trailers. However, most existing recommendation methods do not consider the influence of user behavior sequences and struggle to handle multimodal information. Summary of the Invention

[0004] Based on the aforementioned technical problems, this invention aims to solve the problem that recommendation methods struggle to handle multimodal information by fusing attention mechanisms and knowledge graphs to process multimodal information and provide accurate recommendations based on the fusion results.

[0005] The first aspect of this invention provides a multimodal knowledge graph recommendation method based on an attention mechanism, the method comprising:

[0006] Obtain user click data on items;

[0007] The user's click on an item is transformed into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector.

[0008] The user feature vector and the item feature vector are processed by intra-module attention mechanism and inter-module attention mechanism to obtain the first candidate recommendation feature vector;

[0009] The entity information weights in the knowledge graph are incorporated into the first candidate recommendation feature vector and the user-item interaction matrix to obtain the second candidate recommendation feature vector.

[0010] Based on the second candidate recommendation feature vector, the predicted behavior probability corresponding to the user's clicked item behavior data is extracted to obtain the target recommendation.

[0011] In some embodiments of the present invention, the step of processing the user feature vector and the item feature vector using intra-module attention mechanism and inter-module attention mechanism to obtain the first candidate recommendation feature vector includes:

[0012] The user feature vector and the item feature vector are respectively input into the trained multimodal knowledge graph recommendation model based on the attention mechanism. The multimodal knowledge graph recommendation model based on the attention mechanism includes a multimodal feature fusion module, and the multimodal feature fusion module includes a self-attention module.

[0013] The attention coefficients of the user feature vector and the item feature vector are obtained through the self-attention module.

[0014] The user feature vector and the item feature vector are weighted and summed with their corresponding attention coefficients to obtain the first user feature vector and the first item feature vector.

[0015] The first user feature vector and the first item feature vector are cross-fused to obtain the first candidate recommendation feature vector.

[0016] In some embodiments of the present invention, the multimodal feature fusion module further includes a collaborative attention module; the step of cross-fusing the first user feature vector and the first item feature vector to obtain the first candidate recommendation feature vector includes:

[0017] The first user feature vector and the first item feature vector are input into the collaborative attention module;

[0018] The collaborative attention module generates collaborative attention scores for the first user feature vector and the first item feature vector, respectively.

[0019] Based on the cross-fusion of the collaborative attention score and the first user feature vector and the first item feature vector, a first candidate recommendation feature vector is obtained.

[0020] In some embodiments of the present invention, the attention-based multimodal knowledge graph recommendation model includes an attention graph convolution module; the step of integrating entity information weights from the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain a second candidate recommendation feature vector includes:

[0021] The first candidate recommendation feature vector is input into the attention map convolution module;

[0022] The attention graph convolution module aggregates the neighborhood entity information of the items in the knowledge graph based on the first candidate recommendation feature vector, and fuses the aggregated information with the item feature vector.

[0023] Based on the fused item feature vector, the first candidate recommendation feature vector, the ID feature vector, and the user-item interaction matrix, the second candidate recommendation feature vector is obtained.

[0024] In some embodiments of the present invention, the attention-based multimodal knowledge graph recommendation model further includes an item pooling layer; based on the fused item feature vector, the first candidate recommendation feature vector, the ID feature vector, and the user-item interaction matrix, a second candidate recommendation feature vector is obtained, including:

[0025] The fused item feature vector is input into the item pooling layer;

[0026] The user feature vector from the first candidate recommendation feature vector is input into the item pooling layer;

[0027] The ID feature vector and the user-item interaction matrix are input into the item pooling layer;

[0028] The item pooling layer outputs a target item vector, a target user vector, and a target user interest profile vector, and uses the target item vector, target user vector, and target user interest profile vector as the second candidate recommendation feature vector.

[0029] In some embodiments of the present invention, the step of extracting the predicted behavior probability corresponding to the user's clicked item behavior data based on the second candidate recommendation feature vector to obtain the target recommendation includes:

[0030] The probability of a user clicking on an item is obtained by performing a full connection between the target item vector, the target user vector, and the target user interest profile vector.

[0031] Target recommendations are obtained from the database of items to be recommended based on the probability of users clicking on them.

[0032] In some embodiments of the present invention, the attention-based multimodal knowledge graph recommendation model further includes a pre-training module, which includes a first unimodal model and a second unimodal model; the model converts the user's click behavior on an item into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector, including:

[0033] Extract the images and text corresponding to the user and the item from the user's click on the item;

[0034] The first trained monomial model processes the images corresponding to users and items, and the second trained monomial model processes the text corresponding to users and items to obtain user feature vectors and item feature vectors.

[0035] The interaction matrix vector between users and items is obtained based on the user feature vector and the item feature vector.

[0036] A second aspect of the present invention provides a multimodal knowledge graph recommendation device based on an attention mechanism, the device comprising:

[0037] The acquisition module is configured to acquire user click data on items.

[0038] The conversion module is configured to convert the user's click on an item into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector.

[0039] The first attention module is configured to process the user feature vector and the item feature vector using intra-module attention mechanism and inter-module attention mechanism to obtain the first candidate recommendation feature vector.

[0040] The second attention module is configured to integrate the entity information weights in the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain the second candidate recommendation feature vector.

[0041] The recommendation module is configured to extract the predicted behavior probability corresponding to the user's clicked item behavior data based on the second candidate recommendation feature vector, and obtain the target recommendation.

[0042] A third aspect of the present invention provides a computer device including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the attention-based multimodal knowledge graph recommendation method described in various embodiments.

[0043] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the attention-based multimodal knowledge graph recommendation method described in the various embodiments.

[0044] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0045] This application performs intramodal feature cross-interaction on single-modal features such as image features, text features, and ID features (multiple single-modal features constitute multimodal features), and then generates corresponding fused vector representations through a collaborative attention mechanism. By using a graph convolutional network to continuously aggregate neighborhood information in the knowledge graph for items clicked in the user's history and currently predicted items, new item vector representations are generated, which can more accurately recommend items needed by users. In particular, by incorporating user historical behavior data and using the attention mechanism to extract user features, this model does not have strict requirements on data and can make recommendations even when modal information is missing, thereby improving recommendation efficiency.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0048] Figure 1 This illustration shows a schematic diagram of the steps of a multimodal knowledge graph recommendation method based on an attention mechanism in an exemplary embodiment of this application;

[0049] Figure 2 This illustration shows a schematic diagram of a multimodal knowledge graph recommendation model based on an attention mechanism in an exemplary embodiment of this application;

[0050] Figure 3 A schematic diagram of the Bert model structure in an exemplary embodiment of this application is shown;

[0051] Figure 4 A schematic diagram of the Bert model structure in an exemplary embodiment of this application is shown;

[0052] Figure 5 This illustration shows a schematic diagram of obtaining ID features by encoding using a one-hot encoding method in an exemplary embodiment of this application;

[0053] Figure 6 This illustration shows a schematic diagram of the structure of a multimodal feature fusion module in an exemplary embodiment of this application.

[0054] Figure 7 A schematic diagram of a dot product-based attention mechanism in an exemplary embodiment of this application is shown;

[0055] Figure 8This illustration shows a feature diagram incorporating entities and relationships (edge ​​information) between entities in a knowledge graph, as shown in an exemplary embodiment of this application.

[0056] Figure 9 This illustration shows a comparison diagram of the attention mechanism in an exemplary embodiment of this application with other methods;

[0057] Figure 10 A schematic diagram of an article pooling layer in an exemplary embodiment of this application is shown;

[0058] Figure 11 A schematic diagram of the structure of a multimodal knowledge graph recommendation device based on an attention mechanism according to an exemplary embodiment of this application is shown.

[0059] Figure 12 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0060] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this application. It will be apparent to those skilled in the art that this application can be implemented without one or more of these details. In other instances, to avoid confusion with this application, some technical features well-known in the art have not been described.

[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0062] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. The drawings are not drawn to scale, and some details may be enlarged and omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0063] The following is in conjunction with the instruction manual appendix. Figure 1 - Appendix Figure 12 Several embodiments are given to describe exemplary implementations according to this application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementations of this application are not limited in any way. Rather, the implementations of this application can be applied to any applicable scenario.

[0064] In some exemplary embodiments of this application, a multimodal knowledge graph recommendation method based on an attention mechanism is provided, such as... Figure 1 As shown, the method includes:

[0065] S1. Obtain user click data on items;

[0066] S2. The user's click on the item behavior is converted into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector;

[0067] S3. Process the user feature vector and the item feature vector using intra-module attention mechanism and inter-module attention mechanism to obtain the first candidate recommendation feature vector;

[0068] S4. Integrate the entity information weights in the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain the second candidate recommendation feature vector;

[0069] S5. Based on the second candidate recommendation feature vector, extract the predicted behavior probability corresponding to the user's clicked item behavior data to obtain the target recommendation.

[0070] refer to Figure 2The attention-based multimodal knowledge graph recommendation model includes a pre-training module, a multimodal feature fusion module, an attention graph convolution module, and an item pooling layer. The pre-training module includes a first single-mode model and a second single-mode model; it transforms the user's click behavior into user feature vectors, item feature vectors, ID feature vectors, and user-item interaction matrix vectors, including: extracting the user and item corresponding images and text from the user's click behavior; processing the user and item corresponding images using the trained first single-mode model, and processing the user and item corresponding text using the trained second single-mode model to obtain user feature vectors and item feature vectors; and obtaining the user-item interaction matrix vector based on the user feature vectors and item feature vectors. Here, the first single-mode model is an image processing model, and the second single-mode model is a text processing model. Specifically, the text processing model can use the Google open-source BERT model to output the vector of the last layer as the vector representation of the text, where the complete structure of the BERT model is as follows: Figure 3 As shown; the second single-model is an image processing model, such as... Figure 4 As shown, ResNet-50 can be used as the image feature extractor. Multiple unimodal features constitute multimodal features. Besides images and text, sometimes there are ID features, i.e., the third unimodal model. Of course, depending on the specific application scenario, there may also be a fourth unimodal model, a fifth unimodal model, and so on. As a variable implementation method, ID features are encoded using a one-hot encoding method, assigning corresponding feature vectors based on the ID from a randomly initialized feature matrix. For example... Figure 5 As shown, first initialize a vector matrix W∈R n×d Where n is the number of entities and d is the vector dimension. Each item is one-hot encoded, denoted as v. one_hot v one_hot ∈R 1×n Except for the item ID, which has a 1 at its index, all other positions are 0. The item ID feature is calculated as v = v one_hot ×W, where W represents the initialization of a vector matrix.

[0071] In one specific implementation, the user feature vector and item feature vector are processed by intra-module attention mechanism and inter-module attention mechanism to obtain the first candidate recommendation feature vector. This includes: inputting the user feature vector and item feature vector into a trained multimodal knowledge graph recommendation model based on attention mechanism, wherein the multimodal knowledge graph recommendation model based on attention mechanism includes a multimodal feature fusion module, and the structural diagram of the multimodal feature fusion module is shown below. Figure 6As shown. The multimodal feature fusion module includes a self-attention module; it obtains the attention coefficients of the user feature vector and the item feature vector through the self-attention module; it then performs a weighted sum of the user feature vector and the item feature vector with their corresponding attention coefficients to obtain the first user feature vector and the first item feature vector; finally, it cross-fuses the first user feature vector and the first item feature vector to obtain the first candidate recommendation feature vector. The attention mechanism performs feature cross-interaction on single modal information to extract more abstract, more general, and semantic-level features. Attention calculation also includes dot product, such as... Figure 7 As shown, for image and text type features, intramodal feature cross-interaction is performed, and corresponding query vector, key vector, and value vector are generated based on the modal features, with all three dimensions remaining consistent. The attention transformation formula is as shown in formula (1):

[0072]

[0073] Where e represents the feature before transformation. Let W be the transformed features, and W be the learned weights, where W∈R i×d .

[0074] The attention scores of the query vector and key vector are calculated to evaluate the correlation between the current feature and the internal features. The attention calculation is as shown in formula (2), where d is used to prevent the value from being too large and difficult to train.

[0075]

[0076] Next, all calculated scores are normalized using formula (3):

[0077]

[0078] Finally, the calculated scores are used to weight and sum all the values ​​to obtain the final output, as shown in formula (4):

[0079]

[0080] The self-attention module performs feature crossing on modal features, and after multi-layer processing, the output modal features are used in the collaborative attention module.

[0081] In some embodiments of the present invention, the multimodal feature fusion module further includes a collaborative attention module; cross-fusing the first user feature vector and the first item feature vector to obtain a first candidate recommendation feature vector includes: inputting the first user feature vector and the first item feature vector into the collaborative attention module; the collaborative attention module generating collaborative attention scores for the first user feature vector and the first item feature vector respectively; and cross-fusing the first user feature vector and the first item feature vector based on the collaborative attention scores to obtain the first candidate recommendation feature vector. Here, the first candidate recommendation vector includes the item's image features, text features, and ID features. Figure 6 The image and text are only shown, but it can be understood that ID features are also included.

[0082] In some embodiments of this application, the multimodal knowledge graph recommendation model based on the attention mechanism includes an attention graph convolution module; integrating entity information weights from the knowledge graph into a first candidate recommendation feature vector and the user-item interaction matrix to obtain a second candidate recommendation feature vector includes: inputting the first candidate recommendation feature vector into the attention graph convolution module; the attention graph convolution module aggregating neighborhood entity information of items in the knowledge graph based on the first candidate recommendation feature vector, and fusing the aggregated information with the item feature vector; and obtaining the second candidate recommendation feature vector based on the fused item feature vector, the first candidate recommendation feature vector, the ID feature vector, and the user-item interaction matrix. A schematic diagram of the aggregation of neighborhood entity information of items in the knowledge graph can be found in [reference needed]. Figure 2 and Figure 8 Items are entities, and users are also entities. The relationships between entities include the user's action of clicking on an item. Figure 8 This diagram illustrates the user feature representation after extracting information from the knowledge graph. The sets of users and items are defined as U = {u1, u2…} and V = {v1, v2…}, and the user-item interaction matrix is ​​defined as Y = {y uv |u∈U,v∈V}. If user u clicks on item v, then y uv =1, otherwise 0. To enhance the feature representation of items, the features of the knowledge graph and the item images and text are used as side information (see...). Figure 8 The edges in the graph represent the items the user has clicked in the past. A knowledge graph G consists of triples (entity-relation-entity), i.e., G = {(h,r,t)|h,t∈E,r∈R}, where E and R are the entity set and relation set, respectively, and h, r, and t are the head entity, relation, and tail entity of the triple. An item v is connected to one or more entities, and the set of neighboring entities of item v is denoted by N(v). The multimodal features of an item are denoted by M, where M = {M... v M t M i}, where Mv M t M i These are the image features, text features, and ID features of the entity, respectively. The task is defined as follows: Given a user set U, an item set V, a user-item interaction matrix Y, a knowledge graph G, and multimodal information M of the items, predict the probability that user u will click on item v. Where Θ represents the model parameters and F is the function for model fitting.

[0083] It should be noted that knowledge graphs generally contain a large amount of redundant information. An entity has many neighboring nodes, but only a few nodes are helpful for the recommendation task. In order to solve the whole problem, this application introduces an attention mechanism on the edges to selectively aggregate entity information. The user's representation is formed by the items that the user has clicked. Items have a large number of entity connections in the knowledge graph. The contribution of entities to the final user representation is different. In order to filter out irrelevant information, the feature propagation path is controlled. Users have different sensitivities to different relations in the knowledge graph. For example, their liking for a movie may be more due to the movie's director or a certain actor, while they may not care about factors such as the release date or music producer. Therefore, when aggregating entity information in the knowledge graph, entity information such as "director" and "actor" should be propagated along the connections between graph nodes as much as possible, while entity information such as "release date" and "music producer" that a user is not very interested in should be propagated as little as possible or not at all. The score function f of formula (5) can be used to calculate the score weight between user u and relation r. It illustrates the importance of the user to the relationship R.

[0084]

[0085] After attention allocation, such as Figure 8 In the right-hand section, information aggregation is more about transmission along a specific path, such as... Figure 8 R1 in the algorithm makes the aggregated information more targeted. In implementing the attention graph convolution module, we compared it with three other methods, and the attention mechanism performed best. For example... Figure 9 The diagrams show summation, splicing, factorization machine (FM), and attention mechanism, respectively. The formula for summation is shown in formula (6):

[0086]

[0087] This method is simple to calculate and does not add dimensions to the items, thus not introducing additional weights. However, this method will cause some loss of information. The calculation formula for the splicing method is as shown in formula (7):

[0088] V b =concate(v1,v2,v3…v n (7)

[0089] The concatenation method increases the original vector dimension without reducing information loss, but it increases the computational cost and the number of model parameters.

[0090] Factorizers start from the perspective of feature crosses, performing pairwise crosses on features. The typical FM algorithm takes first-order cross features and second-order cross features as inputs to the model, and directly uses second-order cross features for calculation. The calculation formula is as shown in formula (8):

[0091]

[0092] As can be seen from the polynomial expansion formula, the result of the formula is equivalent to the sum of the pairwise products of all features.

[0093] In the attention mechanism, given the vector e of the current item to be tested v Item vector e with user's historical clicks v e v e v , will e v As a query vector, e v e v e v Treat the query as a key-value vector and calculate the attention coefficient between the query and each value. There are multiple approaches. Two attention calculation methods were explored: one based on vector dot product and the other based on the fusion of multiple methods. The attention calculation method based on vector dot product is shown in formula (9):

[0094]

[0095] Considering that the dot product approach is too simplistic, this study explored a multi-method attention mechanism to calculate the score, such as... Figure 10 As shown, the summation and product between two features are first calculated, then the two vectors are concatenated with the above results, and several feature vectors are passed through a pyramid-shaped fully connected network to finally output the score.

[0096] In some embodiments of this application, the attention-based multimodal knowledge graph recommendation model further includes an item pooling layer (the item pooling layer is also as follows...). Figure 10(As shown); Based on the fused item feature vector, the first candidate recommendation feature vector, and the user-item interaction matrix of the ID feature vector, a second candidate recommendation feature vector is obtained, including: inputting the fused item feature vector into the item pooling layer; inputting the user feature vector from the first candidate recommendation feature vector into the item pooling layer; inputting the user-item interaction matrix into the item pooling layer; the item pooling layer outputs the target item vector, the target user vector, and the target user interest profile vector, and uses the target item vector, the target user vector, and the target user interest profile vector as the second candidate recommendation feature vector.

[0097] In other embodiments of this application, the predicted behavior probability corresponding to the user's clicked item behavior data is extracted based on the second candidate recommendation feature vector to obtain the target recommendation, including: performing a full connection on the target item vector, the target user vector, and the target user interest profile vector to obtain the probability of the user clicking the item; and obtaining the target recommendation from the database to be recommended based on the probability of the user clicking the item. Here, the target user interest profile vector can also be represented as a user behavior vector, as shown in formula (10):

[0098]

[0099] Among them, e u For the user vector, e i Let e ​​be the vector of the item. b The user's behavior vector is used. Except for the last fully connected layer which uses the sigmoid function, the activation functions of the other layers are ReLU. mlp is a multi-layer fully connected network.

[0100] It should be noted that the attention-based multimodal knowledge graph recommendation model adopts the conventional training method in existing technologies, but uses a different loss function. Given the user-item interaction matrix, the multimodal information of the items, and the knowledge graph, the click probability of the user on the item can be calculated using formula (10). Using user interaction data with items as training data, the error between the model's predicted probability and the actual situation is calculated. This error is then backpropagated, and the model parameters are adjusted to reduce the error, ultimately achieving convergence. The model's error function is shown in formula (11):

[0101]

[0102] The model's time complexity mainly consists of three parts: the multimodal feature fusion module, the attention map convolution module, and the item pooling layer. The time complexity of the multimodal feature fusion module is O(m). 2 d 2 The time complexity of the attention graph convolution module is O(K). l d2 The time complexity of the item pooling layer is O(sd). 2 ), where m, d, K, l, and s represent the number of modal information channels, vector dimension, number of neighborhood samples, depth of graph convolution, and length of user behavior data, respectively. The overall time complexity of the model is O((m 2 +K l +s)d 2 ).

[0103] This application performs intramodal feature cross-interaction on single-modal features such as image features, text features, and ID features (multiple single-modal features constitute multimodal features), and then generates corresponding fused vector representations through a collaborative attention mechanism. By using a graph convolutional network to continuously aggregate neighborhood information in the knowledge graph for items clicked in the user's history and currently predicted items, new item vector representations are generated, which can more accurately recommend items needed by users. In particular, by incorporating user historical behavior data and using the attention mechanism to extract user features, this model does not have strict requirements on data and can make recommendations even when modal information is missing, thus improving recommendation efficiency.

[0104] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0105] In some exemplary embodiments of this application, a multimodal knowledge graph recommendation apparatus based on an attention mechanism is also provided to execute the multimodal knowledge graph recommendation method based on an attention mechanism described in various embodiments of this application, such as... Figure 11 As shown, the device includes:

[0106] Module 1101 is configured to acquire user click data on items.

[0107] The conversion module 1102 is configured to convert the user's click on an item into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector.

[0108] The first attention module 1103 is configured to process the user feature vector and the item feature vector using an intra-module attention mechanism and an inter-module attention mechanism to obtain a first candidate recommendation feature vector.

[0109] The second attention module 1104 is configured to integrate the entity information weights in the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain the second candidate recommendation feature vector.

[0110] The recommendation module 1105 is configured to extract the predicted behavior probability corresponding to the user's clicked item behavior data based on the second candidate recommendation feature vector, and obtain the target recommendation.

[0111] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0112] It should also be emphasized that the system provided in this application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0113] Please refer to the following. Figure 12 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 12 As shown, the computer device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the attention-based multimodal knowledge graph recommendation method provided in any of the foregoing embodiments of this application.

[0114] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0115] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 201 is used to store programs. After receiving an execution instruction, processor 200 executes the program. The attention-based multimodal knowledge graph recommendation method disclosed in any of the foregoing embodiments of this application can be applied to processor 200, or implemented by processor 200.

[0116] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0117] This application also provides a computer-readable storage medium corresponding to the attention-based multimodal knowledge graph recommendation method provided in the foregoing embodiments, wherein a computer program is stored thereon, and the computer program, when run by a processor, executes the attention-based multimodal knowledge graph recommendation method provided in any of the foregoing embodiments.

[0118] In addition, examples of the computer-readable storage medium may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be described in detail here.

[0119] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal knowledge graph recommendation method based on attention mechanism provided in any of the foregoing embodiments. The method includes: acquiring user click behavior data; converting the user click behavior into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector; processing the user feature vector and item feature vector using intra-modal attention mechanism and inter-modal attention mechanism to obtain a first candidate recommendation feature vector; integrating entity information weights from the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain a second candidate recommendation feature vector; and extracting the predicted behavior probability corresponding to the user click behavior data based on the second candidate recommendation feature vector to obtain a target recommendation.

[0120] It should be noted that the algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application. Numerous specific details are set forth in the specification provided herein. However, it is to be understood that embodiments of this application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0121] Those skilled in the art will understand that the various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art should understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application.

[0122] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An attention mechanism-based multi-modal knowledge graph recommendation method, characterized in that, The method comprises: obtaining behavior data of a user clicking an item; transforming the user's behavior of clicking the item into a user feature vector, an item feature vector, an ID feature vector, and a user-item interaction matrix vector; performing intra-module attention mechanism processing and inter-module attention mechanism processing on the user feature vector and the item feature vector to obtain a first candidate recommendation feature vector; integrating entity information weights in a knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain a second candidate recommendation feature vector; extracting a predicted behavior probability corresponding to the behavior data of the user clicking the item based on the second candidate recommendation feature vector to obtain a target recommendation; the intra-module attention mechanism processing and inter-module attention mechanism processing on the user feature vector and the item feature vector to obtain the first candidate recommendation feature vector comprises: inputting the user feature vector and the item feature vector into a trained multi-modal knowledge graph recommendation model based on an attention mechanism, wherein the multi-modal knowledge graph recommendation model based on the attention mechanism comprises a multi-modal feature fusion module, and the multi-modal feature fusion module comprises a self-attention module; obtaining attention coefficients of the user feature vector and the item feature vector through the self-attention module; performing weighted sum calculation on the user feature vector and the item feature vector respectively with their corresponding attention coefficients to obtain a first user feature vector and a first item feature vector; cross-fusing the first user feature vector and the first item feature vector to obtain the first candidate recommendation feature vector. 2.The attention mechanism based multi-modal knowledge graph recommendation method of claim 1, wherein, The multi-modal feature fusion module further comprises a collaborative attention module; the cross-fusion of the first user feature vector and the first item feature vector to obtain the first candidate recommendation feature vector comprises: inputting the first user feature vector and the first item feature vector into the collaborative attention module; the collaborative attention module generates collaborative attention scores of the first user feature vector and the first item feature vector respectively; cross-fusing the first user feature vector and the first item feature vector based on the collaborative attention scores to obtain the first candidate recommendation feature vector. 3.The attention mechanism based multi-modal knowledge graph recommendation method of claim 1, wherein, The multi-modal knowledge graph recommendation model based on the attention mechanism comprises an attention graph convolution module; the integration of entity information weights in the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain the second candidate recommendation feature vector comprises: inputting the first candidate recommendation feature vector into the attention graph convolution module; the attention graph convolution module aggregates neighborhood entity information of the item in the knowledge graph according to the first candidate recommendation feature vector, and fuses the aggregated information with the item feature vector; based on the fused item feature vector, the first candidate recommendation feature vector, the ID feature vector, and the user-item interaction matrix, a second candidate recommendation feature vector is obtained. 4.The attention mechanism based multi-modal knowledge graph recommendation method of claim 3, characterized in that, The attention mechanism-based multi-modal knowledge graph recommendation model further comprises an item pooling layer; based on the fused item feature vector, the first candidate recommendation feature vector and the ID feature vector, and the user-item interaction matrix, a second candidate recommendation feature vector is obtained, comprising: inputting the fused item feature vector into the item pooling layer; inputting the user feature vector in the first candidate recommendation feature vector into the item pooling layer; inputting the ID feature vector and the user-item interaction matrix into the item pooling layer; The item pooling layer outputs a target item vector, a target user vector and a target user interest portrait vector, and takes the target item vector, the target user vector and the target user interest portrait vector as the second candidate recommendation feature vector. 5.The attention mechanism based multi-modal knowledge graph recommendation method of claim 4, characterized in that, The target recommendation is obtained by extracting the predicted behavior probability corresponding to the user's click behavior data on the item based on the second candidate recommendation feature vector, comprising: fully connecting the target item vector, the target user vector and the target user interest portrait vector to obtain the probability of the user clicking on the item; According to the probability of the user clicking on the item, the target recommendation is obtained from the recommendation database. 6.The method of claim 1 or 2 or 3 or 4 or 5, wherein, The attention mechanism-based multi-modal knowledge graph recommendation model further comprises a pre-training module, and the pre-training module comprises a first single model and a second single model; the user's click behavior on the item is converted into a user feature vector, an item feature vector, an ID feature vector and a user-item interaction matrix vector, comprising: extracting the pictures and texts corresponding to the user and the item in the user's click behavior on the item; processing the pictures corresponding to the user and the item through the trained first single model, and processing the texts corresponding to the user and the item through the trained second single model to obtain the user feature vector and the item feature vector; According to the user feature vector and the item feature vector, a user-item interaction matrix vector is obtained.

7. An attention mechanism-based multi-modal knowledge graph recommendation device, characterized in that, The device comprises: an acquisition module configured to acquire user behavior data on an item; a conversion module configured to convert the user's click behavior on the item into a user feature vector, an item feature vector, an ID feature vector and a user-item interaction matrix vector; a first attention module configured to perform intra-model attention mechanism processing and inter-model attention mechanism processing on the user feature vector and the item feature vector to obtain a first candidate recommendation feature vector; a second attention module configured to integrate entity information weights in the knowledge graph into the first candidate recommendation feature vector and the user-item interaction matrix to obtain a second candidate recommendation feature vector; a recommendation module configured to extract a predicted behavior probability corresponding to the user's click behavior data on the item based on the second candidate recommendation feature vector to obtain a target recommendation; The first attention module is further configured to input the user feature vector and the item feature vector into a trained multi-modal knowledge graph recommendation model based on an attention mechanism respectively, wherein the multi-modal knowledge graph recommendation model based on the attention mechanism comprises a multi-modal feature fusion module, and the multi-modal feature fusion module comprises a self-attention module; the self-attention module is used to obtain an attention coefficient of the user feature vector and an attention coefficient of the item feature vector; the user feature vector and the item feature vector are weighted and added with the corresponding attention coefficients respectively to obtain a first user feature vector and a first item feature vector; and the first user feature vector and the first item feature vector are cross-fused to obtain a first candidate recommendation feature vector. 8.A computer device, comprising a memory and a processor, and characterized in that, The computer readable instructions are stored in the memory and are executed by the processor to cause the processor to perform the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images