An efficient knowledge distillation knowledge graph recommendation method, device and medium

Abstract

This invention discloses an efficient knowledge graph recommendation method, device, and medium for knowledge distillation. The method includes: transferring knowledge from a large knowledge graph-based teacher model to a small single-layer graph attention network student model through four distillation tasks: 1) distillation of user embeddings and item embeddings; 2) angular distillation of user embeddings and item embeddings; 3) distillation of user and item prediction results; and 4) distillation of user and item prediction results. Input data is then acquired and input into the student model obtained through the four distillation tasks, and the recommendation result is output. This invention performs knowledge distillation not only at the embedding and prediction levels but also at the angular attribute and knowledge graph attention information levels. This not only improves the convergence speed of model training but also more effectively extracts information from the knowledge graph, mitigating data noise and cold start problems in knowledge graphs.

Description

Technical Field

[0001] This invention relates to the fields of e-commerce product recommendation, video recommendation, and news recommendation, and in particular to a knowledge graph recommendation method, device, and medium with efficient knowledge distillation. Background Technology

[0002] Recommender systems are widely used in various service platforms such as social media, video recommendation, and e-commerce to recommend items of interest to users based on their preferences, alleviating the problem of information overload. To predict user preferences from interaction data, classic collaborative filtering (CF) learns the latent representations of graph data structures and has achieved great success in practical recommendation scenarios. Examples include early matrix factorization, later neural network-based latent user and item embedding prediction techniques (such as Autorec, NCF, and ACF), and the recently rapidly developing CF models based on graph convolutional neural networks (such as NGCF and LightGCN).

[0003] Although the aforementioned CF-based recommendation algorithms demonstrate good performance in real-world scenarios, they still suffer from data sparsity and cold-start issues. To alleviate this problem, many works introduce knowledge graphs (KGs) into recommendations, providing rich auxiliary information for generating potential user and item embeddings and enhancing the representation process of users and items by encoding additional semantic relevance of items. While existing KG-based recommendation methods have achieved good results, they still have two shortcomings: 1) Real-time performance issues. In addition to traditional user interaction information, KG-based recommendation methods also introduce a large amount of auxiliary information through triples {h,r,t} to provide contextual information and assist in recommendations. This undoubtedly increases the amount of data processing. At the same time, existing popular KG-based recommendation algorithms often require a large amount of complex computation to fully utilize and capture valuable information in the KG, which seriously increases the inference time of the model and affects the real-time performance of recommendations. For example, KGCL, KGIN, and KACL all require precise representations generated from knowledge graph triples {h,r,t} during the prediction phase. After a series of complex calculations, they generate user or item embeddings to calculate user preferences for items. Compared to recommendation methods that don't incorporate KG information, this introduces significant additional computational overhead. 2) Noise issues in knowledge graphs. In real-world applications, knowledge graphs are often sparse and noisy, containing numerous topic-irrelevant connections between items and entities. Because the Trans algorithm requires a sufficient number of recommendation-related triples {h,r,t} linking to an entity to accurately simulate semantic transformations in KG, current KG models are working to address this issue and improve model robustness, but an effective solution has yet to be found.

[0004] In the past, knowledge distillation (KD) has been effectively applied in other fields such as computer vision and natural language processing. Early work transferred knowledge from the teacher model to the student model at the prediction level, while later work proposed transferring knowledge from the teacher model to the intermediate layer. Knowledge distillation is a highly efficient method for transferring knowledge from a large model to a small model. Inspired by this, GLCF applied KD to the recommender system domain, combining contrastive learning and knowledge distillation to achieve adaptive knowledge transfer from GNN models to MLP models. This compressed the original GNN model while improving its performance. However, KG-based tasks involve not only highly sparse user and item interaction data but also sparse knowledge graph triple data, which increases the difficulty of knowledge distillation. Specifically, extracting only user and item-related knowledge while ignoring knowledge graph-related knowledge makes it difficult to improve the performance of the compressed KG model. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a knowledge graph recommendation method, device and medium with high efficiency knowledge distillation.

[0006] The first technical solution adopted in this invention is:

[0007] An efficient knowledge graph recommendation method using knowledge distillation includes the following steps:

[0008] Knowledge is transferred from a large knowledge graph (KG)-based teacher model to a small single-layer graph attention network (GAT) student model through four distillation tasks. The four distillation tasks include:

[0009] 1) Distillation of user embedding and item embedding;

[0010] 2) Angular distillation of user embedding and item embedding;

[0011] 3) Distillation of user and item prediction results;

[0012] 4) Distillation of attention information in knowledge graphs;

[0013] Obtain the input data, input it into the student model obtained through four distillation tasks, and output the recommendation results.

[0014] Furthermore, the knowledge graph consists of two graphs: a user-item interaction graph and an item-item interaction knowledge graph;

[0015] Leveraging the richness of information mining through knowledge graph recommendation models and the simplicity and efficiency of single-layer attention networks, knowledge is distilled from a knowledge graph-based teacher model to a single-layer graph attention network-based student model. The framework of the teacher model is shown in the following formula:

[0016]

[0017] In the formula, This represents the embedding of the teacher model user u. Let represent the attention score of item i for user u in the teacher model. Let f(·) represent the embedding of item i in the teacher model, and let N be the aggregator of the user-item interaction graph in the teacher model. u In the user-item interaction graph, the neighbors of user u are represented. This represents the attention score of entity e to item i in the teacher model. Let N represent the embedding of entity e in the teacher model, g(·) represent the aggregator of the knowledge graph in the teacher model, and N represent the embedding of entity e in the teacher model. i Represents the neighbors of item i in the knowledge graph; The embedding of the teacher model relation r represents the teacher model relation r. A function representing the calculation of the teacher model's attention score;

[0018] The framework of the student model is shown below:

[0019]

[0020] In the formula, Represents the embedding of student model item i. This represents the attention score of entity e to item i in the student model. This represents the embedding of the student model entity e.

[0021] Furthermore, the distillation of the user embedding and item embedding includes:

[0022] A contrastive learning approach is used to perform knowledge distillation on the embeddings of users and items, sampling a batch of users and items from the user-item interaction graph at each training step. And apply the following contrast loss to the corresponding user / item embeddings:

[0023]

[0024] In the formula, u i This represents the i-th user. Represents the embedding of user i in the student model. Let τ1 represent the l-th layer GNN embedding of user i in the teacher model, τ1 represent the temperature hyperparameter of distillation, and L represent the total number of GNN aggregation layers. Represents the embedding of user i′ in the student model, u i′ This represents the i′-th user. Represents a set of users; v j This represents the j-th item. Represents the embedding of student model item j. v represents the l-th layer GNN embedding of item j in the teacher model; j′ This represents the j′-th item. Represents a collection of items. This represents the embedding of student model item j′.

[0025] Furthermore, the angular distillation of the user embedding and item embedding includes:

[0026] By distilling the embeddings of users, positive items, and negative items corresponding to the teacher and student models from the perspective difference, the perspective difference of the three embeddings is calculated to keep the two models consistent in terms of embedding distance.

[0027] In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v k Samples are taken individually from the overall user and item set with uniform probability to calculate the teacher model and student model (u i ,v j ) and (u i ,v k The angle between them is shown in the following formula:

[0028]

[0029] In the formula, w i,j,k This represents the angle between user i, positive item j, and negative item k. This represents the distance between the standardized embeddings of user i and the positive sample j of item. The distance between the standardized embeddings of user i and the negative sample k of item is represented by x, where T represents the transpose. i x represents the embedding of user i. j Let x represent the embedding of the positive sample j of item j. k Let represent the embedding of the negative sample k of item, and ‖·‖2 represent the L2 paradigm;

[0030] The student model is represented as The teacher model is represented as follows Distillation is performed by minimizing the HuberLoss loss function:

[0031]

[0032] In the formula, u i Let v represent the i-th user. j Let v represent the j-th item. k This represents the k-th item.

[0033] Furthermore, the distillation of the user and item prediction results includes:

[0034] In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v k Samples are taken individually from the overall user and item set with uniform probability to calculate the teacher model and student model (u i ,v j ) and (u i ,v k The difference in preferences between them is shown in the following formula:

[0035]

[0036] In the formula, z i,j,k Represents a triple (u i ,v j ,v k User preference score, y i,j y represents user i's predicted preference for item positive sample j. i,k x represents user i's predicted preference for item negative sample k. i x represents the embedding of user i. j Let x represent the embedding of the positive sample j of item j. k Represents the embedding of the negative sample k of item;

[0037] The scores given by the student model are represented as The scores given by the teacher model are represented as follows Distillation of user and item prediction results is performed by minimizing the following loss function:

[0038]

[0039] In the formula, τ2 represents the temperature hyperparameter.

[0040] Furthermore, the distillation of the knowledge graph attention information includes:

[0041] The relationship between the triples in the teacher model is defined as attention, as shown in the following formula:

[0042]

[0043] In the formula, LeakyReLU represents a nonlinear activation function. The embedding of the teacher model relation r represents the teacher model relationship. This represents the embedding of the teacher model item i. The embedding of the teacher model entity e is represented. This indicates that feature transformation is performed after embedding and concatenating the teacher model item i and entity e;

[0044] The student model's attention is initialized directly, denoted as .

[0045] Distillation of the teacher model's attention information using the L2 paradigm:

[0046]

[0047] In the formula, M is the amount of attention information.

[0048] Furthermore, the knowledge graph recommendation method also includes the following steps:

[0049] We implement collaborative filtering-based supervision and introduce additional recommendation loss to learn user preferences, i.e., learn knowledge from hard labels.

[0050] Furthermore, the collaborative filtering-based supervision introduces an additional recommendation loss to learn user preferences, i.e., learning knowledge from hard labels, including:

[0051] User embedding after distillation and project embedding The user preference score is calculated using the inner product as the scoring function:

[0052]

[0053] BPRloss is used to capture preferences between user-item pairs to optimize the parameters of the student model:

[0054]

[0055] In the formula, O={(u i ,v j ,v k )|(u i ,v j )∈O + ,(u i ,v k )∈O -} is the observed interaction information O + and unobserved interaction information O - The training dataset consists of σ(), which represents the sigmoid function. This represents the predicted preference of user i for positive sample j in the student model. This represents the predicted preference of user i for the negative sample k of item in the student model;

[0056] With the reinforcement of four distillation tasks and the regularization paradigm, the overall optimization objective is as follows:

[0057]

[0058] In the formula, λ1, λ2, λ3, λ4, and λ5 are the weights of different distillation tasks and regularization paradigms; This represents the weight decay regularization for a single-layer graph attention network. This represents all the weight parameters of a single-layer GAT network. L represents F The square of the paradigm.

[0059] The second technical solution adopted in this invention is:

[0060] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement an efficient knowledge graph recommendation method for knowledge distillation as described above.

[0061] The third technical solution adopted in this invention is:

[0062] A computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement an efficient knowledge graph recommendation method for knowledge distillation as described above.

[0063] The fourth technical solution adopted in this invention is:

[0064] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.

[0065] The beneficial effects of this invention are as follows: This invention successfully transfers knowledge from a large knowledge graph (KG)-based teacher model to a small single-layer graph attention network (GAT) student model through four distillation methods. In addition to the classic knowledge distillation at the embedding and prediction levels, knowledge distillation is also achieved at the angular attribute and knowledge graph attention information levels. This not only improves the convergence speed of model training, but also extracts information from the knowledge graph more effectively, alleviating the data noise problem and cold start problem in KG. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of a knowledge graph-based teacher model in an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of a student model based on a single-layer graph attention network in an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of distillation for user embedding and item embedding in an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram of the angle distillation of user embedding and item embedding in an embodiment of the present invention;

[0071] Figure 5 This is a schematic diagram of the distillation of user and item prediction results in an embodiment of the present invention;

[0072] Figure 6 This is a schematic diagram of the distillation of attention information in the knowledge graph in an embodiment of the present invention;

[0073] Figure 7 This is a flowchart illustrating the steps of an efficient knowledge distillation method for recommending knowledge graphs, as described in an embodiment of the present invention. Detailed Implementation

[0074] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0075] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0076] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0077] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0078] Terminology Explanation:

[0079] KGCL: A comparative learning model for knowledge graphs used for recommendations.

[0080] KGIN: A model that uses knowledge graphs to learn the intent behind interactions for recommendation purposes.

[0081] GLCF: Graphless Collaborative Filtering Model.

[0082] RKD: Relational Knowledge Distillation Model.

[0083] In summary, the research content of this invention is to explore some new methods for distilling knowledge graph recommendation models based on the characteristics of existing knowledge graphs and referring to existing knowledge distillation methods in other fields. The goal is to significantly reduce the time and space complexity of the original knowledge graph recommendation model while retaining the characteristics of the original model, such as still being able to mine a large amount of information between items, still improving the cold start problem, and further enhancing the performance of the model to surpass that of pre-trained teacher models.

[0084] In its research on recommendation algorithms based on knowledge graph knowledge distillation, this invention, based on the current state of knowledge distillation research and the characteristics of knowledge graph recommendation algorithms, proposes the following key issues to be addressed:

[0085] (1) There is a lot of auxiliary information between items in a knowledge graph. Existing knowledge graph recommendation models can effectively extract this information and improve the accuracy of recommendations. How to extract as much information as possible after knowledge distillation of these models will be a key issue.

[0086] (2) While performing knowledge distillation on the model to reduce the time and space complexity of model reasoning, this invention also aims to improve the performance of the student model and surpass the teacher model, which will be a major challenge in the research process.

[0087] (3) There is currently little research on knowledge distillation for knowledge graphs. We need to refer to knowledge distillation in other fields and propose a knowledge distillation method that is applicable to knowledge graph recommendation models and applicable to any model related to knowledge graphs. This is also a key problem that needs to be solved urgently.

[0088] Based on this, this invention proposes a novel, highly efficient knowledge distillation framework based on knowledge graph extraction, named EKRD. This framework eliminates the complex operations between the {h,r,t} embeddings of the knowledge graph triples, mapping them to a single-layer graph attention network (GAT), significantly improving the performance and efficiency of the recommender. Specifically, for knowledge transfer unique to knowledge graphs, we propose an alignment framework based on an attention mechanism, mapping information from the {h,r,t} embeddings of the teacher model to the attention parameters of the student model, thereby effectively extracting rich auxiliary information from the knowledge graph and mitigating data noise issues. Furthermore, in addition to extracting knowledge from the soft labels of the teacher model at the prediction layer, we also extract higher-order embeddings of users and items at the embedding layer, ensuring that the extracted knowledge comes not only from the teacher model's predictions but also from the potential higher-order collaborative semantics stored in the embeddings. Moreover, we propose an embedding angle alignment framework to further enhance knowledge transfer between embeddings while mitigating the interference of over-smoothing and noise effects.

[0089] Example 1

[0090] like Figure 7 As shown, this embodiment provides an efficient knowledge graph recommendation method using knowledge distillation, including the following steps:

[0091] Knowledge is transferred from a large knowledge graph (KG)-based teacher model to a small single-layer graph attention network (GAT) student model through four distillation tasks. The four distillation tasks include:

[0092] 1) Distillation of user embedding and item embedding;

[0093] 2) Angular distillation of user embedding and item embedding;

[0094] 3) Distillation of user and item prediction results;

[0095] 4) Distillation of attention information in knowledge graphs;

[0096] Obtain the input data, input it into the student model obtained through four distillation tasks, and output the recommendation results.

[0097] The EKRD proposed in this invention successfully transfers knowledge from a large knowledge graph (KG)-based teacher model to a small single-layer graph attention network (GAT) student model through four distillation methods. In addition to the classic knowledge distillation at the embedding and prediction levels, this embodiment also performs knowledge distillation at the angular attribute and knowledge graph attention information levels. This not only improves the convergence speed of model training but also more effectively extracts information from the knowledge graph, mitigating the data noise and cold-start problems inherent in KG models. This invention provides new research possibilities for the application of knowledge distillation in recommender systems. Extensive comprehensive experiments demonstrate that, compared to state-of-the-art models, EKRD not only effectively improves recommendation accuracy but also significantly enhances recommendation efficiency.

[0098] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0099] (1) Teacher Model and Student Model

[0100] A knowledge graph mainly consists of two parts: a user-item interaction graph and an item-item interaction graph. It is based on an adjacency matrix. Building a user-item interaction graph The node set If y u,i =1 then in Generate an edge (u, i) in the middle. Let the triplet be... This represents a knowledge graph. Leveraging the richness of information mining in knowledge graph recommendation models and the simplicity and efficiency of single-layer attention networks (GAT), this embodiment distills knowledge from a knowledge graph-based teacher model to a single-layer GAT-based student model. (See [link to relevant documentation]). Figure 1 The framework of the teacher model is shown in the following formula:

[0101]

[0102] See Figure 2 The framework of the student model is shown below:

[0103]

[0104] in addition, Simply initialize it.

[0105] The research method of this invention mainly includes four aspects: distillation of user embedding and item embedding, angle distillation of user embedding and item embedding, distillation of user and item prediction results, and distillation of attention information in knowledge graph.

[0106] (2) Distillation of User Embedding and Item Embedding

[0107] See Figure 3 This embodiment employs a contrastive learning approach to perform knowledge distillation on the embeddings of users and items. Specifically, in each training step, a batch of users and items are sampled from the user-item interaction graph. And apply the following contrast loss to the corresponding user / item embeddings:

[0108]

[0109] In order to distill more knowledge from the student model from the lack of higher-order patterns in collaborative filtering based on a single-layer GAT, this embodiment uses only the higher-order node embeddings of the teacher model.

[0110] (3) Angle distillation of user embedding and item embedding

[0111] Simply distilling higher-order patterns in user and item embeddings is insufficient; distillation of the angles between user and item embeddings is also necessary to extract more hidden information from the embeddings. Inspired by RKD, see [link to RKD documentation]. Figure 4 This embodiment uses angle difference distillation to calculate the angle differences between the embeddings of users, positive items, and negative items corresponding to the teacher and student models, ensuring that the two models maintain consistency in embedding distance. In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v kSamples are taken individually from the overall user and item set with uniform probability to calculate the teacher model and student model (u i ,v j ) and (u i ,v k The angle between them is shown in the following formula:

[0112]

[0113] In this embodiment, the angle of the student model is represented as... The teacher model is represented as follows Then distillation is performed by minimizing the Huber Loss loss function:

[0114]

[0115] (4) Distillation of User and Item Prediction Results

[0116] Inspired by the BPR loss for ranking in recommender systems, this embodiment adjusts two models for the task of ranking user preferences. See [link to relevant documentation] Figure 5 In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v k Samples are taken individually from the overall user and item set with uniform probability, and then the results are calculated in the teacher model and the student model (u i ,v j ) and (u i ,v k The difference in preferences between them is shown in the following formula:

[0117]

[0118] In this embodiment, the score given by the student model is represented as... The scores given by the teacher model are represented as follows Then, the distillation of user and item prediction results is performed by minimizing the following loss function:

[0119]

[0120] (5) Distillation of attention information in knowledge graphs

[0121] The most distinctive feature of knowledge graphs compared to other recommendation models is their introduction of a large amount of auxiliary information. Knowledge graph-based recommendation models not only mine information between users and items, but also between items, effectively mitigating the cold start and oversmoothing problems in recommendation systems. Therefore, it is important to analyze the triples in the knowledge graph. The mutual information between them is distilled.

[0122] See Figure 6 First, in this embodiment, the relationship between the teacher model triples is defined as attention, as shown in the following formula:

[0123]

[0124] The student model's attention is initialized directly, denoted as .

[0125] Then, the attention information of the teacher model is distilled using the L2 paradigm:

[0126]

[0127] Where M is the amount of attention information.

[0128] (6) Model prediction and optimization

[0129] Finally, we implement collaborative filtering-based supervision, introducing an additional recommendation loss to learn user preferences, i.e., learning knowledge from hard labels. Specifically, we analyze the distilled user embeddings... and project embedding The user preference score is calculated using the inner product as the scoring function:

[0130]

[0131] To optimize the parameters of the student model, we employ BPRloss to capture preferences between user-item pairs. Generally, it assumes that for a given user, the predicted score for their historical items should be higher than that for unobserved items.

[0132]

[0133] Among them, O={(u i ,v j ,v k )|(u i ,v j )∈O + ,(u i ,v k )∈O -} is the observed interaction information O + and unobserved interaction information O - The training dataset consists of four parts; σ() represents the sigmoid function. Under the reinforcement of the four distillation tasks and the regularization paradigm, the overall optimization objective is as follows:

[0134]

[0135] Where λ1, λ2, λ3, λ4, and λ5 are the weights for different distillation tasks and regularization paradigms. This represents the weight decay regularization of a single-layer GAT network.

[0136] Example 2

[0137] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 7 This paper presents an efficient knowledge graph recommendation method based on knowledge distillation.

[0138] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0139] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0140] Since this electronic device is the electronic device corresponding to the efficient knowledge distillation knowledge graph recommendation method of this invention, and the principle of this electronic device in solving the problem is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0141] Example 3

[0142] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 7 This paper presents an efficient knowledge graph recommendation method based on knowledge distillation.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0144] Since this storage medium is the storage medium corresponding to the efficient knowledge distillation knowledge graph recommendation method of this invention, and the principle of this storage medium in solving the problem is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0145] Example 4

[0146] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of an efficient knowledge distillation knowledge graph recommendation method according to various exemplary embodiments of this application as described above. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0147] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0148] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0149] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A knowledge graph recommendation method with high efficiency in knowledge distillation, characterized in that, Includes the following steps: Through four distillation tasks, knowledge is transferred from a large knowledge graph-based teacher model to a small single-layer graph attention network student model. The four distillation tasks include: 1) distillation of user embeddings and item embeddings; 2) distillation of user embeddings and item embeddings from different angles; 3) distillation of user and item prediction results; and 4) distillation of knowledge graph attention information. Obtain input data and input it into the student model trained through four distillation tasks. The student model then outputs recommendation results. The knowledge graph recommendation method also includes the following steps: User embedding after distillation and project embedding The user preference score is calculated using the inner product as the scoring function: BPRloss is used to capture preferences between user-item pairs to optimize the parameters of the student model: In the formula, O={(u i ,v j ,v k )|(u i ,v j )∈O + ,(u i ,v k )∈O - } is the observed interaction information O + and unobserved interaction information O - The training dataset consists of σ(), which represents the sigmoid function. This represents the predicted preference of user i for positive sample j in the student model. This represents the predicted preference of user i for the negative sample k of item in the student model; With the reinforcement of four distillation tasks and the regularization paradigm, the overall optimization objective is as follows: In the formula, λ1, λ2, λ3, λ4, and λ5 are the weights of different distillation tasks and regularization paradigms; The distillation loss function represents the user embedding and item embedding. The angle distillation loss function represents the user embedding and item embedding. The distillation loss function represents the prediction results for users and items. The distillation loss function represents the attention information in a knowledge graph; This represents the weight decay regularization for a single-layer graph attention network. This represents all the weight parameters of a single-layer GAT network. L represents F The square of the paradigm.

2. The knowledge graph recommendation method with high efficiency knowledge distillation according to claim 1, characterized in that, The knowledge graph consists of two parts: a user-item interaction graph and an item-item interaction knowledge graph. Leveraging the richness of information mining through knowledge graph recommendation models and the simplicity and efficiency of single-layer attention networks, knowledge is distilled from a knowledge graph-based teacher model to a single-layer graph attention network-based student model. The framework of the teacher model is shown in the following formula: In the formula, This represents the embedding of the teacher model user u. Let represent the attention score of item i for user u in the teacher model. Let N represent the embedding of item i in the teacher model, f(·) represent the aggregator of the user-item interaction graph in the teacher model, and N represent the embedding of item i in the teacher model. u In the user-item interaction graph, the neighbors of user u are represented. This represents the attention score of entity e to item i in the teacher model. Let N represent the embedding of entity e in the teacher model, g(·) represent the aggregator of the knowledge graph in the teacher model, and N represent the embedding of entity e in the teacher model. i Represents the neighbors of item i in the knowledge graph; The embedding of the teacher model relation r represents the teacher model relationship. A function representing the calculation of the teacher model's attention score; The framework of the student model is shown below: In the formula, Represents the embedding of student model item i. This represents the attention score of entity e to item i in the student model. This represents the embedding of the student model entity e.

3. The knowledge graph recommendation method with high efficiency knowledge distillation according to claim 1, characterized in that, The distillation of the user embedding and item embedding includes: A contrastive learning approach is used to perform knowledge distillation on the embeddings of users and items, sampling a batch of users and items from the user-item interaction graph at each training step. And apply the following contrastive loss to the corresponding user / item embeddings: In the formula, u i This represents the i-th user. Represents the embedding of user i in the student model. Let τ1 represent the l-th layer GNN embedding of user i in the teacher model, τ1 represent the temperature hyperparameter of distillation, and L represent the total number of GNN aggregation layers. Represents the embedding of user i′ in the student model, u i′ This represents the i′-th user. Represents a set of users; v j This represents the j-th item. Represents the embedding of student model item j. v represents the l-th layer GNN embedding of item j in the teacher model; j′ This represents the j′-th item. Represents a collection of items. This represents the embedding of student model item j′.

4. The knowledge graph recommendation method with high efficiency knowledge distillation according to claim 1, characterized in that, The angle distillation of the user embedding and item embedding includes: By distilling the embeddings of user positive samples, item positive samples, and item negative samples corresponding to the teacher model and student model from the angle difference, the angle difference of the three embeddings is calculated to keep the two models consistent in embedding distance. In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v k Samples are taken individually from the overall user and item set with uniform probability to calculate the teacher model and student model (u i ,v j ) and (u i ,v k The angle between them is shown in the following formula: In the formula, w i,j,k This represents the angle between user i, positive item j, and negative item k. This represents the distance between the standardized embeddings of user i and the positive sample j of item. The distance between the standardized embeddings of user i and the negative sample k of item is represented by x, where T represents the transpose. i x represents the embedding of user i. j Let x represent the embedding of the positive sample j of item j. k Let represent the embedding of the negative sample k of item, and ‖·‖2 represent the L2 paradigm; The student model is represented as The teacher model is represented as follows Distillation is performed by minimizing the Huber Loss function: In the formula, u i Let v represent the i-th user. j Let v represent the j-th item. k This represents the k-th item.

5. The knowledge graph recommendation method with high efficiency knowledge distillation according to claim 1, characterized in that, The distillation of the user and item prediction results includes: In each training step, a batch of triples is randomly sampled. Where u i ,v j ,v k Samples are taken individually from the overall user and item set with uniform probability to calculate the teacher model and student model (u i ,v j ) and (u i ,v k The difference in preferences between them is shown in the following formula: In the formula, z i,j,k Represents a triple (u i ,v j ,v k User preference score, y i,j y represents user i's predicted preference for item positive sample j. i,k x represents user i's predicted preference for item negative sample k. i x represents the embedding of user i. j Let x represent the embedding of the positive sample j of item j. k Represents the embedding of the negative sample k of item; The scores given by the student model are represented as The scores given by the teacher model are represented as follows Distillation of user and item prediction results is performed by minimizing the following loss function: In the formula, τ2 represents the temperature hyperparameter.

6. The knowledge graph recommendation method with high efficiency knowledge distillation according to claim 1, characterized in that, Distillation of the knowledge graph attention information includes: The relationship between the triples in the teacher model is defined as attention, as shown in the following formula: In the formula, LeakyReLU represents a nonlinear activation function. The embedding of the teacher model relation r represents the teacher model relationship. This represents the embedding of the teacher model item i. The embedding of the teacher model entity e is represented. This indicates that feature transformation is performed after embedding and concatenating the teacher model item i and entity e; The student model's attention is initialized directly, denoted as . Distillation of the teacher model's attention information using the L2 paradigm: In the formula, M is the amount of attention information.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 6.

Images