Pruning method and device for graph neural network recommendation model and storage medium

By pruning based on the structural importance and information quality of edges in the graph neural network recommendation model, the problem of efficiency bottleneck in the graph neural network recommendation system is solved, and more efficient and accurate user item recommendation is achieved, which significantly improves the accuracy and Training speed.

CN118917352AActive Publication Date: 2024-11-08ZHEJIANG UNIV +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410941390.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

Recommendation systems based on graph neural networks are difficult to parallelize on large graphs, resulting in increased computing demand and efficiency bottlenecks. The existing simplified module methods are limited by design structures, and the knowledge distillation method only accelerates the inference process but extends the training time.

Method used

During the training process of recommended model, a binary graph is constructed by obtaining the information of interest from users and items, and the structural importance and information quality of edges are calculated. The edges with smaller contributions are deleted based on the pruning rate, and the graph structure is optimized to realize the pruning of the model. .

Benefits of technology

The model training efficiency and prediction accuracy are improved, and more efficient and accurate user item recommendations are achieved. The accuracy of the test results is improved by 7.7%, and the training speed is increased by 88%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917352A_ABST
    Figure CN118917352A_ABST
Patent Text Reader

Abstract

The invention discloses a pruning method and device for a graph neural network recommendation model and a storage medium, and relates to the field of model pruning and computer deep learning. According to given user set article interaction information, a constructed user-article bipartite graph and a pruning algorithm can calculate edge reliability, structural importance and information quality in a model training process so as to generate a score of each edge in the user-article bipartite graph, and the score is pruned so as to reduce the scale of the graph; model training is accelerated, the accuracy of model prediction is improved, and article recommendation is performed for the user based on a recommendation model of the pruned graph neural network. According to the method, edges with small contributions can be deleted in the training process of the recommendation model based on the graph neural network, the graph structure is optimized, and the aggregation speed is increased while information transmission is more effective.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of model pruning and computer deep learning, and in particular to a pruning method, device and storage medium for a graph neural network recommendation model. Background Art

[0002] Graph neural networks have shown significant effects in recommendation systems due to their ability to capture high-order collaborative signals. Existing recommendation models based on graph neural networks usually include the following process: 1) construct a bipartite graph based on user-item interactions; 2) apply a multi-layer graph neural network to the constructed graph to learn user and item embedding vectors; 3) generate predictions based on the similarity of the embedding vectors.

[0003] Although graph neural networks have shown effectiveness in recommendation systems, they also bring a large computational burden. Domain aggregation, as a core component, requires repeated calculation of information propagation between a large number of neighbor nodes in each batch, which has become a major efficiency bottleneck. Taking the classic LightGCN as an example, the introduction of graph neural networks has increased the computational requirements by 6 times. The main reason is that it is difficult to parallelize graph neural networks on large graphs. These problems hinder the practical deployment of recommendation systems based on graph neural networks.

[0004] To overcome this efficiency limitation, several strategies have been proposed: some recent works suggest using simplified modules to approximate graph neural networks. Although efficient, these methods are limited by specific design structures and perform poorly compared to recent advanced graph neural network-based techniques. Another strategy is to use knowledge distillation to transfer knowledge from graph neural network-based methods to non-graph neural network methods. However, this approach only speeds up the inference process and still prolongs the training time due to the need to train a graph neural network-based teacher model. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose a pruning method, device and storage medium for a graph neural network recommendation model, which performs pruning during the training of the recommendation model, thereby improving the efficiency of model training and the accuracy of model prediction, and achieving more efficient and accurate user item recommendations.

[0006] The objective of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a pruning method for a graph neural network recommendation model, the method comprising the following steps:

[0007] (1) Obtain a set of users, a set of items, and information about users’ interests in items, and construct a user-item bipartite graph with users and items as nodes and information about interests as edges;

[0008] (2) Calculate the structural importance of edges in the user-item bipartite graph based on the data of interest;

[0009] (3) Randomly initialize the embedding vectors of users and items, and set the pruning rate and number of iterations.

[0010] (4) Train the recommendation model based on graph neural network and update the embedding vectors of users and items;

[0011] (5) Based on the updated embedding vectors, the final embedding vectors of the user and item are obtained respectively; the reliability and information quality of the edges in the user-item bipartite graph are calculated;

[0012] (6) Comprehensively consider the structural importance, reliability, and information quality of the edges in the user-item bipartite graph, delete the edges in the user-item bipartite graph according to the set pruning rate, update the user-item bipartite graph, and complete one iteration;

[0013] (7) Repeat steps (4) to (6) for iterative updating until the number of iterations is reached, and obtain the user-item bipartite graph and the embedding vectors of users and items after the iteration is completed, which are used to recommend items to users based on the recommendation model based on the pruned graph neural network.

[0014] Furthermore, the structural importance of the edge (u,i) in the user-item bipartite graph The calculation method is:

[0015] 1) Construct an adjacency matrix

[0016] 2) Calculate A 2 The diagonal matrix Λ;

[0017] 3) Define the structural importance matrix B = A 3 -AΛ-ΛA+A;

[0018] 4) Definition B ui is the value of the u-th row and i-th column of the structural importance matrix B;

[0019] 5) Structural Importance S I (u,i)=B ui ;

[0020] 6) Calculate the structural importance of all edges in edge set E and obtain set S I ={S I (u,i)|(u,i)∈E};

[0021] 7) Calculate the structural importance S of edge (u,i) I (u,i), and normalize it:

[0022] Furthermore, the final embedding vector e of the user u and the final embedding vector e of the item i The calculation method is:

[0023] 1) Calculate the embedding vectors of users and items passing through the lth layer of the graph neural network: N u represents the neighbor set of user u, d u is the degree of user u, N i represents the neighbor set of item i; d i is the degree of item i;

[0024] 2) The final embedding vectors of users and items are defined as: L is the number of layers of the graph neural network.

[0025] Furthermore, the edge reliability S of the edge (u,i) in the user-item bipartite graph is R The calculation method of (u,i) is: calculate the co-similarity of the embedding vectors of the two nodes connected by the edge (u,i), that is, the edge reliability

[0026] Furthermore, the information quality of the edge (u,i) in the user-item bipartite graph is The calculation method is:

[0027] 1) Define the information quality from user u to item i The quality of information from item i to user u

[0028] 2) Define the information quality of edge (u,i)

[0029] 3) Calculate the information quality of all edges in edge set E and obtain set S Q ={S Q (u,i)|(u,i)∈E};

[0030] 4) Calculate the information quality S of edge (u,i) Q (u,i), and normalize it:

[0031] Furthermore, the score of each edge in the edge set E of the user-item bipartite graph is calculated The weight exponents α1 and α2 are hyperparameters, S R (u,i) is the edge reliability between user u and item i, is the normalized structural importance of the edge, is the information quality of the normalized edge; according to the pruning rate, delete the [|E|*p%] edges with the lowest scores to obtain a new edge set E′, and update the user-item bipartite graph.

[0032] In a second aspect, the present invention also provides a pruning device for a graph neural network recommendation model, comprising a memory and one or more processors, wherein the memory stores executable code, and wherein when the processor executes the executable code, it implements the pruning method for a graph neural network recommendation model.

[0033] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, characterized in that when the program is executed by a processor, the pruning method for a graph neural network recommendation model is implemented.

[0034] In a fourth aspect, the present invention also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the pruning method for a graph neural network recommendation model.

[0035] Beneficial effects of the present invention: According to the method of the present invention, in the training process of the recommendation model based on the graph neural network, the edges with smaller contributions can be deleted, the graph structure can be optimized, the information transmission can be made more efficient, and the aggregation speed can be accelerated, thereby achieving more efficient and accurate user item recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a flow chart of a pruning method for a graph neural network recommendation model provided by the present invention;

[0038] Figure 2 It is a structural diagram of a pruning device for a graph neural network recommendation model provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] In order to explain the specific implementation of the present invention in detail, the Amazon-book dataset is used as an example. The dataset contains 52,643 users, 91,599 items (the items in this invention are books on the Amazon platform), and 2,984,108 interaction records between users and books on the platform, such as user clicks on books of interest. Figure 1 As shown, the present invention provides a method comprising the following steps:

[0041] 1. Given the user set U, item set I and click rate matrix According to actual needs or experience, the pruning ratio p% is set to 50%, and the number of iterations T is set to 1000. The user set U contains 52643 elements, the item set I contains 91599 items, and the click rate matrix R contains 2984108 interaction records.

[0042] 2. Construct a point set V = U ∪ I containing the user set and the item set, and an edge set E = {(u,i)|u∈U,i∈I,R ui =1}, and the user-item bipartite graph G(V,E), R ui is the value of the u-th row and i-th column of the click rate matrix R. A value of 1 indicates that user u clicked on item i, and a value of 0 indicates that user u did not click on item i. For user u∈U, the neighbor set of user u is defined as N u ={i|R ui =1}, the degree d of user u u =|N u |; For item i∈I, define the neighbor set of item i as N i = {u|R ui =1}, the degree d of item i i =|N i |;

[0043] 3. Calculate the structural importance of each edge (u,i) in set E

[0044] The calculation method of structural importance is: first construct the adjacency matrix Calculate A 2 The diagonal matrix Λ of the structure is then calculated as the structural importance matrix B = A 3 -AΛ-ΛA+A, get the structural importance S I (u,i)=B ui , B ui is the value of the u-th row and i-th column of the structural importance matrix B, indicating the number of quadrilaterals that the edge (u,i) can form with other edges in the user-item bipartite graph G(V,E); it is necessary to calculate the structural importance of all edges in the edge set E and obtain the set S I ={SI (u,i)|(u,i)∈E}, then the structural importance S of edge (u,i) I (u,i) is normalized:

[0045] 4. The embedding vector of user u is represented as The embedding vector of item i is represented as Randomly initialize the embedding vectors of all users and items, and the size of each embedding vector is set to 64;

[0046] 5. Set the size of each batch to 2048, and one training run contains batch, train a recommendation model based on graph neural network once according to the user-item bipartite graph G(V,E), that is, the user-item bipartite graph G(V,E), and update the user's embedding vector and the embedding vector of the item

[0047] In an embodiment of the present invention, the LightGCN model is used for training to update the embedding vector of user u and the embedding vector of item i The LightGCN described above uses a 3-layer network. Among them, the LightGCN model is used for training to update the user's embedding vector and the embedding vector of the item This is the prior art in this field and will not be described in detail here.

[0048] 6. Based on the updated user embedding vector and item embedding vector, calculate the user's final embedding vector e u and the final embedding vector e of the item i ;

[0049] The calculation method is: first calculate the embedding vectors of users and items that pass through different layers of graph neural networks: Then calculate the final embedding vectors of users and items:

[0050] 7. According to the user's final embedding vector e u and the final embedding vector e of the item i , calculate the edge reliability S of each edge (u,i) in the set E R (u,i);

[0051] The edge reliability is the co-similarity of the final embedding vectors of the two nodes connected by the edge (u, i), and the calculation formula is:

[0052]

[0053] 8. Based on the user's final embedding vector e u and the final embedding vector e of the item i , calculate the information quality of each edge (u,i) in the set E

[0054] Information Quality The calculation method is: according to the information quality from user u to item i and the quality of information from item i to user u Calculate the information quality of all edges in E Get the set S Q ={S Q (u,i)|(u,i)∈E}; then the information quality S of edge (u,i) Q (u,i) is normalized:

[0055] 9. According to the calculation formula of the score of edge (u,i) The weight indexes α1 and α2 are hyperparameters. According to experience, α1 = 0.75 and α2 = 0.25 are set to calculate the score of each edge in the set E.

[0056] 10. Delete the [|E|*p%] edges with the lowest scores, which are [2984108*50%] edges in this invention, to obtain a new edge set E′, and update the user-item bipartite graph G(V,E′);

[0057] 11. Iterate steps 5-10 1000 times, and finally use the user-item bipartite graph G(V,E′) and the embedding vectors of users and items after 1000 iterations for prediction of the LightGCN model to recommend items to users.

[0058] 12. Test results: The LightGCN model after applying the method of the present invention was tested on the test set of the Amazon-book dataset. Compared with the original LightGCN model, the accuracy was improved by 7.7% and the training speed was improved by 88%.

[0059] Corresponding to the aforementioned embodiment of a pruning method for a graph neural network recommendation model, the present invention also provides an embodiment of a pruning device for a graph neural network recommendation model.

[0060] See also Figure 2A pruning device for a graph neural network recommendation model provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a pruning method for a graph neural network recommendation model in the above embodiment.

[0061] An embodiment of a pruning device for a graph neural network recommendation model provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From a hardware perspective, if Figure 2 As shown, it is a hardware structure diagram of a pruning device for a graph neural network recommendation model provided by the present invention, in which any device with data processing capability is located, except Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0062] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0063] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0064] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a pruning method for a graph neural network recommendation model in the above embodiment is implemented.

[0065] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0066] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the pruning method for a graph neural network recommendation model.

[0067] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A pruning method for a graph neural network recommendation model, characterized in that: The method comprises the following steps: (1) Obtain a set of users, a set of items, and information about users’ interests in items, and construct a user-item bipartite graph with users and items as nodes and information about interests as edges; (2) Calculate the structural importance of edges in the user-item bipartite graph based on the data of interest; (3) Randomly initialize the embedding vectors of users and items, and set the pruning rate and number of iterations. (4) Train the recommendation model based on graph neural network and update the embedding vectors of users and items; (5) Based on the updated embedding vectors, the final embedding vectors of the user and item are obtained respectively; the reliability and information quality of the edges in the user-item bipartite graph are calculated; (6) Comprehensively consider the structural importance, reliability, and information quality of the edges in the user-item bipartite graph, delete the edges in the user-item bipartite graph according to the set pruning rate, update the user-item bipartite graph, and complete one iteration; (7) Repeat steps (4) to (6) for iterative updating until the number of iterations is reached, and obtain the user-item bipartite graph and the embedding vectors of users and items after the iteration is completed, which are used to recommend items to users based on the recommendation model based on the pruned graph neural network.

2. A pruning method for a graph neural network recommendation model according to claim 1, characterized in that: Structural importance of edge (u, i) in user-item bipartite graph The calculation method is: 1) Construct an adjacency matrix 2) Calculate A 2 The diagonal matrix Λ; 3) Define the structural importance matrix B = A 3 -AΛ-ΛA+A; 4) Definition B ui is the value of the u-th row and i-th column of the structural importance matrix B; 5) Structural importance S I (u, i) = B ui ; 6) Calculate the structural importance of all edges in edge set E and obtain set S I ={S I (u, i)|(u, i)∈E}; 7) Calculate the structural importance S of edge (u, i) I (u, i), and normalize it:

3. A pruning method for a graph neural network recommendation model according to claim 1, characterized in that: The final embedding vector e of the user u and the final embedding vector e of the item i The calculation method is: 1) Calculate the embedding vectors of users and items passing through the lth layer of the graph neural network: N u represents the neighbor set of user u, d u is the degree of user u, N i represents the neighbor set of item i; d i is the degree of item i; 2) The final embedding vectors of users and items are defined as: L is the number of layers of the graph neural network.

4. A pruning method for a graph neural network recommendation model according to claim 3, characterized in that: The edge reliability S of the edge (u, i) in the user-item bipartite graph R The calculation method of (u, i) is: calculate the co-similarity of the embedding vectors of the two nodes connected by the edge (u, i), that is, the edge reliability 5. A pruning method for a graph neural network recommendation model according to claim 3, characterized in that: The information quality of the edge (u, i) in the user-item bipartite graph The calculation method is: 1) Define the information quality from user u to item i The quality of information from item i to user u 2) Define the information quality of edge (u, i) 3) Calculate the information quality of all edges in edge set E and obtain set S Q ={S Q (u, i)|(u, i)∈E}; 4) Calculate the information quality S of edge (u, i) Q (u, i), and normalize it:

6. A pruning method for a graph neural network recommendation model according to claim 1, characterized in that: Calculate the score of each edge in the edge set E of the user-item bipartite graph The weight indexes α1 and α2 are hyperparameters, S R (u, i) is the edge reliability between user u and item i, is the normalized structural importance of the edge, is the information quality of the normalized edge; according to the pruning rate, delete the [|E|*p%] edges with the lowest scores to obtain a new edge set E′, and update the user-item bipartite graph.

7. A pruning device for a graph neural network recommendation model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a pruning method for a graph neural network recommendation model as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a pruning method for a graph neural network recommendation model as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a pruning method for a graph neural network recommendation model as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Inductive graph neural network pruning method, system and device and storage medium

    CN114997378A

  • Heterogeneous graph neural network noise tolerance method based on Transform and comparative learning

    CN117787379A

  • Graph neural network pruning method and equipment based on distillation sub-graph and storage medium

    CN117933302A

  • Adaptive graph neural network joint sparse framework and method

    CN118095344A

  • Automatic channel pruning via graph neural network based hypernetwork

    US20230084203A1