Recommendation system-oriented attention network architecture and optimization method thereof

Abstract

The invention discloses a recommendation system-oriented attention network architecture and an optimization method thereof, which are characterized in that a recommendation system of a self-attention mechanism and a residual network architecture is adopted, and a multi-level semantic relationship between a user and an article is constructed by using a knowledge graph through a graph convolutional network model constructed layer by layer; multi-hop neighbor information between a user and an article is aggregated through a basic self-attention mechanism to dynamically capture potential interest association of the user to aggregate multi-level and high-order association information between the user and the article, and dynamic personalized weight adjustment and stable information transmission are realized in combination with the self-attention mechanism and residual connection. Compared with the prior art, the method has the advantages of good personalized expression, high recommendation precision and high calculation efficiency, not only reduces the calculation complexity of a recommendation system on a large-scale knowledge graph, but also enhances the diversity and interpretability of recommendation results, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of recommendation systems, and in particular to an architecture of an attention network for recommendation systems and an optimization method thereof. Background Art

[0002] With the development of Internet technology, recommendation systems have become an important tool to help users filter information and reduce information overload. Traditional recommendation system methods are mainly based on collaborative filtering, which generates recommendation results through historical interaction data between users and items. However, collaborative filtering is not good at dealing with data sparsity and cold start problems. In addition, methods based on collaborative filtering cannot effectively capture the semantic relationship between users and items. In order to solve the above problems, knowledge graphs are gradually introduced into recommendation systems. Knowledge graphs connect the attributes of users and items to form a graph structure containing rich semantic information, which can enhance the recommendation ability of recommendation systems. At present, existing knowledge graph embedding methods attempt to map entities and relationships in knowledge graphs to low-dimensional spaces, which are then used in recommendation systems. However, traditional KGE methods mainly focus on relational reasoning within knowledge graphs (such as link prediction), which is difficult to be directly used for modeling user interests in recommendation systems.

[0003] In recent years, the application of graph neural networks in graph data modeling has gradually increased, especially graph convolutional neural networks have shown potential in processing high-order structures and semantic information. The GCN method integrates the local neighborhood information of each node into the node representation through a neighbor aggregation mechanism. This method has received widespread attention and application in recommendation systems. However, existing GCN-based knowledge graph recommendation methods, such as RippleNet and PER, have problems such as high computational overhead, reliance on manually designed meta-paths, and difficulty in effectively modeling relationship weights, which affects the performance on large-scale datasets.

[0004] On the other hand, attention mechanism has been widely used as an important means to enhance the expressiveness of models. The attention mechanism was first successful in the field of natural language processing. It highlights key information by assigning weights, thereby improving the efficiency of the model in processing long texts and multi-relational data. For recommendation systems, attention mechanism helps the model capture more accurate preference information in the interaction between users and items. In the context of applying graph convolutional neural networks (GCN) to knowledge graph recommendation systems, attention mechanism is particularly helpful to enhance the model's capture of users' personalized interests. Typical methods include graph attention networks, which aggregate information by assigning different weights to neighbor nodes, thereby distinguishing the importance of different relationships and neighbors. In addition, attention mechanism has gradually been introduced into recommendation methods based on knowledge graphs. For example, in the RippleNet model, multi-layer memory networks are used to propagate user preferences, and different attention weights are assigned to each layer to distinguish the importance of relationships. This method overcomes the problem of uneven information propagation in traditional GCNs in complex graph structures to a certain extent.

[0005] In summary, existing recommendation systems have data sparsity, cold start problems, and insufficient expression of high-order associations and user personalized interests. Although knowledge graphs, graph convolutional networks, and attention mechanisms have shown significant advantages in recommendation systems, they still face many limitations in practical applications. These limitations are specifically manifested in the following aspects:

[0006] 1) Computational complexity and efficiency issues: As the scale of recommendation systems and knowledge graphs expands, the computational overhead of recommendation methods based on graph convolution and attention mechanisms on large-scale knowledge graphs is huge. After the graph convolutional neural network (GCN) is combined with the attention mechanism, the neighbors of each node need to calculate personalized attention weights, and the increase in the number of neighbor nodes and the number of depth layers of the graph structure will cause the computational complexity to grow exponentially. This low computational efficiency not only leads to high training costs, but also seriously affects the response speed of the model in real-time recommendation systems. Therefore, current recommendation systems based on GCN and attention mechanisms often require neighbor sampling or model simplification in large-scale applications, but this will lead to information loss and affect the accuracy of recommendations. At the same time, even if sampling or simplification is performed, the computational cost of the model is still higher than that of traditional recommendation algorithms, making it difficult to meet real-time requirements.

[0007] 2) Weight stability and noise issues of the attention mechanism: In the knowledge graph, the neighbor relationships of nodes are complex and diverse. The attention mechanism is easily disturbed by noise data when calculating the weights of different relationships, resulting in unstable weight distribution. This instability is more prominent in situations where the data is sparse or noisy. For example, when the user's historical interaction data is limited, it may be difficult for the model to accurately learn the user's true preferences, resulting in incorrect weight distribution. In addition, when the attention mechanism aggregates high-order relationship information layer by layer in a multi-layer GCN, there is a cumulative effect of noise, which will make the higher-order node representations more seriously affected by noise, resulting in a decrease in the accuracy and stability of the recommendation results. Existing models lack effective mechanisms to suppress the propagation of noise, especially in deep models, where noise may be amplified during multi-layer propagation, affecting the generalization performance of the model.

[0008] 3) Limitations of personalized expression: The attention mechanism in existing graph convolutional recommendation systems mostly uses a fixed weight calculation method. Although this method can take into account the personalized needs of users to a certain extent, it still has limitations. Current models usually calculate attention weights based on the global information of user-item interactions, and lack a fine-grained expression of the dynamic changes in user preferences under different semantic relationships. For example, different users may focus on different attributes (such as type, director, or actor) when facing the same item, and existing models cannot flexibly adjust attention weights based on the focus of specific users. Therefore, it is difficult for existing methods to effectively generate more personalized recommendation results, resulting in insufficient diversity of recommendations. In addition, user interests change dynamically, and the static attention mechanism of existing models is difficult to adapt to real-time changes in user preferences, which limits the personalization and accuracy of recommendation results.

[0009] 4) Limitations in capturing high-order relationship information: High-order relationships in knowledge graphs play an important role in describing complex user interests, but in practical applications, it is difficult to capture and effectively utilize high-order relationships. Existing GCN and attention mechanisms mainly focus on first-order or second-order neighbor information, and information gradually decays at higher levels, resulting in insufficient modeling of long-distance dependencies. When the model attempts to capture higher-order relationships, the noise accumulation and loss of information are obvious, making it impossible for high-order relationships to effectively enhance recommendation accuracy. In addition, since the weighting method of the attention mechanism easily distracts attention among high-order neighbors, it makes it difficult for the model to focus on a few key nodes, which in turn affects the quality of recommendations. Therefore, existing recommendation systems based on GCN and attention have great limitations in deep information propagation and high-order dependency modeling.

[0010] 5) Lack of explainability and transparency: In the recommendation system, the explainability and transparency of the recommendation results are particularly important, especially when users have doubts about the rationality of the recommendation results. However, the attention mechanism and graph convolutional network perform complex information aggregation in a multi-level, multi-relational structure, making the model decision process difficult to trace and understand. Most of the existing attention-based GCN models focus on the recommendation effect itself and lack the design to explain the recommendation decision, resulting in the lack of explainability of the model when generating recommendations, and it is difficult for users to understand the basis of the recommendation results. This lack of explainability limits the application of the model, especially in scenarios that require high explainability (such as medical and financial recommendations). Summary of the invention

[0011] The purpose of the present invention is to provide an attention network architecture and an optimization method for a recommendation system in response to the deficiencies of the prior art. The present invention adopts a method of constructing a graph convolutional network model layer by layer, uses a knowledge graph to construct a multi-level semantic relationship between users and items, aggregates multi-hop neighbor information between users and items through a basic self-attention mechanism, and dynamically captures the user's potential interest associations. The multi-level and high-order association information between users and items is aggregated, and the self-attention mechanism and residual connection are combined to achieve dynamic personalized weight adjustment and stable information transmission. The knowledge graph is applied in the recommendation system through a graph convolutional neural network to improve the accuracy and diversity of recommendations. At the same time, the use of the attention mechanism further enhances the recommendation performance and effectively enhances the model's ability to capture user interests. The use of score prediction, negative sampling strategy and multiple evaluation indicators in this method ensures the accuracy and diversity of the model, enabling the recommendation system to meet the diverse needs of users and effectively solving the problems of data sparsity and cold start. While ensuring the accuracy of recommendations, it reduces the computational overhead of the model, enabling the model to achieve fast recommendations and real-time updates on large-scale knowledge graphs, greatly improving the personalized expression of user preferences. The method applies knowledge graphs in the recommendation system through graph convolutional neural networks to improve the accuracy and diversity of recommendations. At the same time, the use of attention mechanisms further enhances the recommendation performance, thereby better meeting the needs of actual applications and having good application prospects.

[0012] The purpose of the present invention is to achieve the following: an attention network architecture for a recommendation system and an optimization method thereof, wherein the recommendation system adopts a self-attention mechanism and a residual network architecture, and uses a graph convolutional network model constructed layer by layer to construct a multi-level semantic relationship between users and items using a knowledge graph, and aggregates multi-hop neighbor information between users and items through a basic self-attention mechanism to dynamically capture the user's potential interest associations and aggregate multi-level and high-order association information between users and items, and combines the self-attention mechanism and residual connection to achieve dynamic personalized weight adjustment and stable information transmission. The network architecture and its optimization method specifically include:

[0013] 1.1 Graph Convolutional Network Layer

[0014] Using the graph convolutional network layer as the basic framework, the graph convolutional network layer (GCN layer) is used to aggregate the neighbor information of users, items and their related attributes in the knowledge graph layer by layer in order to construct a multi-level representation of users and items. Through the graph convolutional network layer, the high-order associations between users and items can be captured, and the model can still provide accurate user preference modeling even when the data is sparse. In each layer of graph convolution operation, the system samples a part of the neighbors from the neighbor node set of the target node to generate the context information of the node. The number of sampled neighbors is controlled by the parameter neighbor_sample_size. This sampling strategy balances information integrity and computational efficiency, allowing the model to obtain rich multi-layer semantics without sacrificing computational speed. At the same time, for each layer of GCN, the representation of the target node is generated by aggregating with the sampled neighbor node information. Aggregation operations usually include summation, mean, etc., and can also be combined with embedded weighted summation to further enhance the feature expression ability of the node. Through multi-layer GCN aggregation, the model integrates the direct and indirect associations between users and items layer by layer, and gradually forms an expression of high-order neighbor relationships. At the same time, the new embedding representation of the target node generated by each layer of the graph convolutional network is updated as the input of the next layer. The layer-by-layer updating process makes the embedding representation of users and items not only contain direct interaction information, but also integrate multi-level contextual associations, thereby gradually forming a more complete user interest representation.

[0015] See also Figure 1 a, shows the relationship between a node and its neighbors in a graph neural network (GNN), with the neighbor ranges of the levels (such as h=1 and h=2) marked.

[0016] See also Figure 1 b, shows how node embedding is updated, where information from different neighbors is aggregated.

[0017] 1.2 Self-Attention Mechanism

[0018] In the recommendation system of the attention network architecture, the self-attention mechanism is used to improve the aggregation effect of neighbor information in the graph convolution network layer. Through the self-attention mechanism, the model can assign different weights to different neighbor nodes, highlight the node information that is more important in the user's interest expression, and make the recommendation results more in line with the user's personalized needs. The self-attention mechanism improves the model's ability to capture user interests in detail while maintaining computational efficiency. Specifically, for each layer of graph convolution, the self-attention mechanism calculates an importance weight for each neighbor node of the target node. The calculation of this weight is based on the embedded representation of the target node and the neighbor node. The weight score is usually generated using a dot product or additive mechanism. The higher the weight, the greater the influence of the neighbor node on the target node. After calculating the weights of the neighbor nodes, the Softmax function is used to normalize all weights to ensure that the sum of the weights is 1. The normalization process not only avoids the excessive influence of a certain node on the target node, but also ensures the stability of information aggregation, so that the neighbor information at different levels can be reasonably expressed. Then, according to the weights generated by the self-attention mechanism, the embedded representation of each neighbor node is multiplied by its corresponding weight, and then the aggregated representation of the target node is obtained by weighted summation. In this way, neighbor nodes with higher weights will occupy a larger proportion in the aggregation, so that the neighbor information highly related to the target node is strengthened and expressed. Finally, the neighbor information aggregated by self-attention weighted is combined with the target node's own embedding to generate a new representation of the target node. This update process is carried out layer by layer, so that the node embedding representation gradually accumulates important neighbor information, forming a hierarchical and personalized expression of user interests.

[0019] See also Figure 2 , the left side is the encoder structure of the standard Transformer, and the right side is the decoder structure, which includes modules such as Multi-Head Attention, Addition and Normalization, and specifically shows the processing flow from sequence input to output.

[0020] 1.3 Residual Connection

[0021] Residual connections are used to solve the gradient vanishing problem that may occur in the graph convolutional network layer, further improve the model, and enhance the stability of high-order information transmission. By adding residual connections between graph convolutional layers, it is ensured that multi-layer graph convolution and self-attention mechanisms can effectively retain the original information during the aggregation process, making high-order semantic expressions more accurate, and significantly improving the training efficiency and expression ability of the model. In the embedding update process of each layer of the graph convolutional network, the input embedding representation of the target node is directly added to the output embedding representation to form a residual connection. This jump connection ensures that the output embedding of each layer not only contains the neighbor information aggregated by the current layer, but also retains the node features of the previous layer.

[0022] Through residual connections, the system fuses the previous layer embedding of the node with the updated embedding representation of the current layer, usually in an additive manner. This feature fusion mechanism allows the node representation to contain multi-level neighbor information while retaining the initial features, ensuring the diversity and accuracy of user interest expressions. At the same time, in deep graph convolution structures, residual connections allow gradients to propagate smoothly from deeper layers to shallower layers, effectively alleviating the gradient vanishing problem in deep networks. By enhancing gradient fluidity, residual connections improve the training stability of the model, allowing the model to better learn and express user preferences in a multi-layer structure. It not only improves the model's ability to express complex knowledge graphs, but also significantly enhances the model's training stability and high-order information aggregation effects, providing greater accuracy and robustness for the recommendation system.

[0023] See also Figure 3 ,This figure shows the skip connection design in the classic residual network, which realizes residual learning by directly adding the input to the output of the network, ensuring that the gradient propagation of the deep network is more stable.

[0024] See also Figure 4 The operation process of the graph convolutional network model of the present invention specifically includes the following steps:

[0025] S1. Data input and preprocessing

[0026] The data input mainly includes: users, items, and multi-layer association information in the knowledge graph that represents user preferences and item attributes. By loading user-item interaction data and knowledge graph data, the system can construct input representations of multi-layer semantic relationships for users and items to support subsequent graph convolution and self-attention mechanism aggregation.

[0027] First, the interaction data between users and items is loaded from the user's historical interaction records. These data are divided into training, validation and test sets for model training and evaluation. The user interaction data is loaded, split and saved to memory through files to provide input for subsequent training and prediction of recommendation scores. Then, by loading multi-hop relationship data from the knowledge graph file, the system will build a knowledge graph consisting of users, items and their attributes. In this knowledge graph, nodes represent entities (such as users, items and their related attributes), and edges represent the associations between these nodes (such as "user-item" interaction relationships and "item-attribute" associations). The constructed knowledge graph further forms an adjacency matrix, which provides a basis for subsequent neighbor sampling and graph convolution processing. Finally, by sampling a fixed number of neighbor nodes for each entity in the knowledge graph (based on the configured neighbor_sample_size parameter), the adjacency matrices adj_entity and adj_relation are constructed, and the neighbor nodes and their associations of each entity are stored in memory. This adjacency information simplifies the aggregation process of multi-layer relationships, enabling the model to quickly find and obtain neighbor information. The knowledge graph is then constructed and represented. The knowledge graph construction and representation process includes knowledge graph data loading, adjacency matrix construction, neighbor sampling, and node and relationship embedding initialization. The knowledge graph can provide efficient data structure support for subsequent graph convolution and self-attention mechanism operations.

[0028] Specifically, the system loads the knowledge graph data from the file, maps each entity and its associated relationship into nodes and edges, and forms a structured graph representation. Each node (such as a user, item, and its attributes) represents an entity in the knowledge graph, and each edge represents the association between entities. For example, the "user-item" edge represents the interaction record between the user and the item, while the "item-attribute" edge represents the association between the item and its specific attributes such as category and author. Then, when constructing the knowledge graph, a fixed number of neighbor nodes are sampled for each node. After the knowledge graph is constructed, each node and relationship will be embedded and initialized to generate a low-dimensional vector representation. These embeddings will be continuously updated in the subsequent graph convolution and self-attention mechanisms, and gradually learn more expressive user and item representations, so that the model can still capture the multi-level semantic information of user interests when the data is sparse. On this basis, data embedding is performed. The data embedding process aims to map the entities and relationships in users, items, and knowledge graphs to low-dimensional vector space so that the graph convolution network and self-attention mechanism can efficiently use these embedded representations, thereby capturing multi-level semantic information in subsequent calculations. Data embedding not only provides the model with initial feature representation, but also lays the foundation for multi-layer semantic expression of personalized recommendations.

[0029] The present invention first generates initial embedding representations for user and item nodes. Each user and item embedding is a randomly initialized low-dimensional vector, which will be gradually updated in the graph convolution and attention layers to capture more expressive user preferences and item features. Then, for the attribute nodes of the item, the system also generates low-dimensional embedding representations. These attribute embeddings are used to enrich the multi-dimensional feature expression of the item during the aggregation process, so that the model can take into account the fine-grained features of the item in user preference modeling, and improve the personalization of the recommendation system. Then, the relationship embedding representation is performed, and the relationship types in the knowledge graph are generated into independent embedding vectors during the data embedding process. These relationship embeddings are used to distinguish the semantics of different relationships when the neighbor information is aggregated, so that the model can perform weighted processing on different types of relationships in the graph convolution, thereby accurately capturing the complex user interest structure. Finally, the embedding representations of all nodes and relationships are generated by randomly initializing or based on pre-trained vectors. During model training, these embeddings are continuously updated in each layer of graph convolution and self-attention operations to gradually reflect the characteristics of nodes and relationships, so that user and item embeddings can form accurate interest representations in multi-layer aggregation.

[0030] S2. Dynamic Modeling of User Preferences

[0031] First, high-order neighbor relationships are modeled layer by layer, which is an important step in constructing user interest representation. Through the stacking of graph convolutional networks and self-attention mechanisms, high-order relationships between users and items are gradually captured, so that the model can identify users' potential long-distance interest associations, thereby improving the accuracy of recommendations. In the first layer of graph convolution of the model, the first-order neighbor information of the target node is first aggregated. These first-order neighbors contain the user's direct interaction items or the basic attributes of the items. Through graph convolution and self-attention mechanisms, the model assigns appropriate weights to different neighbor nodes during the aggregation process, thereby highlighting the direct associations that have a greater impact on user preferences. After aggregating the first-order neighbor information, the model gradually expands to high-order neighbors through multiple layers of graph convolution. Each layer of graph convolution and self-attention stacking will further pass on the node representation generated by the previous layer, thereby capturing the second-order, third-order and higher-order neighbor information. This layer-by-layer expansion allows the embedding of users and items to gradually integrate the rich semantic information of high-order neighbors, which helps the model fully understand the multi-level interests of users. Finally, through the self-attention mechanism, different neighbor nodes in each layer of aggregation process will obtain different weights according to the user's interest preferences. In this way, the model can dynamically adjust the semantic importance of different high-order neighbor relationships to generate a more personalized representation of user interests. For example, if a user prefers items in a specific category, the relevant high-order neighbor nodes will receive higher weights to ensure that the model accurately captures this interest preference.

[0032] Finally, the weight of personalized interests is adjusted. In order to express the personalized interests of users more accurately, the model dynamically adjusts the weights of neighbor nodes through the self-attention mechanism, thereby realizing the differentiated expression of personalized interests. First, the system preliminarily generates the user's preference representation based on the user's historical interaction data and the relevant node information in the knowledge graph. The user's preference information comes not only from the directly interacted items, but also from the attributes of the items and their related associated nodes. This information provides the basis for the subsequent self-attention mechanism to calculate the weights. Then, in each layer of graph convolution and self-attention mechanism, the model dynamically calculates the importance of neighbor nodes according to the user's preference characteristics and generates a set of personalized weight assignments. By performing dot product or additive calculations on the embedded representations of the target node and each neighbor node, the system can assign different weights to each neighbor node to highlight the more representative nodes in the user's preferences. In the multi-layer graph convolution structure, the dynamic weights calculated at each layer will be passed layer by layer, so that the user's interest expression can be consistent in the aggregation process at different levels. The dynamic weight adjustment of each layer of the self-attention mechanism enables the user embedding representation finally generated by the model to reflect the user's personalized preferences. Through this personalized adjustment of the weights of neighbor nodes, the model finally generates an embedding representation that reflects the user's multi-level preferences. This user interest representation can highlight the user's key interests in the knowledge graph, making the recommendation results more in line with the user's actual needs.

[0033] S3. Output layer evaluation

[0034] S31. Embedding Aggregation and Representation Generation

[0035] The embedding aggregation and representation generation step is used to integrate the multi-level embedding information of users and items into the final user-item interaction representation. By merging the embedding representations of users, items and their high-order neighbor nodes, a representation that can comprehensively reflect user preferences and item characteristics is generated. First, users and items are aggregated. After the dynamic weight adjustment of multi-layer graph convolution and self-attention mechanism, the embedding representation of user (item) nodes integrates information from different neighbor nodes layer by layer. The system aggregates these multi-layer embeddings to build a representation that represents the user's comprehensive preferences, ensuring that the user embedding can fully reflect its multi-level interest structure in the knowledge graph. Then, after generating the aggregated embeddings of users and items, the system combines the user embedding with the item embedding to generate a user-item interaction representation. This interaction representation can integrate the personalized preferences of users and the multi-dimensional characteristics of items, providing input for subsequent rating prediction. Common combination methods include operations such as inner product, splicing or dot product to generate the user's matching degree to the target item. On this basis, layer-by-layer aggregation is further used to ensure that the high-order association information of users and items in the knowledge graph is also included in the final interaction representation. This fusion of high-order information enables the model to generate accurate expressions of user interests even in cold-start and data-sparse conditions, providing a more accurate recommendation basis for the recommendation system.

[0036] S32. Loss Function and Optimization

[0037] In the recommendation system of the present invention, the loss function and optimization method are used to measure the error between the predicted score and the actual interaction data, and to improve the recommendation accuracy by optimizing the model parameters. The cross entropy loss based on binary classification is used as the loss function, and the parameters of the model are optimized in combination with L2 regularization. Specifically, the loss function includes: basic loss and L2 regularization loss. The basic loss compares the predicted score of the model (unnormalized log probability) with the actual label (0 or 1). First, Sigmoid activation is applied to the predicted score, which is converted into a probability value, and then the cross entropy loss between the predicted value and the actual label is calculated. At the same time, in order to prevent the model from overfitting during training, an L2 regularization term is added to the loss function to impose constraints on the weights of users, entities, relationship embedding matrices, and aggregators. The L2 regularization term is calculated by a function, and the sum of the squares of each parameter is multiplied by the regularization coefficient and added to the basic loss. In this way, L2 regularization limits excessive changes in model parameters, thereby improving the generalization ability of the model. The specific optimization method is as follows:

[0038] The code calculates the error between the model prediction result and the true label through the cross entropy loss function, and introduces the L2 regularization term to prevent the model from overfitting and enhance its generalization ability. This design ensures that the model can accurately fit the data in user preference prediction, while limiting the increase of parameters to reduce the risk of overfitting.

[0039] In the recommendation system of the attention network architecture, the model usually needs to predict the user's preference score for the item. To optimize this task, the code uses binary cross entropy loss, which calculates the error between the model's predicted probability and the true label. Assume that the predicted preference probability is The true label is y, and the binary cross entropy loss function is Loss bce It is defined by the following formula (g):

[0040]

[0041] Where N represents the number of samples, is the preference probability of the i-th sample predicted by the model. This loss function enables the model to accurately predict user preferences by maximizing the probability difference between positive and negative samples.

[0042] In the code implementation, the calculation of cross entropy loss is implemented through 'tf.nn.sigmoid cross entropy withlogits'. This function first performs Sigmoid transformation on the unnormalized score of the model, and then calculates the cross entropy loss, thereby improving numerical stability and avoiding gradient disappearance or explosion.

[0043] At the same time, in order to prevent the model from overfitting, the code introduces the L2 regularization term in the loss function. L2 regularization limits the amplitude of the model parameters to avoid the model from overfitting the training data and enhance its performance on the test set. The L2 regularization term function L2_loss is defined by the following formula (h):

[0044]

[0045] Among them, λ is a hyperparameter of regularization strength, which controls the degree of constraint of L2 regularization on the model. By imposing constraints on the embedding vector, L2 regularization can prevent model instability caused by excessive parameters and improve the generalization ability of the model.

[0046] In the code, L2 regularization is usually implemented by calculating the square sum of each embedding matrix through the 'tf.nn.|2loss' function and multiplying it by the regularization coefficient λ. This regularization term is added to the cross entropy loss to form the total loss function Total_Loss of the model defined by the following formula (i):

[0047] Total_Loss=Loss bce +L2_loss(i).

[0048] By combining cross entropy loss with L2 regularization, the total loss function of the model not only focuses on prediction accuracy, but also constrains model parameters. Cross entropy loss drives the model to distinguish the probability difference between positive and negative samples, while L2 regularization effectively suppresses the overfitting problem caused by excessive parameters.

[0049] In the application part of negative sampling strategy in loss, the code introduces negative sampling strategy to help the model better learn the difference between user preferences and non-preferences. Especially when the data is sparse or the user has not interacted with a large number of items, the negative sampling strategy can improve the training efficiency and prediction effect of the model. In the specific implementation, the negative sampling strategy is used to select negative samples from the set of items that have not been interacted with, so that the model not only focuses on the items that the user is interested in, but also effectively identifies the items that the user is not interested in. Negative sampling is a strategy commonly used in recommendation systems, which aims to randomly select some items from the set of items that the user has not interacted with as negative samples. In this way, the model can learn the difference between positive and negative samples and enhance the ability to recognize user interests. For user u and item i, if user u has never interacted with item i, the item is regarded as a potential negative sample, and the label is y ui = 0. For positive samples, i.e., items j that the user has interacted with, the label is y uj = 1. During the training process, the code randomly selects a certain number of negative samples from the set of items that the user has not interacted with to form positive and negative sample pairs, thus constructing richer training data. This sampling strategy not only avoids the calculation of too much irrelevant information, but also ensures the quality of negative samples.

[0050] In the calculation of cross entropy loss, the negative sampling strategy allows the model to focus on the score difference between positive and negative samples to maximize the scoring probability of positive samples and minimize the scoring probability of negative samples. The binary cross entropy loss function of the model under the negative sampling strategy can be expressed by the following formula (j):

[0051]

[0052] Among them, Loss bce is the binary cross entropy loss function; N and M represent the number of positive and negative samples respectively; Pos is the positive sample set; Neg is the negative sample set; y ui =1 indicates a positive sample label, y uk =0 indicates a negative sample label; and are the predicted probabilities of the model for positive and negative samples, respectively. This loss function maximizes the difference in the model's scores between positive and negative samples, allowing the model to clearly identify the user's true interests. At the same time, in the negative sampling strategy, the code controls the ratio of positive and negative samples by setting the number of negative samples. Usually, each positive sample is set to correspond to a certain number of negative samples to ensure that the model is not overly biased towards positive or negative samples during training. The number of negative samples can be controlled by parameters in the code to balance computational efficiency and model effect.

[0053] In the update process of the Adam optimizer, the code uses the Adam optimizer to update the model parameters to accelerate convergence and enhance the stability of training. The Adam optimizer combines momentum (for smoothing gradients) and adaptive learning rate (for adjusting step size) to enable the model to quickly descend and converge in the initial stage. At the same time, the Adam optimizer corrects the deviation of momentum and root mean square to ensure that the model does not fall into a local minimum in the later stages of training, thereby improving the generalization ability of the model.

[0054] The update of the Adam optimizer must first initialize the first-order moment and second-order moment estimates, that is, for each parameter, Adam maintains two variables, namely the first-order moment (momentum estimate) m and the second-order moment (RMS estimate) v. These variables are initialized to 0, and the sliding average of the historical gradient and the square of the gradient is recorded for the subsequent calculation of the adaptive learning rate. Then, the gradient is calculated, that is, in each round of update, the Adam optimizer first calculates the loss function relative to the current parameter θ t The gradient g t , which is expressed by the following formula (a):

[0055]

[0056] Among them, Loss(θ t ) is the loss function value under the current parameter state; is the gradient, which represents the current loss function for θ t The partial derivative of measures the change of loss under the current parameters.

[0057] Then, the first-order and second-order moment estimates are updated, and the historical gradient and the gradient square are estimated by sliding average. Adam uses two hyperparameters β 1 and β 2 Controlling the decay rate of momentum, the first-order and second-order moment estimates are updated by the following equations (b) to (c):

[0058] m t =β 1 ·m t-1 +(1-β 1 )·g t (b);

[0059]

[0060] Among them, m t is the current momentum estimate, v t is the RMS estimate. Set β 1 = 0.9 and β 2 = 0.999 to ensure that the momentum and gradient variance can be updated stably.

[0061] Finally, to offset the bias caused by the initial values ​​of momentum and RMS estimates being zero, Adam introduces a bias correction term, which is m t and v t The following formulas (d) to (e) are used to correct the unbiased estimate:

[0062]

[0063] in, and It is the corrected momentum and RMS estimate, which improves the stability of initial training.

[0064] After this series of processes, the Adam optimizer updates the parameters through an adaptive learning rate, and its update is defined by the following formula (f):

[0065]

[0066] Among them, α is the learning rate, ∈ is a very small number (such as 10 -8 ), used to prevent division by zero errors. Adam's adaptive learning rate adjustment maintains the stability of training under different gradient scales.

[0067] The Adam optimizer combines momentum (for smoothing gradients) and adaptive learning rate (for adjusting step size) to enable the model to quickly descend and converge in the initial stage. At the same time, the Adam optimizer corrects the deviation of momentum and root mean square to ensure that the model will not fall into a local minimum in the later stages of training, thereby improving the generalization ability of the model.

[0068] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0069] 1) Solve data sparsity and cold start problems

[0070] Traditional recommendation systems, such as collaborative filtering-based methods, rely on historical interaction data between users and items. In situations where data is sparse or new users or items are present, the recommendation effect often drops significantly. This invention uses the rich relationship information of the knowledge graph and combines it with the basic self-attention mechanism to enhance the association representation between users and items. Even when the interaction data is limited, the system can obtain more comprehensive user preference information from the knowledge graph, thereby effectively alleviating the cold start and data sparsity problems.

[0071] 2) Capturing high-level semantic associations to improve recommendation accuracy

[0072] User interests usually span multiple fields and levels, and traditional recommendation systems find it difficult to fully capture the multi-level relationships of users in the knowledge graph. Through the self-attention mechanism, the present invention can aggregate the information of different neighbor nodes on the knowledge graph and distinguish the importance of different nodes, so that high-order semantic associations can be effectively captured. This high-order association information enables the system to consider longer-distance potential interest relationships when making recommendations, thereby improving the accuracy of recommendations.

[0073] 3) Improve the personalized expression of user preferences

[0074] Self-attention can weight the user's interests in different neighbor nodes in the knowledge graph without increasing complexity, highlighting the important information in the user's preferences. The present invention uses the basic self-attention mechanism to dynamically adjust the weight of neighbor information during the aggregation process, thereby further optimizing the recommendation effect in terms of personalized expression. This method not only improves the accurate representation of user interests, but also avoids the complex attention calculation burden, ensuring the efficiency of the recommendation system.

[0075] 4) Improve the computational efficiency of the recommendation system

[0076] Compared with the complex attention mechanism, the self-attention mechanism has lower computational complexity and can efficiently aggregate information in large-scale knowledge graphs. Through the optimized graph convolutional network structure, the present invention reduces the computational overhead of the model while ensuring the recommendation accuracy, so that the model can achieve fast recommendation and real-time update on large-scale knowledge graphs, thus meeting the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the graph convolutional layer structure and aggregation mechanism;

[0078] Figure 2 Schematic diagram of the self-attention module structure;

[0079] Figure 3 Schematic diagram of residual network module;

[0080] Figure 4The following is a pseudo code of the operation process of the graph convolutional network model of the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0082] Example 1

[0083] See also Figure 1 to Figure 4 The present invention adopts a recommendation system with a self-attention mechanism and a residual network architecture. Through a graph convolutional network model constructed layer by layer, a knowledge graph is used to construct a multi-level semantic relationship between users and items. The multi-hop neighbor information between users and items is aggregated through a basic self-attention mechanism to dynamically capture the user's potential interest associations. The multi-level and high-order association information between users and items is aggregated. The self-attention mechanism and residual connection are combined to achieve dynamic personalized weight adjustment and stable information transmission. The network architecture and its optimization method specifically include:

[0084] 1. Data loading and preprocessing

[0085] 1.1 Loading user and item interaction data

[0086] First, use the load_rating function to load the user-item interaction data file into memory. This function supports file formatting and stores the interaction records into a list structure by parsing the user ID, item ID, and interaction behavior (such as rating, click, etc.) of each data. In order to ensure the uniqueness and efficient access of the data, load_rating generates both the user set and the item set to remove duplicate entries. After the data is loaded, the system enters the data partitioning stage and calls the dataset_split function to partition the data into training, validation, and test sets. The code uses random sampling to partition, and controls the size of the validation and test sets according to the set ratio parameters. For example, if the validation and test ratios are set to 0.2, 60% of the data will be used for training, 20% for validation, and 20% for testing. The partitioned data will be stored in lists or arrays for subsequent training and evaluation. In terms of batch processing, we designed the next_batch function to load data according to the set batch size during model training. The batch processing process obtains a set of data fragments by random index to ensure that each batch size is consistent (specified by the parameter batch_size). This optimizes memory usage, reduces I / O operations, and speeds up the model training process.

[0087] 1.2 Knowledge Graph Data Loading and Construction

[0088] In this section, the code loads the knowledge graph file through the load_kg function, representing the entities and relations of the knowledge graph in the form of triples. The load_kg function parses each record in the data file, each record includes a head node (entity 1), a relationship (such as item type), and a tail node (entity 2). The triples are parsed one by one and stored in a nested dictionary structure, with the dictionary key being the head node and the value being a list containing the relationship and the tail node. Subsequently, the system constructs the adjacency matrices adj_entity and adj_relation for efficient access. During the construction process, the code limits the number of neighbor nodes by traversing each entity and its corresponding neighbors. If the number of node neighbors exceeds this set value, a specified number of neighbor nodes are randomly sampled from it to reduce computational complexity and memory requirements. For the generated adjacency matrix, adj_entity and adj_relation store the neighbor node ID and its relationship type ID respectively. In this way, in the graph convolution operation, the model can directly find neighbor information by ID from the adjacency matrix, further accelerating the neighbor information aggregation process. This design ensures that the model can efficiently and quickly access knowledge graph information while utilizing limited neighbor information to improve node representation capabilities, providing rich context for graph convolution and attention mechanism operations.

[0089] 1.3 Data segmentation and batch processing

[0090] In this section, the code first initializes low-dimensional embedding vectors for users, items, and relationships through the init_embedding function. Specifically, the system generates independent embedding vectors for each user and item, which are usually randomly initialized at the beginning. The embedding matrices for users and items are generated through the embedding layer, and the dimensions are set by the embedding dimension parameter in the model configuration file to ensure that each user and item has a unique representation in the embedding space. In terms of relational embeddings in the knowledge graph, the system also creates embedding vectors for each relationship type and stores them through embedding matrices initialized to random values. These relational embeddings help distinguish different semantic relationships in subsequent graph convolution operations. In order to speed up calculations and improve the generalization ability of the model, the code uses regularization parameters in the embedding initialization stage to prevent vector overfitting and excessive parameters. In addition, during data preprocessing, the system stores all embedded data in embedding matrices, which are used in conjunction with the aforementioned adjacency matrices adj_entity and adj_relation to ensure that the model can efficiently obtain the embedded representation of each user, item, and relationship in each layer of the graph convolution and self-attention mechanism.

[0091] 2. Embedding initialization and adjacency matrix construction

[0092] 2.1 Initialization of User, Item, and Relation Embeddings

[0093] In the embedding initialization phase, the code generates low-dimensional vector representations for each user, item, and relationship in the knowledge graph through the embedding layer. These embedding vectors are used for model input and subsequent graph convolution operations. First, the system generates an embedding matrix for each user and item. Assuming the dimension of the embedding is d, the user embedding matrix is The item embedding matrix is Where |U| and |I| represent the total number of users and items, respectively. The initial embedding vector of each user and item can be expressed as follows (k):

[0094]

[0095] Among them, e u 、e u , d are user embedding vector, item embedding vector and dimension respectively.

[0096] This initialization method ensures the dispersion of each vector in the embedding space, making the embedding uniformly distributed in the initial state. Then, the code generates an independent embedding representation for each relationship in the knowledge graph. The relationship embedding matrix can be expressed as Where |R| is the total number of relations, then the initialization of the relation vector is expressed by the following formula (I):

[0097]

[0098] During the embedding layer initialization process, the code usually introduces L2 regularization to avoid overfitting problems caused by excessive embedding vector amplitude. The regularization term loss function L2_loss is defined by the following formula (h):

[0099]

[0100] Among them, λ is a hyperparameter of regularization strength, which controls the constraint strength of L2 regularization on the model.

[0101] 2.2 Constructing the adjacency matrix and neighbor sampling

[0102] In the stage of building the adjacency matrix and neighbor sampling, the code generates the adjacency matrix adj_entity and adj_relation by parsing the triple relationship in the knowledge graph, which is used to store the neighbor node and relationship information of each node. Then, the code constructs the adjacency dictionary through the triple (head entity, relationship, tail entity), where the head entity is the key and the tail entity and the corresponding relationship are the values, which is defined by the following (m) formula:

[0103] kg_dict[v]={(r,u)|(v,r,u)∈Knowledge Graph} (m).

[0104] Among them, kg_dict[v] represents the neighbor information of the graph and the node, expressed in a dictionary; v is a node; u is the neighbor node of v; r is the relationship type.

[0105] This dictionary structure supports fast search and indexing. At the same time, in order to control the amount of calculation, we limit the number of neighbors of each node through neighbor sampling. The set neighbor sampling size is neighbor_sample_size. For neighbor sets that exceed the limit, the code randomly samples neighbor_sample_size neighbors. The adjacency matrix structure is expressed as the following (n) to (o):

[0106]

[0107] adj_relation[v]={r|(v, r, u)∈Knowledge Graph} (o).

[0108] in, Represents the neighbor set of node v. For nodes with insufficient number of neighbors, the system uses padding operations to make up for them to ensure that the matrix dimensions are consistent and facilitate batch processing.

[0109] 3. Model architecture design

[0110] 3.1 Graph Convolutional Layer Design

[0111] The graph convolution layer achieves the capture of high-order semantic information of nodes layer by layer through the combination of neighbor feature aggregation, residual connection and self-attention mechanism. Each layer of graph convolution not only retains the original node information, but also can generate high-quality node embedding representation for recommendation tasks.

[0112] Specifically, the code implements the graph convolution layer through the 'SumAggregatorWithResidualAndAttention' class, combining neighbor feature aggregation, residual connection and attention mechanism, so that the model can capture the high-order semantic information of nodes layer by layer and generate the final node embedding representation. In the 'mixneighborvectors' method, the code first calculates the similarity between the user embedding and the neighbor relationship embedding to generate the attention weight of each node. The specific calculation is as follows:

[0113] user_relation_scores=softmax(mean(user_embeddings×neighbor_relations))

[0114] Among them, user_relation_scores represents the similarity score of user embedding and neighbor relationship embedding. The Softmax function is used to normalize it to ensure that the attention weight of each neighbor is between 0 and 1 and the sum is 1, so that more important neighbors are assigned higher weights when aggregated.

[0115] After obtaining the attention weight, the code multiplies it by the neighbor node feature element by element to achieve the aggregation of weighted neighbor features. The specific calculation formula is:

[0116] neighbors_aggregated=reduce_mean(user_relation_scores_normalized×neighbor_vectors)

[0117] Among them, neighbors_aggregated represents the aggregated representation of the node's neighbor features after weighted summation. Through this aggregation, the model generates an embedded representation of each node containing high-order neighbor information, providing rich context information for subsequent graph convolution layer feature updates. At the same time, in the 'call' method, the code implements a residual connection by adding the aggregated neighbor features 'neighbors-agg' to the current node's own features 'self-vectors', which is expressed by the following (p) formula:

[0118] output=self_vectors+neighbors_agg(p).

[0119] This residual connection design ensures that the original information of the node will not be lost in multi-layer convolution, enhances the gradient fluidity of the model, and effectively prevents the gradient vanishing problem in deep networks.

[0120] In the implementation of the graph convolution layer, the code introduces a self-attention mechanism to assign dynamic weights to each neighbor node. The self-attention mechanism can adjust the neighbor weights according to the similarity with the current node to ensure the personalization and accuracy of feature representation. Specifically, the code obtains the attention weight by calculating the dot product of the user embedding and the neighbor feature, and assigns different weights to different neighbors during the aggregation process.

[0121] 3.2 Integration of Self-Attention Mechanism

[0122] In the integration of the self-attention mechanism, the code provides dynamic weighting capabilities for neighbor feature aggregation in graph convolution by introducing the self-attention mechanism, so that the model can highlight the most relevant neighbor node information in multi-level feature aggregation. This design enables each node to assign different weights according to the importance of its neighbors, thereby generating more accurate and personalized embedding representations.

[0123] The core of the self-attention mechanism is to generate an attention weight based on similarity for each neighbor node. Specifically, the model calculates the attention score of the neighbor node through the dot product of the user embedding and the neighbor relationship embedding, which is calculated by the following (q) formula:

[0124] user_relation_scores=softmax(mean(user_embeddings.neighbor_relations))(q).

[0125] Among them, user_relation_scores represents the similarity score between the user embedding and the neighbor relationship embedding. It is normalized by the Softmax function to ensure that the sum of the attention weights of all neighbors is 1, thereby highlighting the most relevant neighbor information during aggregation.

[0126] After obtaining the attention weights, the code multiplies the attention weights element-by-element with the embedding representations of neighbor nodes by expanding the dimension to achieve weighted summation of neighbor feature aggregation, which is specifically expressed by the following (r) formula:

[0127] neighbors_aggregated=reduce_mean(user_relation_scores_normalized×neighbor_vectors)(r).

[0128] Among them, neighbors_aggregated represents the aggregation result of neighbor features after weighted summation.

[0129] Through weighted aggregation, the model can dynamically adjust the weights of neighbor features according to user preferences to generate more accurate node embedding representations. In the process of multi-layer graph convolution, the self-attention mechanism adjusts the neighbor weights of each node layer by layer, so that the aggregation process can continuously optimize the representation according to the node relationship. This dynamic weighting process provides personalized weight allocation for each layer of convolution, allowing the model to capture the points of interest and feature differences of nodes at different levels.

[0130] By integrating the self-attention mechanism, the model can assign personalized weights to different neighbor nodes in the graph convolution operation, so that each layer of convolution can flexibly capture the most relevant node features. This design not only improves the model's accuracy in capturing user interests, but also ensures that node embedding obtains rich semantic expressions in multi-layer feature aggregation. Ultimately, the attention mechanism enhances the adaptability of the recommendation system in complex graph structures, making the recommendation results more in line with user needs.

[0131] 3.3 Implementation of residual connection

[0132] Residual connections introduce a stable feature preservation mechanism in multi-layer graph convolution, allowing each layer of convolution to obtain high-order feature information of neighbors while retaining the original representation of nodes. This structure enables the model to enhance the richness of feature expression when processing multi-layer convolutions and avoid the common information attenuation problem in deep networks.

[0133] In the implementation of residual connection, the code ensures that the original information of the node is retained during the multi-layer feature aggregation process by introducing residual connection for the graph convolution layer. The residual design prevents the problem of gradual attenuation of information in deep networks and improves gradient flow, enabling the model to effectively train deep networks. The residual connection ensures that the final representation of the node not only contains the neighbor information aggregated by the current layer, but also retains the own characteristics of the previous layer. In the 'call' method of the 'SumAggregatorWithResidualAndAttention' class, the design of the residual connection directly adds the node's own embedding to the aggregation result of the neighbor features to form the final feature representation of the node. Assume that the self-embedding of node v is represented as The embedding after neighbor aggregation is Then the residual connection is expressed by the following formula (s):

[0134]

[0135] in, The output embedding representation of node v at the k+1th layer contains the aggregated information of neighbor features while retaining the own features of node v at the kth layer.

[0136] In addition, the code also ensures that the original information of the node will not be completely replaced in the multi-layer graph convolution by adding its own features to the neighbor aggregation results. The residual connection retains the original representation of the current node in the feature update, so that the node representation can not only receive multi-level neighbor features, but also maintain its own independent semantic information. This weighted feature update design ensures that the model does not lose important original node information in the deep structure. This process is done through 'output =

[0137] selfvectors+neighborsagg'. Among them, 'selfvectors' represents the original embedding vector of the node, and

[0138] 'neighborsagg' is the result of neighbor feature aggregation weighted by the attention mechanism. The direct addition operation combines these two parts of information together to generate the final output embedding of the node.

[0139] 4. Loss Function and Optimization Strategy

[0140] 4.1 Cross Entropy Loss Calculation and Regularization

[0141] In this part, the code calculates the error between the model prediction result and the true label through the cross entropy loss function, and introduces the L2 regularization term to prevent the model from overfitting and enhance its generalization ability. This design ensures that the model can accurately fit the data in user preference prediction, while limiting the increase of parameters to reduce the risk of overfitting.

[0142] In the recommendation system, the model usually needs to predict the user's preference score for the item. To optimize this task, the code uses binary cross entropy loss, which calculates the error between the model's predicted probability and the true label. Assume that the predicted preference probability is The true label is y, and the binary cross entropy loss function is Loss bce It is defined by the following formula (g):

[0143]

[0144] Where N represents the number of samples, is the preference probability of the i-th sample predicted by the model. This loss function enables the model to accurately predict user preferences by maximizing the probability difference between positive and negative samples.

[0145] In the code implementation, the calculation of cross entropy loss is implemented through 'tf.nn.sigmoid cross entropy withogits'. This function first performs a Sigmoid transformation on the unnormalized score of the model, and then calculates the cross entropy loss, thereby improving numerical stability and avoiding gradient vanishing or exploding. At the same time, in order to prevent the model from overfitting, the code introduces the L2 regularization term in the loss function. L2 regularization prevents the model from overfitting the training data by limiting the amplitude of the model parameters and enhances its performance on the test set. In the code, the implementation of L2 regularization is usually implemented through the 'tf.nn.l2loss' function to calculate the square sum of each embedding matrix and multiply it by the regularization coefficient λ. This regularization term is added to the cross entropy loss to form the total loss function Total_Loss of the graph convolutional network model represented by the following formula (i):

[0146] Total_Loss=Loss bce +L2_loss(i).

[0147] By combining cross entropy loss with L2 regularization, the total loss function of the model not only focuses on prediction accuracy, but also constrains model parameters. Cross entropy loss drives the model to distinguish the probability difference between positive and negative samples, while L2 regularization effectively suppresses the overfitting problem caused by excessive parameters.

[0148] 4.2 Application of Negative Sampling Strategy in Loss

[0149] In the application of negative sampling strategy in loss, the code introduces negative sampling strategy to help the model better learn the difference between user preferences and non-preferences. Especially when the data is sparse or the user has not interacted with a large number of items, the negative sampling strategy can improve the training efficiency and prediction effect of the model. In the specific implementation, the negative sampling strategy is used to select negative samples from the set of uninteracted items, so that the model not only focuses on the items that the user is interested in, but also effectively identifies the items that the user is not interested in.

[0150] Negative sampling is a common strategy in recommendation systems that aims to randomly select some items from a set of items that users have not interacted with as negative samples. In this way, the model can learn the difference between positive and negative samples and enhance the ability to identify user interests. For user u and item i, if user u has never interacted with item i, then the item is considered a potential negative sample and its label is y ui = 0. For positive samples, i.e., items j that the user has interacted with, the label is y uj = 1. During the training process, the code randomly selects a certain number of negative samples from the set of items that the user has not interacted with to form positive and negative sample pairs, thus constructing richer training data. This sampling strategy not only avoids the calculation of too much irrelevant information, but also ensures the quality of negative samples.

[0151] In the calculation of cross entropy loss, the negative sampling strategy allows the model to focus on the score difference between positive and negative samples to maximize the scoring probability of positive samples and minimize the scoring probability of negative samples. The binary cross entropy loss function of the model under the negative sampling strategy can be expressed by the following formula (j):

[0152]

[0153] Among them, N and M represent the number of positive and negative samples respectively; Pos is the positive sample set; Neg is the negative sample set; y ui =1 indicates a positive sample label, y uk =0 indicates a negative sample label; and are the predicted probabilities of the model for positive and negative samples, respectively. This loss function maximizes the difference in the model's scores between positive and negative samples, allowing the model to clearly identify the user's true interests. At the same time, in the negative sampling strategy, the code controls the ratio of positive and negative samples by setting the number of negative samples. Usually, each positive sample is set to correspond to a certain number of negative samples to ensure that the model is not overly biased towards positive or negative samples during training. The number of negative samples can be controlled by parameters in the code to balance computational efficiency and model effect.

[0154] 4.3 Update process of Adam optimizer

[0155] In the update process of the Adam optimizer, the code uses the Adam optimizer to update the model parameters to accelerate convergence and enhance the stability of training. The Adam optimizer combines momentum (for smoothing gradients) and adaptive learning rate (for adjusting step size) to enable the model to quickly descend and converge in the initial stage. At the same time, the Adam optimizer corrects the deviation of momentum and root mean square to ensure that the model does not fall into a local minimum in the later stages of training, thereby improving the generalization ability of the model.

[0156] The update of the Adam optimizer must first initialize the first-order moment and second-order moment estimates, that is, for each parameter θ, Adam maintains two variables, namely the first-order moment (momentum estimate) m and the second-order moment (RMS estimate) v. These variables are initialized to 0, and the sliding average of the historical gradient and the square of the gradient is recorded for the subsequent calculation of the adaptive learning rate. Then the gradient is calculated, that is, in each round of update, the Adam optimizer first calculates the loss function relative to the current parameter θ t The gradient g t , which is expressed by the following formula (a):

[0157]

[0158] Among them, Loss(θ t ) is the loss function value under the current parameter state; is the gradient, which represents the current loss function for θ t The partial derivative of measures the change of loss under the current parameters.

[0159] Then, the first-order and second-order moment estimates are updated by the following equations (b) to (c):

[0160] m t =β 1 ·m t-1 +(1-β 1 )·g t (b);

[0161]

[0162] Adam uses two hyperparameters β to estimate the historical gradient and the square of the gradient by sliding average. 1 and β 2 Controls the decay rate of momentum. Among them, m t is the current momentum estimate, v t is the RMS estimate. Set β 1 = 0.9 and β 2= 0.999 to ensure that momentum and gradient variance can be updated stably. Finally, in order to offset the deviation caused by the initial value of momentum and RMS estimation being zero, Adam introduces a bias correction term, which is m t and v t The following formulas (d) to (e) are used to correct the unbiased estimate:

[0163]

[0164] in, and It is the corrected momentum and RMS estimate, which improves the stability of initial training.

[0165] After this series of processes, the Adam optimizer updates the parameters through the adaptive learning rate using the following formula (f):

[0166]

[0167] Among them, α is the learning rate, ∈ is a very small number (such as 10 -8 ) to prevent division by zero errors.

[0168] Adam's adaptive learning rate adjustment enables the stability of training under different gradient scales. The Adam optimizer combines momentum (for smoothing gradients) and adaptive learning rate (for adjusting step size) to enable the model to quickly descend and converge in the initial stage. At the same time, the Adam optimizer ensures that the model will not fall into a local minimum in the later stages of training by correcting the deviation of momentum and root mean square, thereby improving the generalization ability of the model.

[0169] 5. Training Process

[0170] 5.1 Batch Training

[0171] To improve training efficiency, the code uses batch processing, that is, processing a group of samples at the same time in each round of training. Batch training significantly speeds up model training by reducing frequent gradient update operations and stabilizes the model's gradient when the data is large. The ratio of positive and negative samples in each batch remains consistent, ensuring that the model can maintain a balance between positive and negative samples in each round of updates, thereby effectively avoiding the bias caused by sample imbalance.

[0172] The process of batch training is as follows:

[0173] 1) Sample batch division: The training data is divided into multiple small batches according to the set batch size batch_size. Each batch contains multiple user-item positive and negative sample pairs, so that the model can process multiple samples in parallel in each round of training, thereby reducing the frequency of gradient updates and improving training efficiency.

[0174] 2) Batch data loading: Before each round of training, the code will load a batch of data through the 'next batch' function, which randomly extracts positive and negative sample pairs of users and items from the preprocessed data according to the batch size, and packages these samples into a training batch for subsequent batch training. By loading data in batches, the system can maximize the use of hardware resources during training.

[0175] 3) Gradient calculation and update: In batch training, the code calculates the loss function of the batch data (such as cross entropy loss) and calculates the gradient through back propagation. All samples in the batch will share the same gradient update, thereby accumulating more information in one parameter update and improving the convergence speed of the model.

[0176] In each batch, the loss function is calculated based on the average of all positive and negative sample pairs. Let the batch size be B, then the batch loss function Loss batch It can be expressed by the following formula (t):

[0177]

[0178] Where: y i Represents the true label of the sample; is the predicted probability of the model.

[0179] The batch loss is back-propagated and parameter updated through the Adam optimizer, so that the model can use the information of all batch samples to optimize parameters in each round of training.

[0180] 5.2 Back Propagation and Gradient Update

[0181] In this part, the code calculates the gradient of the loss function to the model parameters through the back-propagation algorithm, and uses this gradient information to update the model parameters, thereby gradually minimizing the loss function. Back-propagation and gradient update are the core steps of model training, ensuring that the model can continuously optimize the parameters so that it can show higher prediction accuracy on positive and negative samples. Among them, the back-propagation algorithm is used to calculate the partial derivative (i.e., gradient) of the loss function with respect to each parameter. After the model generates the predicted probability of the user for the item, the code obtains the error between the predicted result and the true label by calculating the cross-entropy loss function. The back-propagation algorithm starts from the output layer and passes the loss gradient back to the input layer layer by layer to calculate the gradient of all parameters. In this process, forward propagation must be performed first, that is, the predicted preference probability of each user-item pair is calculated. Then, the total loss value of the model Loss(θ) is calculated by the cross entropy loss function, where θ is the set of all parameters of the model. Then the output layer gradient is calculated. The code will calculate the gradient of the loss to the output probability according to the cross entropy loss function. Assume that the predicted probability is The true label is y, then the gradient of the cross entropy loss to the predicted probability is expressed by the following formula (u):

[0182]

[0183] Then, the code calculates the gradients of the hidden layer and input layer parameters layer by layer using the chain rule. For the weight matrix W and bias b in each layer, the gradients calculated by the code are expressed by the following formulas (v) to (w):

[0184]

[0185] Among them, h represents the activation output of this layer.

[0186] By calculating these partial derivatives layer by layer, the back propagation algorithm can obtain the gradient information of each parameter under the current loss state. Finally, after obtaining the gradients of all parameters, the code updates each parameter through an optimizer (such as Adam). Suppose the target gradient of a parameter is Then the parameter update is expressed by the following formula (x):

[0187]

[0188] Where: θ t+1 is the updated parameter value; α is the learning rate, which is used to control the gradient step size.

[0189] 6. Evaluation Indicators

[0190] AUC and F1-Score are commonly used binary classification indicators in recommendation systems, which measure the model's ranking ability and the comprehensive performance of positive and negative sample classification respectively. This system uses these two as evaluation indicators, where AUC represents the score of the model's ability to distinguish between positive and negative samples, and the overall ranking ability of the model is evaluated by calculating the area under the ROC curve. In the code, the true positive rate (TPR) and false positive rate (FPR) are calculated according to different thresholds, and then the ROC curve is integrated. The code obtains the AUC value by the following (y) formula:

[0191]

[0192] In the code implementation, A∪C is calculated by the 'roc aucscore' function in an external library (such as 'sklearn.metrics'). A higher AUC indicates that the model has a strong ability to distinguish between positive and negative samples.

[0193] The F1-Score is the harmonic mean of precision and recall, and is used to measure the model's ability to identify and cover positive samples. Precision represents the proportion of samples that are actually positive among all samples predicted by the model to be positive samples, which is expressed by the following formula (z):

[0194]

[0195] The recall rate indicates the proportion of samples that are successfully predicted as positive samples by the model among all samples that are actually positive samples. It is expressed by the following formulas (I) to (II):

[0196]

[0197] F1-Score provides an overall evaluation of the model's ability to identify positive samples by balancing precision and recall. The code calculates this metric through the 'flscore' function to ensure that the model accurately captures user interests.

[0198] The present invention not only effectively reduces the computational complexity of the recommendation system on large-scale knowledge graphs, but also enhances the diversity and interpretability of the recommendation results. Experimental results show that compared with traditional methods, the present invention has significantly improved recommendation accuracy, personalized expression and computational efficiency. The above is only a further explanation of the present invention and is not intended to limit this patent. All equivalent implementations of the present invention should be included in the scope of the claims of this patent.

Claims

1. An attention network architecture for a recommendation system and an optimization method thereof, characterized in that a recommendation system using a self-attention mechanism and a residual network architecture is used, a graph convolutional network model is constructed layer by layer, a multi-level semantic relationship between users and items is constructed using a knowledge graph, and multi-hop neighbor information between users and items is aggregated through a basic self-attention mechanism to dynamically capture the user's potential interest associations and aggregate multi-level and high-order association information between users and items. The dynamic personalized weight adjustment and stable information transmission are realized by combining the self-attention mechanism and the residual connection. The network architecture and the optimization method thereof specifically include:

1. Architecture design of graph convolutional network model 1.1: Use the graph convolutional network layer to aggregate the neighbor information of users, items and their related attributes in the knowledge graph layer by layer to build a multi-level representation of users and items. In each layer of graph convolution operation, the recommendation system samples a part of the neighbors from the neighbor node set of the target node to generate the context information of the node. The number of sampled neighbors is controlled by the parameter neighbor_sample_size; 1.2: For each layer of the graph convolutional network, the representation of the target node is aggregated by summing, averaging, or weighted summing combined with embedding with the sampled neighbor node information; 1.3: Through multi-layer graph convolutional networks, the direct and indirect associations between users and items are integrated layer by layer to form an expression of high-order neighbor relationships. At the same time, the new embedding representation of the target node generated by each layer of graph convolutional networks will be updated as the input of the next layer. The layer-by-layer updating process makes the embedding representation of users and items not only contain direct interaction information, but also integrate multi-level contextual associations to form a more complete user interest representation; 2. Self-Attention Mechanism 2.1: For each layer of graph convolution, the self-attention mechanism generates weights for each neighbor node of the target node using a dot product or additive mechanism; 2.2: Use the Softmax function to normalize all weights to ensure that the sum of the weights is 1; 2.3: According to the weights generated by the self-attention mechanism, the embedding representation of each neighbor node is multiplied by its corresponding weight, and the aggregated representation of the target node is obtained by weighted summation, that is, the neighbor information of self-attention weighted aggregation; 2.4: Combine the self-attention weighted aggregated neighbor information with the target node’s own embedding to generate a new representation of the target node; (III) Residual Connection 3.1: In the embedding update process of each layer of the graph convolutional network, the previous layer embedding of the node is fused with the updated embedding representation of the current layer by addition to form a residual connection; The recommendation system adopts the attention network architecture and its optimized implementation, which specifically includes the following steps: S1. Data input and preprocessing S1.1: Load the interaction data between the user and the item from the user's historical interaction records, load it through a file, split it and save it to the memory; the file loading uses the load_rating function to load the user and item interaction data file into the memory, parse the user ID, item ID and interaction behavior of each data, and store the interaction record in a list structure; the interaction data is divided into training, validation and test sets for model training and evaluation in a ratio of 6:2:2; S1.2: By loading multi-hop relationship data from the knowledge graph file, a knowledge graph consisting of users, items and their attributes is constructed. The knowledge graph is loaded using the load_kg function, and the entities and relationships of the knowledge graph are represented in the form of triples; the nodes in the knowledge graph represent entities, i.e., users, items and their related attributes, and the edges represent the associations between these nodes, i.e., the "user-item" interaction relationship and the "item-attribute" association; S1.3: Sample a fixed number of neighbor nodes for each entity in the knowledge graph, construct adjacency matrices adj_entity and adj_relation, and store the neighbor nodes of each entity and their associations in memory; S1.4: Use the init_embedding function to randomly generate low-dimensional embedding representations for user and item nodes, as well as low-dimensional embedding representations for item attribute nodes; S1.4: Generate independent embedding vectors for the relationship types in the knowledge graph during the data embedding process, and generate embedding representations of all nodes and relationships through random initialization or based on pre-trained vectors; S2. Dynamic Modeling of User Preferences S2.1: In the first layer of graph convolution in the graph convolutional network model, the first-order neighbor information of the target node is aggregated first. After aggregating the first-order neighbor information, it is gradually expanded to the layer-by-layer modeling of high-order neighbor relationships through multi-layer graph convolution. Each layer of graph convolution and self-attention stacking will further pass the node representation generated by the previous layer, thereby capturing the second-order, third-order and higher-order neighbor information; S2.2: Through the self-attention mechanism, different neighbor nodes in each layer of aggregation process will obtain different weights according to the user's interest preferences, so that the graph convolutional network model can dynamically adjust the semantic importance of different high-order neighbor relationships, thereby generating a more personalized user interest representation; S2.3: The recommendation system initially generates a user preference representation based on the user's historical interaction data and the relevant node information in the knowledge graph. Then, in each layer of graph convolution and self-attention mechanism, the graph convolution network model dynamically calculates the importance of neighbor nodes based on the user's preference characteristics and generates a set of personalized weight assignments. S2.4: Perform dot product or additive calculations on the embedding representations of the target node and each neighbor node. The recommendation system assigns different weights to each neighbor node to highlight the more representative nodes in the user's preferences. In the multi-layer graph convolution structure, the dynamic weights calculated at each layer are passed layer by layer, so that the user's interest expression can be consistent in the aggregation process at different levels. Through this personalized adjustment of the weights of neighbor nodes, the graph convolution network model finally generates an embedding representation that reflects the user's multi-level preferences. S3. Output layer evaluation S3.1: Embedding Aggregation and Representation Generation Merge the embedding representations of users, items, and their high-order neighbor nodes to generate aggregate embeddings of users and items, and combine them with item embeddings to generate user-item interaction representations, wherein the combination includes: inner product, concatenation, or dot product operations; S3.2 Loss Function and Optimization The parameters of the graph convolutional network model are optimized using the cross entropy loss based on binary classification as the loss function, wherein the loss function calculates the error between the predicted probability of the graph convolutional network model and the true label, that is, the predicted score of the unnormalized log probability compared with the actual label of 0 or 1; the predicted score is activated by sigmoid to convert it into a probability value, and then the cross entropy loss between the predicted value and the actual label is calculated; the L2 regularization term is added to the loss function to impose constraints on the weights of users, entities, relationship embedding matrices, and aggregators; the L2 regularization term is calculated by a function, and the sum of the squares of each parameter is multiplied by the regularization coefficient and added to the basic loss; S3.3: Use Adam optimizer to update graph convolutional network model parameters S3.3.1: For each parameter , Adam maintains two variables, namely the first-order moment and the second moment , initialize the first-order moment and the second-order moment to 0, and record the sliding average of the historical gradient and the gradient square; S3.3.2: In each update round, the Adam optimizer calculates the loss function relative to the current parameters using the following formula (a): Gradient : (a); in, is the loss function value under the current parameter state; is the gradient; S3.3.3: To estimate the historical gradient and the squared gradient by sliding average, Adam uses two hyperparameters and Control the decay rate of momentum and update the first-order and second-order moment estimates using the following equations (b) to (c): （b）； （c）； in, is the current momentum estimate; is the root mean square estimate; S3.3.4: and Corrected to an unbiased estimate by (d) to (e) below: （d）; （e）; in, and are the corrected momentum and RMS estimates, respectively; is the t-th power of the first-order momentum decay rate; is the t-th power of the second-order momentum decay rate; S3.3.5: Update the parameters using the following formula (f) through adaptive learning rate: （f）； in, is the learning rate; To prevent division by zero errors for very small numbers.

2. The attention network architecture method for recommendation system according to claim 1, characterized in that: The binary cross entropy loss of step S32 It is defined by the following formula (g): （g）； in, is the sample size; The model predicts The preference probability of samples; is the predicted preference probability; is the true label; The regularization term loss function It is defined by the following formula (h): （h）； in, is a hyperparameter of regularization strength, which controls the degree of constraint of L2 regularization on the model; , , are the embedding vectors of users, items, and relations, respectively; I and U are the total number of users and items, respectively; The basic loss addition is to add the regularization term to the cross entropy loss to form the total loss function of the graph convolutional network model represented by the following formula (i): : （i）。 3. The attention network architecture method for recommendation system according to claim 2, characterized in that: The binary cross entropy loss function of the graph convolutional network model in the cross entropy loss calculation is expressed by the following formula (j) under the negative sampling strategy: （j）； in, and are the number of positive and negative samples respectively; is the positive sample set; is the negative sample set; is the positive sample label, is the negative sample label; and are the predicted probabilities of the model for positive and negative samples, respectively.

Images