A dynamic weighting-based non-sampling knowledge graph embedding training method

Abstract

The application belongs to the technical field of digitization equipment, and discloses a non-sampling knowledge graph embedding training method based on dynamic weighting, which comprises the following steps: constructing a knowledge graph embedding model based on a translation distance model based on historical maintenance records, sensor data and expert knowledge of equipment; in the knowledge graph embedding model based on the translation distance model, the performance weakening problem of the model caused by the loss function based on square is relieved through a non-sampling loss function based on a probability prediction function; the loss function of negative triplets is dynamically weighted by using a weight factor, so as to reduce the adverse effects of easy-to-split negative triplets on model training; the training set and the test set are divided, the knowledge graph embedding model is trained, the model parameters are optimized, and potential problems of equipment are found. The application reduces the adverse effects of easy-to-split negative triplets on model training, thereby improving the precision of model training.

Description

Technical Field

[0001] This invention belongs to the field of digital equipment technology, and in particular relates to a non-sampling knowledge graph embedding training method based on dynamic weighting. Background Technology

[0002] Currently, knowledge graphs (KGs), as an important carrier for storing and processing structured data, and as an advanced knowledge interconnection solution, provide a wide range of intelligent system power for many applications, thus attracting widespread attention in academia and industry. Knowledge graph embedding (KGE) embeds entities and relations in the KG into a continuous vector space, which can preserve the inherent structure of the knowledge graph while facilitating operations such as entity search and matching, thereby making the large-scale application of knowledge graphs a reality. Therefore, in recent years, knowledge graph embedding has occupied a dominant position in the vast field of structured knowledge interconnection, and its effectiveness has been demonstrated in equipment digitization scenarios. For example, in equipment fault prediction, fault prediction models are established based on historical maintenance records, sensor data, and expert knowledge to identify potential problems in advance and avoid unexpected downtime.

[0003] In recent years, researchers have proposed various KGE models, among which classic models include DistMult, SimpleE, ComplEx, TransE, and RotatE. Figure 1 As shown, to better extract useful information from the dataset and obtain better knowledge embeddings, a common training method for KGE is to simultaneously provide the model with positive and negative samples, i.e., positive triples and negative triples, and then optimize the training by calculating the loss functions of the positive and negative triples based on the constructed objective function, thereby obtaining the embedding vector representation of each element of the knowledge graph. The dataset typically only contains positive samples (all triples in the dataset can be considered positive samples for the model), while negative samples are constructed based on the positive samples using certain methods. Therefore, the training of most models relies on negative sampling, which is the process of constructing and selecting negative samples relative to the positive samples based on a certain strategy. However, while training based on negative sampling improves training efficiency, it also presents some challenges. First, sampling only a portion of the negative triples for training can reduce the accuracy of model training. Second, the different negative triples obtained from sampling in different training rounds can affect the stability of model training.

[0004] Inspired by recent research in non-sampling recommendation systems, some researchers have attempted to apply non-sampling methods to KGE training. They have proposed a non-sampling training method that treats all non-positive triples composed of entities and relations in the knowledge graph as negative triples for training, thus addressing the issues of training accuracy and stability inherent in negative sampling-based training methods by skipping the negative sampling process. However, this training method, employing a squared-based loss function, weakens the training performance of the original knowledge graph embedding model. Furthermore, the large number of easily separable negative triples far from the model's decision boundary are treated the same as other negative triples, such as hard-to-separate negative triples, in training (this application refers to negative triples that are difficult to distinguish as positive or negative samples as hard-to-separate negative triples; easily separable negative triples are the opposite of hard-to-separate negative samples). This makes it difficult for the model to learn useful gradient information from the training samples, hindering proper parameter updates and thus affecting the model's convergence speed and training accuracy.

[0005] ComplEx explored the impact of the number of negative samples for each positive sample on model performance on the FB15K dataset. For example... Figure 2 As shown, the model's performance improves with increasing negative sample count. However, once the number of negative samples reaches a certain threshold, the model's performance actually decreases with further increases in the number of negative samples. This experimental result indicates that sampling more negative samples for training usually yields better results. However, once the number of negative samples reaches a certain level, excessive negative samples participating in training can actually reduce the effectiveness of model training due to a large number of easily separable negative samples.

[0006] Existing KGE models can be broadly classified into three categories:

[0007] Geometric transformation-based methods: These methods model semantic relationships between entities and relations through geometric transformations. Among them, the most classic TransE model views a relationship as a translation transformation between the head entity vector and the tail entity vector in the same space. Models such as TransH, TransR, TransD, and RotatE are extensions of TransE and can handle more complex relationships.

[0008] Tensor decomposition-based methods: These models construct a score function through tensor decomposition to measure the probability of a triple being true. Among them, the widely used DistMult model the score function as the inner product of two vectors representing the head and tail entities and the diagonal matrix of the relation. Subsequent models such as ComplEx, HolE, ANALOGY, and SimplE are improvements and extensions of the DistMult model, aiming to achieve better embedding performance.

[0009] Neural network-based methods: These models utilize the nonlinear transformation characteristics of deep neural networks such as CNNs, RNNs, and GNNs to model complex relationships between entities and relations. Specifically, ConvE and ConvKB models use CNNs to model the latent semantics between entities and relations; Gradner et al. use RNNs to model relation paths to capture longer relation dependencies in the KG; and R-GCN, RGHAT, and CompGCN utilize GNNs to capture structured information in the KG.

[0010] Negative sampling methods in knowledge embedding models can be broadly categorized into three types: random negative sampling, adversarial negative sampling, and additional data-enhanced sampling (ADS). These will be discussed in detail below. Random negative sampling is the most basic and widely used method. TransE randomly selects an entity from the entity set and replaces it in a positive triple to generate a negative sample. To achieve better training results, TransH proposed an improved strategy: for 1-N relation triples, the head entity is given a higher replacement probability; correspondingly, for N-1 relation triples, the tail entity is given a higher replacement probability. Subsequent models such as TransR and TransD have adopted this sampling strategy. In summary, random negative sampling often results in overly simple negative samples, i.e., easily simplistic negative samples, which fail to provide useful information for model optimization and training, leading to the model getting trapped in local optima.

[0011] Adversarial negative sampling, based on the principles of GANs, utilizes the semantic information of samples in the embedding space to select negative samples, aiming to sample better, harder-to-classify negative samples. However, it suffers from drawbacks such as complex frameworks, unstable training results, and long training times. To address these issues, RotatE proposes a self-adversarial negative sampling strategy that dynamically adjusts the sampling probability of negative samples during model training to sample harder-to-classify negative samples for training.

[0012] Negative sampling methods that introduce additional information introduce external information during the negative sampling process to increase the sampling probability of hard-to-distinguish negative samples.

[0013] Regardless of the negative sampling method used, training based on negative sampling only utilizes a portion of the negative sample information in the dataset for training. This inevitably wastes the training information of unsampled negative samples. Furthermore, since the negative samples sampled in different rounds are different, it also has a certain impact on the stability of model training.

[0014] Non-sampling knowledge graph embedding models: To overcome the high training complexity and prediction bias caused by the over-reliance on negative sample selection in negative sampling, a regularization term is added to the loss function to prevent the model from generating high scores for all triples, thus eliminating the need for negative sampling. Although this method does not explicitly use negative sampling, the calculation of the regularization term implicitly uses all possible triples in the knowledge graph, including positive and negative triples. KG-NSF proposes that it does not require negative samples to participate in training, and directly performs KGE learning based on the cross-correlation matrix of the embedding vectors of entities and relations in the knowledge graph. However, due to the need to calculate multiplication between large matrices, this method has high space complexity. The non-sampling knowledge graph embedding framework NS-KGE treats all non-positive triples composed of entities and relations as negative triples for training. This method solves the shortcomings of negative sampling training by skipping negative sampling. However, this method uses a squared-based loss function, which weakens the training performance of the original KGE model, and treating all negative triples equally in training will lead to a decrease in training effect. Summary of the Invention

[0015] To address this, this application proposes a dynamically weighted, scalable non-sampled knowledge graph embedding framework. First, a non-sampled training method is employed, treating all non-positive triples composed of entities and relations in the knowledge graph as negative triples for training, thus mitigating the reduced training accuracy and stability caused by negative sampling-based training. Second, the least-squares-based non-sampled loss function is improved by interpreting the predicted value based on the scoring function as the probability value of a triple being predicted as a positive triple, simplifying the definition of a new non-sampled loss function and improving model training performance. Finally, both Focal-Loss weighting factors and self-adversarial negative sampling weighting factors are used to dynamically weight the loss function of negative triples, reducing the impact of easily separable negative triples on model training. To evaluate the performance of the proposed dynamically weighted non-sampled KGE model training framework, experiments were conducted on five KGE models: DistMult, SimpleE, ComplEx, TransE, and RotatE, validating the effectiveness of the proposed framework.

[0016] To achieve the above objectives, this application discloses a dynamically weighted, non-sampled knowledge graph embedding training method, which includes the following steps:

[0017] A knowledge graph embedding model based on a translation distance model is constructed based on the equipment's historical maintenance records, sensor data, and expert knowledge.

[0018] In the knowledge graph embedding model based on the translation distance model, the performance degradation problem caused by the squared loss function is alleviated by the non-sampling loss function based on the probability prediction function.

[0019] A weighting factor is used to dynamically weight the loss function of negative triples to reduce the adverse effects of easily separable negative triples on model training.

[0020] By dividing the dataset into training and testing sets, the knowledge graph embedding model is trained, the model parameters are optimized, and potential equipment problems are discovered.

[0021] Preferably, the knowledge graph embedding model based on the translation distance model includes at least one of the TransE and RotatE models.

[0022] Preferably, for the knowledge graph embedding model G based on the translation distance model, for the triple (h, r, t), where h, t ∈ E, E is the entity set, h and t are the head and tail entities in the triple, r ∈ R, R is the relation set, r is the relation, and f r (h,t) represents the true label value of the triple (h,r,t), where, for a positive triple (h,r,t), f r (h,t) = 1 indicates that the head entity h and the tail entity t can be connected through the relation r, while for the negative triple (h,r,t), f r (h,t)=0 indicates that the head entity h and the tail entity t cannot be connected by relation r; The predicted function value is represented by the score function of the knowledge graph embedding model, which aims to determine whether the head entity h and the tail entity t can be connected by the relation r. The training framework of the model aims to learn the vector representations h, r, t of entities h, t and relation r in the knowledge graph embedding model by minimizing the difference between the true label values ​​of positive and negative triples and the predicted values ​​based on the score function.

[0023] Preferably, the The probability prediction function P(r|h,t) is used to predict the triple (h,r,t) as a positive triple; for the triple (h,r,t), its probability prediction function P(r|h,t) is defined as:

[0024] P(r|h,t)=sigmoid(δ r -F r (h,t))

[0025] Where, δ r >0 indicates a threshold hyperparameter F related to relation r. r (h,t) is the score function of the triple (h,r,t); if the score function F of (h,r,t) is... rThe value of (h,t) is greater than the threshold δ r If the value of P is close to 0, that is, if the probability of (h,r,t) being predicted as a positive triplet is close to 0, then it is predicted as a negative triplet; conversely, if the value of the prediction probability function is close to 1, then it is predicted as a positive triplet.

[0026] The training framework uses a least-squares-based loss function to define its loss function for optimizing training, i.e.

[0027]

[0028] C hrt The weights of the corresponding triples (h, r, t) represent the importance of the triples in model training.

[0029] Expanding the above equation with respect to both positive and negative sample sets, we get:

[0030]

[0031] Among them, C + C - S represents the weights of the positive and negative triples. + It is a set of positive triples, S - It is a set of negative triples;

[0032] For a positive triple, its true value f r (h,t)=1, for a negative triple, its true value f r (h,t)=0,

[0033] Substituting into the above equation, we get:

[0034]

[0035] Preferably, since the probability prediction function P(r|h,t) of the triplet takes values ​​in the range of [0,1], therefore, P(r|h,t)≥0, 1-P(r|h,t)≥0;

[0036] The loss function is simplified to the following definition:

[0037]

[0038] Preferably, in the loss function, a model-based weighting factor is added to all negative triples to dynamically adjust the impact of different negative triples on model training during the training process; the negative triple weighting factor includes a Focal-Loss weighting factor and a self-adv weighting factor.

[0039] Preferably, the negative triplet Focal-Loss weighting factor for:

[0040]

[0041] Where β1,γ>0 are hyperparameters, β1 is used to balance the weights between positive and negative samples, and P F P represents the probability that a negative triplet is misclassified, i.e., the probability that a negative triplet is misclassified as a positive triplet; for a negative triplet (h,r,t), P F (r|h,t) is defined as:

[0042] P F (r|h,t)=sigmoid(δ r -F r (h,t))

[0043] Where, δ r >0 is the threshold hyperparameter, F r (h,t)>0 is the score function for the triple (h,r,t);

[0044] The role of the Focal-Loss weighting factor is as follows: During model training, for easily separable negative triples, its P... F The closer it is to 0, the higher its weighting factor C. - The closer it is to 0, the more important it is for indistinguishable negative triples, and its P... F The larger the value of C is than 0, the better. - The value is also greater than 0; therefore, the model dynamically adjusts the weights of easily separable and hard-to-separate negative triples based on the predicted probability. By scaling the weight values ​​to a greater extent, the weight of easily separable negative triples is reduced, so that the model pays less attention to this part of the samples, thereby reducing the impact of easily separable negative triples on model training; for hard-to-separate negative triples, their weights are less than the scaling degree of easily separable negative triples, so that the model training focuses on hard-to-separate negative triples, thereby improving the model training effect.

[0045] Combining the negative triplet Focal-Loss weighting factor, the final loss function is obtained as follows:

[0046]

[0047] Where β1,γ>0 are hyperparameters.

[0048] Preferably, the self-adv weighting factor for:

[0049]

[0050] Where β2, α>0 are hyperparameters, β2 is used to balance the weights between positive and negative samples; the self-adv weight factor C - Its function is: the negative triplet (h) after function transformationj ',r,t j The weight of the score function value of ') out of the sum of the score function values ​​of all negative triplet functions with relation r after transformation;

[0051] Combining the self-adv weighting factor, the final loss function is as follows:

[0052]

[0053] Where β2,α>0 are hyperparameters.

[0054] Preferably, for TransE, the scoring function F of the triples r The definition of is:

[0055] F r =||h+rt||2

[0056] The probability prediction function P(r|h,t) based on TransE is defined as:

[0057] P(r|h,t)=sigmoid(δ r -||h+rt||2)

[0058] For RotatE, the scoring function F of the triples r The definition of is:

[0059]

[0060] The probability prediction function P(r|h,t) based on RotatE is defined as follows:

[0061]

[0062] Calculate the model's score function F r After obtaining the probability prediction function P(r|h,t), we substitute it into the loss function to obtain the final loss function of the training framework. Then, we perform optimization training to learn the optimal embeddings h, r, t for h, r, t.

[0063] The beneficial technical effects of this application include:

[0064] First, a novel non-sampled knowledge graph embedding framework is proposed, which alleviates the performance degradation problem of the squared loss function by simplifying the definition of the non-sampled loss function based on the probability prediction function.

[0065] Secondly, two weighting factors are used to dynamically weight the loss function of negative triples to reduce the adverse effects of easily separable negative triples on model training.

[0066] Furthermore, the proposed framework is scalable and can be applied to almost all negative sampling-based KGE models. Using several KGE models as examples, the application of this training framework to existing KGE models is demonstrated. Attached Figure Description

[0067] Figure 1 A schematic diagram of a traditional training method based on negative sampling;

[0068] Figure 2 Experimental results showing the impact of negative sampling quantity on model training;

[0069] Figure 3 A flowchart of the present invention. Detailed Implementation

[0070] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0071] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0072] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0073] The technical solutions provided in this application involve technologies such as machine learning and natural language processing in artificial intelligence, which are specifically described and illustrated through the following embodiments.

[0074] A formal description is provided for the dynamic weighted, extensible, non-sampling knowledge graph embedding framework proposed in this application.

[0075] Table 1 Basic Symbols in this Application

[0076]

[0077] Table 1 lists the basic notation used. Given a knowledge graph G, for a triple (h, r, t), where h, t ∈ E, r ∈ R, f r (h,t) represents the true label value of the triple (h,r,t), where, for a positive triple (h,r,t), f r (h,t) = 1 indicates that the head entity h and the tail entity t can be connected through the relation r, while for the negative triple (h,r,t), f r (h,t)=0 indicates that the head entity h and the tail entity t cannot be connected by relation r. This represents the predicted function value of the score function in a knowledge graph embedding model, aiming to determine whether head entity h and tail entity t can be connected through relation r. A general KGE training framework aims to learn the vector representations h, r, t of entities h, t and relation r in the KG by minimizing the difference between the true label values ​​of positive and negative triples and the predicted values ​​based on the score function. Generally, non-sampling-based training frameworks use a least-squares-based loss function to define their optimization training loss function, i.e.

[0078]

[0079] Among them, C hrt The weights of the corresponding triples (h, r, t) represent the importance of the triples in model training. In traditional KGE models based on negative sampling, such as the TransE model, since it only samples a portion of the non-positive triples in the knowledge graph as negative triples for training, the C of the positive triples and the sampled negative triples can be considered equal. hrt The value was uniformly set to 1, while the C of the unsampled negative triples was... hrt The value is set to 0; while in the NS-KGE model framework, the weights C of all positive triples are... hrt The weights were uniformly set to 1, and the weights of all negative triples were uniformly set to a positive real number less than 1. The optimal weight value obtained through experiments was 0.001.

[0080] refer to Figure 3 The non-sampling knowledge graph embedding training method disclosed in this application includes the following steps:

[0081] A knowledge graph embedding model based on a translation distance model is constructed based on the equipment's historical maintenance records, sensor data, and expert knowledge.

[0082] In the knowledge graph embedding model based on the translation distance model, the performance degradation problem caused by the squared loss function is alleviated by the non-sampling loss function based on the probability prediction function.

[0083] A weighting factor is used to dynamically weight the loss function of negative triples to reduce the adverse effects of easily separable negative triples on model training.

[0084] By dividing the dataset into training and testing sets, the knowledge graph embedding model is trained, the model parameters are optimized, and potential equipment problems are discovered.

[0085] In one embodiment, the knowledge graph is composed of heterogeneous information from sensor data, where temperature, pressure, and rotational speed are considered nodes, and interactions (such as the relationship between temperature and rotational speed) are considered edges. Therefore, knowledge containing a large amount of information can be obtained. Figure 3 Tuples include topological structure and semantic relationships.

[0086] Existing non-sampling-based training frameworks transform the optimization training of each original KGE model into a squared-based loss function approach, thus weakening the training performance of the original KGE models. Therefore, consider... The significance of this can be understood as the probability prediction function P(r|h,t) that the triple (h,r,t) is predicted to be a positive triple. For the triple (h,r,t), its probability prediction function P(r|h,t) is defined as:

[0087] P(r|h,t)=sigmoid(δ r -F r (h,t)) (2)

[0088] Where, δ r >0 indicates a threshold hyperparameter F related to relation r. r Let (h,t) be the score function of the triple (h,r,t). If the score function F of (h,r,t) is... r The value of (h,t) is greater than the threshold δ r The value of P will be close to 0, that is, the probability of (h,r,t) being predicted as a positive triplet is close to 0, so it is predicted as a negative triplet; conversely, the prediction probability function value is close to 1, so it is predicted as a positive triplet. Therefore, the range of P(r|h,t) is [0,1].

[0089] Substituting the probability prediction function P(r|h,t) into formula (1), we get:

[0090]

[0091] Expanding the formula using positive and negative sample sets, we get:

[0092]

[0093] Among them, C + C- This represents the weights of the positive and negative triples. For a positive triple, its true value f is... r (h,t)=1, for a negative triple, its true value f r Substituting (h,t)=0 into formula (4), we get:

[0094]

[0095] Since the probability prediction function P(r|h,t) of the triplet takes values ​​in the range [0,1], therefore, P(r|h,t)≥0 and 1-P(r|h,t)≥0. To simplify calculations and improve training efficiency, this application simplifies the loss function of the non-sampled knowledge embedding model as follows:

[0096]

[0097] As can be seen from formula (6), compared with the loss function based on least squares, the loss function of the simplified non-sampled knowledge graph embedding framework is defined based on the probability prediction function defined by the original KGE model, which can maintain the characteristics of the original KGE model and thus improve training performance.

[0098] For non-sampling training, since the dataset only contains positive samples, all negative samples are constructed later based on the closed-world hypothesis, i.e., all triples consisting of entities and relations in the dataset that are not in the dataset are negative samples. Since positive samples are the primary basis for training, they should be treated with equal importance during training. Based on this, this application assigns weight scores (C) to all positive triples. + The default value is set to 1. However, for the negative triples constructed later, there are a large number of easily separable negative triples with obvious errors. These easily separable negative triples cannot provide effective information for gradient descent during model optimization, making the model prone to getting trapped in local optima and failing to learn the optimal embedding. Therefore, treating these numerous easily separable negative triples the same as other negative triples, such as hard-to-separate negative triples, during training can easily affect the direction of gradient descent, thus affecting training accuracy and convergence speed. To address this, this application considers adding a model-based weighting factor to the loss function for all negative triples, dynamically adjusting the impact of different negative triples on model training during the training process. Next, two different negative triplet weighting factors will be introduced.

[0099] (1) Focal-Loss weighting factor

[0100] The core idea of ​​Focal Loss is to scale the loss function of the model as a whole, with easier samples being scaled more than harder samples. This reduces the weight of easier samples in the loss function and highlights the weight of harder samples, allowing the model to focus more on harder samples during training by reducing the influence of easily classified negative samples.

[0101] Define the Focal-Loss weighting factor for negative triples. for:

[0102]

[0103] Where β1,γ>0 are hyperparameters, β1 is used to balance the weights between positive and negative samples, and P F This represents the probability that a negative triplet is misclassified, specifically the probability that a negative triplet is misclassified as a positive triplet. For a negative triplet (h,r,t), P... F (r|h,t) is defined as:

[0104] P F (r|h,t)=sigmoid(δ r -F r (h,t)) (8)

[0105] Where, δ r >0 is the threshold hyperparameter, F r (h,t)>0 is the score function for the triple (h,r,t). It can be seen that P F It is consistent with the probability function P defined by formula (2).

[0106] The Focal-Loss weighting factor can be understood as: the score function F of the negative triple (h,r,t). r The larger the value of (h,t) is than the threshold δ, the better. r Prediction probability function P F The closer the value of is to 0, the closer the probability that the negative triplet is predicted as a positive triplet is to 0, i.e., the easier it is to identify a negative triplet. F Approaching 0; correspondingly, the P-value of indistinguishable negative triples is close to 0. F Greater than 0. Since β1,γ>0, 1≥P F ≥0, we get C - ≥0 indicates that C - With P F Positive correlation, i.e., P F The closer C is to 0, the better. - The closer P is to 0, the better. F The larger C is, the better. - The larger the value, the greater the P-value. In model training, for easily separable negative triples, its P-value is... F The closer it is to 0, the higher its weighting factor C. -The closer it is to 0, the more indistinguishable negative triples are, and for such triples, their P... F The larger the value of C is than 0, the better. - The value is also greater than 0. Therefore, the model dynamically adjusts the weights of easily separable and difficult negative triples based on the predicted probability. Compared to difficult negative triples, the weight of easily separable negative triples is reduced by scaling the weight value to a greater extent, thus reducing the model's focus on these samples and minimizing their impact on model training. For difficult negative triples, the scaling degree of their weight is lower than that of easily separable negative triples, which is equivalent to indirectly increasing the weight value of difficult negative triples, allowing the model training to focus on difficult negative triples and thus improving the model training effect. This is similar to students studying for exams. For knowledge points that have already been mastered, such as easily separable triples, focusing on these knowledge points will not have a significant effect on improving grades, so less effort can be spent. However, for knowledge points that have not yet been mastered well, such as difficult negative triples, more effort is needed to strengthen the mastery of these knowledge points, which is the key to improving grades.

[0107] Combining formulas (6), (7), and (8), we can obtain:

[0108]

[0109] Where β1,γ>0 are hyperparameters.

[0110] (2) Self-adv weighting factor

[0111] Self-adversarial negative sampling posits that, during the negative sampling process, for hard-to-distinguish negative triples, since the model's classification ability for these triples is insufficient, their sampling probability should be increased to strengthen the training of hard-to-distinguish negative samples and improve the model's classification ability for hard-to-distinguish negative triples; while for easy-to-distinguish negative triples, they should be sampled with the smallest possible sampling probability to ultimately improve the model's training effect.

[0112] For easily separable negative triples, since they don't provide much meaningful information for training, their weight in the loss function should be significantly smaller than that of difficult-to-separate negative triples; conversely, for difficult-to-separate negative triples that provide more meaningful training information, they should be assigned a larger weight in the loss function. Based on this, this application defines a self-adv weight factor. for:

[0113]

[0114] Here, β2, α>0 are hyperparameters, with β2 also serving to balance the weights between positive and negative samples. The defined weight factor C... - In fact, it is: the negative triplet (h) after function transformation j ',r,t jThe proportion of the score function value of ') to the sum of the score function values ​​of all negative triplet functions with relation r after transformation.

[0115] Combining formulas (6) and (10), we can obtain:

[0116]

[0117] Where β2,α>0 are hyperparameters.

[0118] The two weighting factors share the same basic principle: reducing the weight of easily separable negative triples and increasing the weight of hard-to-separate negative triples. The difference lies in the calculation method. The Focal-Loss weighting factor directly yields a weight by calculating the score function value of the triple, while the self-adversarial negative sampling-based weighting factor requires calculating the function values ​​of all negative triples with relation r, resulting in a relative weight value. This application will experimentally examine the training effectiveness of the two weighting factors.

[0119] In one embodiment, five typical knowledge graph embedding models, namely DistMult, SimpleE, ComplEx, TransE, and RotatE, will be used as examples to demonstrate the application of the newly proposed non-sampling training framework on different KGE models.

[0120] For TransE, the scoring function F of the triples r The definition of is:

[0121] F r =||h+rt||2 (12)

[0122] Therefore, the probability prediction function P(r|h,t) based on TransE can be defined as:

[0123] P(r|h,t)=sigmoid(δ r -||h+rt||2) (13)

[0124] For RotatE, the scoring function F of the triples r The definition of is:

[0125]

[0126] Therefore, the probability prediction function P(r|h,t) based on RotatE can be defined as:

[0127]

[0128] Calculate the model's score function F rAfter obtaining the probability prediction function P(r|h,t), we substitute it into formulas (9) and (11) to obtain the final loss function of the training framework. Then, we perform optimization training to learn the optimal embeddings h, r, t for h, r, t.

[0129] This application will evaluate the training performance of the NSDW-KGE framework through experiments.

[0130] This application will conduct experiments on the CMAPSS benchmark dataset used in the NS-KGE framework to evaluate the training performance of the NSDW-KGE framework. Both datasets are commonly used benchmark datasets in knowledge graph embedding. Specifically, UCIMachine Learning Repository1.CMAPSSData Set1 (Commercial Modular Aero-Propulsion System Simulation) is a dataset for predicting the remaining service life (RUL) of aero-engines. It contains sensor data from multiple flight cycles, such as temperature, pressure, and speed. UCI CMAPSSDataSet2.Pump Sensor Data comes from an industrial pump system and contains time-series data from multiple sensors, which can be used for fault detection and diagnosis.

[0131] Following the common dataset setup method for knowledge graph embedding experiments, the triplet set was divided into a training set, a validation set, and a test set, accounting for approximately 85%, 5%, and 10%, respectively. The specific experimental steps for link prediction are as follows:

[0132] First, the optimal parameters of the model are obtained by learning through the training set and validation set, including the optimal embedding representation of all entities and relations.

[0133] Then, a link prediction task is performed on the test set to evaluate the model's ability to predict missing head or tail entities in triples, i.e., the ability to predict t given (h,r) or h given (r,t). Specifically, for each test triple (h,r,t), the tail entity t is replaced with each entity x in the knowledge graph to obtain a set of candidate triples (h,r,x). The score function F(h,r,x) of this set of candidate triples is calculated, and the replaced triples are sorted in ascending order based on their scores. This gives the ranking of the test triple (h,r,t) among all candidate triples, denoted as rank(t). Similarly, the ranking of all candidate triples (x,r,t) obtained by replacing the head entity h with each entity x in the knowledge graph, denoted as rank(h), can also be obtained.

[0134] Furthermore, this application employs a "Filter" setting. Specifically, before ranking each test triplet on the test set, candidate triplets present in the training, validation, and test sets are removed from this set of candidate triplets. This is because the candidate triplet is actually correct, and its ranking based on the objective function score is reasonably higher than the original test triplet's ranking. Removing the candidate triplet before obtaining the ranking score for each test triplet ensures that it does not participate in the ranking, thus eliminating the influence of interfering factors.

[0135] In the link prediction task, this application uses the following evaluation metrics to assess the quality of the learned embeddings.

[0136] (1) MR value, which is the average of the rankings of all test triples, is defined as follows:

[0137]

[0138] (2) MRR value, which is the average of the reciprocals of the rankings of all test triples, is defined as follows:

[0139]

[0140] (3) Hits@N, which is the ratio of the number of test triples with a ranking no greater than N to the total number of test triples. In this experiment, this application used three indicators: Hits@1, Hits@3 and Hits@10.

[0141] A larger MRR and Hits@N values, or a smaller MR value, indicate better model performance; conversely, a smaller MRR and Hits@N values ​​indicate worse model performance.

[0142] Comparison methods are divided into three categories:

[0143] (1) Original knowledge graph embedding model, i.e., knowledge graph embedding model that does not use a non-sampling framework. This application selects two classic models, TransE and RotatE, as comparison methods.

[0144] TransE: Translation Embedding Model, which minimizes the distance between the head entity vector and the tail entity vector after translation of the relation vector.

[0145] RotatE: A rotational embedding model in complex space where the distance between the head entity vector and the tail entity vector is minimized after rotation by the relation vector.

[0146] (2) KGE models using the NS-KGE framework, i.e., versions of the original knowledge graph embedding models that apply the NS-KGE framework, including two models: NS-TransE and NS-ComplEx (RotatE has an NS-KGE version).

[0147] (3) KGE models using the NSDW-KGE framework, i.e., versions of the original knowledge graph embedding models that apply the NSDW-KGE framework, including two models: NSDW-ComplEx and NSDW-RotatE. Since two dynamic weighting methods are applied, they can be further subdivided into the Focal Loss version and the Self-adv version of NSDW-KGE, denoted as NSDW-TransE-FL, NSDW-RotatE-FL, and NSDW-TransE-SA, respectively.

[0148] The learning rate λ used in the Stochastic Gradient Descent (SGD) method is {0.0005, 0.0001, 0.01, 0.1}, and the hyperparameters β1 and β2 range from {0.01, 0.1, 1, 10, 100}. The hyperparameter γ ranges from {0.1, 0.2, 0.5, 1, 2, 5}. The embedding dimension d ranges from {100, 300, 500, 1000}, α ∈ {0.1, 0.3, 0.5, 0.8, 8, 1.0}, and the batch size is 128. The model is trained iteratively for 1000 epochs on each dataset. The optimal parameters are determined by the Hits@10 values ​​on the validation set. The optimal parameter configurations for the model on the two datasets are shown in Table 2.

[0149] Table 2 Optimal Hyperparameter Settings

[0150]

[0151] The model can also address the shortcomings of data sparsity and imbalance to some extent. Because all entities and relationships in the dataset either form positive samples for the model or construct negative samples needed for model training, the samples participating in training are enhanced to a certain extent, and the imbalance in the frequency of certain entities and relationships in the dataset is alleviated to some extent. The reason for saying "to some extent" is that the useful training information provided by easily distinguishable negative samples is limited.

[0152] The model works for three reasons: first, sample augmentation (significantly increasing the number of negative training samples); second, it addresses the data imbalance problem to some extent; and third, it dynamically weights the loss function to reduce the impact of easily separable negative triples on model training. Experimental results are shown in Table 3 and... Figure 3 As shown.

[0153] Table 3 Link Prediction Results

[0154]

[0155] This application proposes a dynamic weighted, scalable non-sampled knowledge graph embedding framework, NSDW-KGE. The model employs a non-sampled training method, treating all non-positive triples composed of entities and relations in the knowledge graph as negative triples for training. This simplifies the definition of a new non-sampled loss function. Two weighting methods, Focal-Loss and self-adv, are used to dynamically weight the loss function of the negative triples. Two KGE models, TransE and RotatE, are used as examples to demonstrate the scalability of this training framework across various KGE models.

[0156] This application proposes a dynamically weighted, scalable, non-sampled knowledge graph embedding method. First, it modifies the least-squares-based training loss function, maintaining the performance of the original knowledge graph embedding model. Second, by dynamically weighting the loss function, it reduces the adverse effects of easily separable negative triples on model training, thereby improving training accuracy. Finally, the application demonstrates its scalability by training on different types of original knowledge graph embedding models. Experiments on benchmark datasets using various knowledge graph embedding models show that this method achieves good training performance and is applicable to a wide range of knowledge graph embedding models.

[0157] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0158] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0159] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0160] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A non-sampling knowledge graph embedding training method based on dynamic weighting, characterized in that, Includes the following steps: A knowledge graph embedding model based on a translation distance model is constructed based on the equipment's historical maintenance records, sensor data, and expert knowledge; the knowledge graph triples in the knowledge graph embedding model include topological structure and semantic relations. In the knowledge graph embedding model based on the translation distance model, the performance degradation problem caused by the squared loss function is alleviated by the non-sampling loss function based on the probability prediction function. A weighting factor is used to dynamically weight the loss function of negative triples to reduce the adverse effects of easily separable negative triples on model training. Divide the dataset into training and testing sets, train the knowledge graph embedding model, optimize the model parameters, and identify potential equipment problems; The knowledge graph embedding model based on the translation distance model includes at least one of the TransE and RotatE models; For the knowledge graph embedding model G based on the translation distance model, for the triple (h, r, t), where h, t ∈ E, E is the entity set, h and t are the head and tail entities in the triple, r ∈ R, R is the relation set, r is the relation, and f r (h, t) represents the true label value of the triple (h, r, t), where for a positive triple (h, r, t), f r (h, t)=1 indicates that the head entity h and the tail entity t can be connected through the relation r. For a negative triple (h, r, t), f r (h, t)=0 indicates that the head entity h and the tail entity t cannot be connected by relation r; The predicted function value is represented by the score function of the knowledge graph embedding model, which aims to determine whether the head entity h and the tail entity t can be connected by the relation r. The training framework of the model aims to learn the vector representations h, r, t of entities h, t and relation r in the knowledge graph embedding model by minimizing the difference between the true label values ​​of positive and negative triples and the predicted values ​​based on the score function. Will The probability prediction function P(r | h, t) is used to predict the triple (h, r, t) as a positive triple; for the triple (h, r, t), its probability prediction function P(r | h, t) is defined as: Where, δ r > 0 indicates a threshold hyperparameter F related to relation r. r Let (h, t) be the score function of the triple (h, r, t); if the score function F of (h, r, t) is... r The value of (h, t) is greater than the threshold δ r If the value of P is close to 0, that is, if the probability of (h, r,t) being predicted as a positive triplet is close to 0, then it is predicted as a negative triplet; conversely, if the value of the prediction probability function is close to 1, then it is predicted as a positive triplet. The training framework uses a least-squares-based loss function to define its loss function for optimizing training, i.e. C hrt The weights of the corresponding triples (h, r, t) represent the importance of the triples in model training. Expanding the above equation with respect to both positive and negative sample sets, we get: Among them, C + C - S represents the weights of the positive and negative triples. + It is a set of positive triples, S - It is a set of negative triples; For a positive triple, its true value f r (h, t)=1, for a negative triple, its true value f r Substituting (h, t)=0 into the above equation, we get: 。 2. The non-sampling knowledge graph embedding training method based on dynamic weighting according to claim 1, characterized in that, Since the probability prediction function P(r | h, t) for the triplet takes values ​​in the range [0, 1], therefore, , ; The loss function can be simplified to the following definition: 。 3. The non-sampling knowledge graph embedding training method based on dynamic weighting according to claim 2, characterized in that, In the loss function, a model-based weighting factor is added to all negative triples to dynamically adjust the impact of different negative triples on model training during the training process. The negative triplet weighting factors include the Focal-Loss weighting factor and the self-adv weighting factor.

4. The non-sampling knowledge graph embedding training method based on dynamic weighting according to claim 3, characterized in that, The negative triplet Focal-Loss weighting factor C F - for: C F - = β1·P F γ Where β1, γ>0 are hyperparameters, β1 is used to balance the weights between positive and negative samples, and P F P represents the probability that a negative triplet is misclassified, i.e., the probability that a negative triplet is misclassified as a positive triplet; for a negative triplet (h, r, t), P F (r |h, t) is defined as: Where, δ r > 0 is the threshold hyperparameter, F r (h, t) > 0 is the score function for the triple (h, r, t); The role of the Focal-Loss weighting factor is as follows: During model training, for easily separable negative triples, its P... F The closer it is to 0, the higher its weighting factor C. - The closer it is to 0, the more indistinguishable negative triples are, and for such triples, their P... F The larger the value of C is than 0, the better. - The value is also greater than 0; therefore, the model dynamically adjusts the weights of easily separable and hard-to-separate negative triples based on the predicted probability. By scaling the weight values ​​to a greater extent, the weight of easily separable negative triples is reduced, so that the model pays less attention to this part of the samples, thereby reducing the impact of easily separable negative triples on model training; for hard-to-separate negative triples, their weights are less than the scaling degree of easily separable negative triples, so that the model training focuses on hard-to-separate negative triples, thereby improving the model training effect. Combining the negative triplet Focal-Loss weighting factor, the final loss function is obtained as follows: Where β1, γ>0 are hyperparameters.

5. The non-sampling knowledge graph embedding training method based on dynamic weighting according to claim 4, characterized in that, self-adv weighting factor C s - for: Where β2, α>0 are hyperparameters, with β2 used to balance the weights between positive and negative samples; the self-adv weight factor C - Its function is: the negative triplet (h) after function transformation j ' , r, t j ' The weight of the score function value of ) out of the sum of the score function values ​​of all negative triplet functions with relation r after transformation; Combining the self-adv weight factor, the final loss function is as follows: Where β2, α>0 are hyperparameters.

6. The non-sampling knowledge graph embedding training method based on dynamic weighting according to claim 5, characterized in that, For TransE, the scoring function F of the triples r The definition of is: The probability prediction function P(r | h, t) based on TransE is defined as: For RotatE, the scoring function F of the triples r The definition of is: The probabilistic prediction function P(r | h, t) based on RotatE is defined as: Calculate the model's score function F r After obtaining the probability prediction function P(r | h, t), we substitute it into the loss function to obtain the final loss function of the training framework. Then, we perform optimization training to learn the optimal embeddings h, r, t for h, r, t.

Images