Data anomaly detection method based on knowledge graph

Abstract

The present invention discloses a data anomaly detection method based on a knowledge graph, which relates to the field of data detection technology and is used to solve the problem of overloaded association of graph entity relationships and increased probability of invalid connections. Through multi-dimensional feature collection and logistic regression model calculation, the data association degree coefficient is accurately obtained. By comparing the association degree coefficient with a preset threshold, it is ensured that the screened related entities have a high correlation, thereby improving the accuracy of data connection. At the same time, the number of alternative connections and the number of screened entities are counted, and the connection rules are dynamically adjusted through weighted calculation of deviation rate and change ratio. By sorting and optimizing the association degree coefficients in the number of alternative connections, a set of fuzzy rules are formulated for fuzzy reasoning based on individual connection relationships, time correlation and data sparsity. The weight result is calculated according to the data association degree coefficient, thereby improving the efficiency of data utilization and increasing the accuracy of anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of data detection, and more specifically, to a data anomaly detection method based on a knowledge graph. Background Art

[0002] Knowledge graph is a technology used for structured storage and expression of knowledge. It builds a semantic network through entities (nodes) and the relationships (edges) between them. Through semantic rules and relationship constraints, it can express complex logical relationships, support dynamic data updates and knowledge supplementation, and infer unclear data based on logical rules and contextual information.

[0003] The existing technology has the following deficiencies:

[0004] Currently, in knowledge graphs, data relevance determines the number of connections between entities and their dynamic changes. The strength of the relevance directly affects the rationality of the connections and the overall structure of the knowledge graph. However, due to the diverse sources of data input and frequent updates, this leads to overloaded relevance, and changes in relevance are not reflected in the graph in a timely manner. This increases the probability of invalid connections, reduces data utilization efficiency, and leads to distortion of the knowledge graph. Therefore, a data anomaly detection method based on knowledge graphs is proposed.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a data anomaly detection method based on a knowledge graph, which solves the problems raised in the above-mentioned background technology by applying different product inspection methods.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: a method for detecting anomalies in data based on a knowledge graph; and a method for detecting anomalies in data based on a knowledge graph, comprising:

[0008] S1: Collect the semantic similarity, entity attribute matching degree, relationship connection level and entity interaction number of each entity, and calculate the data association degree coefficient by substituting it into the logistic regression.

[0009] S2: Obtain the data association coefficient. Taking one entity as an individual, count the data association coefficients of all other entities related to it, and compare them with the preset association threshold. Select other entities marked as related entities and count their number as the number of candidate connections. At the same time, count the number of screened entities and compare them with the abnormal threshold. Based on the comparison results, enable the connection adjustment rule, perform weighted calculation based on the collection deviation rate and the change ratio of the number of candidate connections to obtain the adjustment ratio, and determine the standard number of connections.

[0010] S3: Obtain the standard number of connections and the number of candidate connections, compare them, generate a connection number adjustment plan, sort the data correlation coefficients in the candidate connection numbers related to the individual according to their numerical values, and delete them from small to large until the number of candidate connections reaches the standard number of connections, thereby obtaining the connection relationship of the individual;

[0011] S4: Obtain the connection relationship of individuals, obtain directly connected entities and indirectly connected entities, record the timestamp sequence of interaction events between individuals and each connected entity, and obtain time correlation and data sparsity;

[0012] S5: Use fuzzy logic to determine the data association degree coefficient and calculate the weight result based on the time association degree and data sparsity.

[0013] In a preferred embodiment, the entity name, label, attribute value and annotation, as well as the entity's adjacent nodes and their connection relationships are obtained through the unique identification of each entity, the entity and its information are converted into vector form, the text description of the entity is encoded using a pre-trained word embedding model, and the entity vector is generated. The semantic similarity between the two entities is calculated based on the entity vector and the cosine similarity. , If represents the i-th alignment, then there are i alignment entities;

[0014] Extract the attributes of each entity from the knowledge graph, define different matching rules according to the different attribute types, compare whether the attribute values ​​are the same for discrete and definite attribute values, that is, calculate the ratio of the number of matching attributes to the total number of attributes to obtain the discrete matching degree, and use the difference to calculate the continuous matching degree for continuous numerical attributes; set different weights for the discrete matching degree and the continuous matching degree, and substitute them into the weighted calculation to obtain the entity attribute matching degree ;

[0015] By constructing the path in the knowledge graph, using breadth-first search to calculate the shortest path, starting from the starting entity node, expanding outward layer by layer until the target entity node is found, the path length is the number of relationship levels ;

[0016] By distinguishing direct interactions from indirect interactions, and recording each time entities interact through connection relationships, it is recorded as a direct interaction. By finding all the paths that the current entity passes through, the number of paths is calculated. On each path, each time it passes through the current entity, it is recorded as an indirect interaction. The direct interactions and indirect interactions are accumulated and calculated to get the number of entity interactions. ;

[0017] The main semantic similarity, entity attribute matching degree, relationship connection level and entity interaction times are substituted into the logistic regression formula to calculate the data association coefficient.

[0018] In a preferred embodiment, the data association coefficients corresponding to all other related entities are counted;

[0019] After obtaining the data correlation coefficients corresponding to all other entities, the data correlation coefficients are compared and analyzed with the iterative correlation thresholds;

[0020] If the data association coefficient is greater than or equal to the association threshold, the other entities corresponding to the data association coefficient are marked as associated entities, and an association signal is generated;

[0021] If the data association coefficient is less than the association threshold, the other entities corresponding to the data association coefficient are marked as screened entities, and a screening signal is generated;

[0022] The correlation coefficient of the data marked as associated entities is counted as the number of candidate connections;

[0023] The correlation coefficient of the data marked as the screened entity is counted and the quantity is determined, and compared with the abnormal threshold;

[0024] If the number of data association coefficients marked as screened entities is greater than or equal to the anomaly threshold, a data anomaly signal is generated. After the screened entities are deleted, the adjustment connection rules are enabled. Otherwise, a data normal signal is generated.

[0025] In a preferred embodiment, the connection rule adjustment is enabled, and the specific steps are as follows:

[0026] The acquisition deviation rate and the change rate of the number of alternative connections;

[0027] The deviation rate is used to measure the difference between the number of screened entities and the anomaly threshold. The logic for obtaining it is to calculate the deviation rate P by subtracting the anomaly threshold from the number of data association coefficients marked as screened entities and then calculating the ratio of the coefficients to the anomaly threshold.

[0028] The change ratio of the number of alternative connections reflects the degree of fluctuation of the current number of alternative connections compared to the average number of alternative connections. It is used to measure the stability of the number of alternative connections. Its acquisition logic is to subtract the historical average number of alternative connections from the current number of alternative connections and calculate the ratio of change R of the number of alternative connections with the historical average number of alternative connections.

[0029] In a preferred embodiment, the deviation rate and the change ratio of the number of candidate connections are substituted into a weighted formula to calculate an adjustment ratio, and corresponding connection rules are added:

[0030] ;

[0031] Where, To adjust the ratio, and are the weights of the influence of the control deviation rate and the change ratio of the number of alternative connections on the adjustment ratio;

[0032] in, , ensuring that the adjustment ratio does not lead to over-correction due to excessive weighting;

[0033] Multiply the adjustment ratio by the number of alternative connections to obtain the standard number of connections B;

[0034] For data marked as screened entities with a correlation coefficient number less than the abnormal threshold, the default adjustment ratio is 1, and the default number of candidate connections is the standard number of connections.

[0035] In a preferred embodiment, if the number of standard connections is the same as the number of candidate connections, the number of candidate connections is directly used as the number of standard connections to complete the adjustment of the individual connection relationship;

[0036] If there is a difference between the standard number of connections and the number of alternative connections, a connection number adjustment plan is generated. The standard number of connections is subtracted from the number of alternative connections to obtain the difference number of connections. The set of data association degree coefficients in the number of alternative connections is deleted from small to large so that the number of alternative connections is the same as the standard number of connections. The individual connection relationships are connected according to the adjusted number of alternative connections.

[0037] In a preferred embodiment, the interaction timestamp sequence between an individual and each directly or indirectly connected entity is extracted through the individual's connection relationship, the standard deviation of the timestamp sequence is calculated, and a normalization formula is introduced to map the time correlation to the interval [0, 1] to obtain the time correlation degree;

[0038] Count the number of direct and indirect connections of an individual, calculate the ratio of the total number of connections of all entities to the number of all entities in the knowledge graph to obtain the average number of connections of entities in the knowledge graph, and calculate the ratio of the number of connections of an individual to the average number of connections of entities in the knowledge graph to obtain data sparsity.

[0039] In a preferred embodiment, time correlation and data sparsity are defined as input variables and divided into different fuzzy sets respectively;

[0040] The weighted result of the data association coefficient calculation is defined as the output variable and divided into fuzzy sets;

[0041] Formulate fuzzy rules to describe the influence of time correlation and data sparsity on the weight calculation results of data correlation coefficient;

[0042] Fuzzy reasoning is performed based on fuzzy rules to determine the data association degree coefficient and calculate the weight result.

[0043] The technical effects and advantages of the present invention are as follows:

[0044] 1. This invention accurately obtains data association coefficients through multi-dimensional feature collection and logistic regression model calculations. By comparing the association coefficients with preset thresholds, it ensures that the selected related entities have a high degree of relevance, thereby improving the accuracy of data connections. At the same time, the number of alternative connections and the number of screened entities are counted, and through weighted calculation of the deviation rate and change ratio, the connection rules are dynamically adjusted to ensure the scientific nature of the standard number of connections. By sorting and optimizing the association coefficients in the number of alternative connections, fine-grained control of individual connections is achieved, which not only avoids the increase of invalid connections but also improves the efficiency of data utilization.

[0045] 2. The present invention obtains individual connection relationships, obtains directly connected entities and indirectly connected entities, records the timestamp sequence of interaction events between individuals and each connected entity, obtains time correlation and data sparsity, formulates a set of fuzzy rules for fuzzy reasoning, determines the data correlation degree coefficient and calculates the weight result, improves the dynamic adaptability of data correlation, enhances the flexibility and stability of connection rules, and increases the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the method for data anomaly detection based on knowledge graph of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] In fact, data anomalies in knowledge graphs are a problem of data relevance. In data relevance, data processing includes but is not limited to data redundancy and duplication, data missingness and incompleteness, data conflict and contradiction, abnormal association and noisy data, and polysemy and semantic ambiguity. Among them, the excessive number of data sources leads to complex relationship connections, which are difficult to handle and may even cause relationship contradictions and conflicts, resulting in data anomalies.

[0049] Example 1

[0050] See also Figure 1 , the data anomaly detection method based on knowledge graph, the specific operation process is as follows:

[0051] S1: Collect the semantic similarity, entity attribute matching degree, relationship connection level and entity interaction number of each entity, and calculate the data association degree coefficient by substituting it into the logistic regression.

[0052] The knowledge graph contains multiple entities, each of which forms corresponding data through different attributes and relationships in the graph, and each entity is represented as a piece of data;

[0053] Semantic similarity is a metric that measures the semantic similarity between two entities in a knowledge graph. It reflects the degree of relevance between two entities in terms of description, context, or logic. The acquisition process uses each entity's unique identifier to obtain its name, label, attribute value, and annotation, as well as its adjacent nodes and their connections. The entity and its information are then converted into vector form. A pre-trained word embedding model is used to encode the entity's textual description to generate an entity vector. This entity vector is then substituted into the cosine similarity to calculate the semantic similarity between the two entities.

[0054] It should be noted that pre-trained word embedding models are language models pre-trained based on large-scale text data. Their core purpose is to map text (words, phrases, sentences, etc.) into a continuous vector space, so that text with similar semantics is closer in the vector space. These models use the contextual relationships of natural language to learn the semantic representation of words or sentences and are the foundation of semantic computing. For example, the static word embedding model Word2Vec is first trained using the Skip-Gram or CBOW method to find words with similar semantics that appear in similar contexts. Each word is output with a fixed vector representation. The vector representation of the vocabulary is then substituted into the cosine similarity calculation to obtain the result.

[0055] Specifically, the calculation formula of cosine similarity is expressed as:

[0056] Assume entities A and B, and the entity vector is and Indicates that the specific cosine similarity formula is as follows:

[0057] ;

[0058] Where, is the dot product of two entity vectors, is the modulus of the entity vector A, is the modulus of the entity vector B; according to the formula, we get the semantic similarity ; Where i is the i-th alignment, and the number of aligned entities is i;

[0059] The degree of entity attribute matching refers to the consistent matching of two entities in the knowledge graph based on attribute information (such as name, type, characteristic value, etc.), which is used to measure the strength of their correlation at the attribute level. Its acquisition logic is to extract the attributes of each entity from the knowledge graph, define different matching rules according to the different attribute types, and compare whether the attribute values ​​are the same for discrete and definite attribute values. That is, the discrete matching degree is obtained by calculating the ratio of the number of matching attributes to the total number of attributes. For continuous numerical attributes, the continuous matching degree is calculated using the difference; different weights are set for the discrete matching degree and the continuous matching degree, and the entity attribute matching degree is obtained by substituting them into the weighted calculation. ;

[0060] Among them, for continuous numerical attributes, the difference is used to calculate the continuous matching degree. The specific calculation formula is:

[0061] ;

[0062] Where, is the continuous matching degree, A and B are the values ​​of the corresponding attributes of the two entities, is the maximum value of the corresponding attributes of the two entities;

[0063] Specifically, the attributes of each entity include but are not limited to basic attributes (such as entity name, identifier, type), numerical attributes (such as quantity, time, etc.), classification attributes (such as classification label, status), text attributes (such as description, definition, etc.), and relationship attributes (such as relationship with other entities, etc.), which are not detailed here;

[0064] It should be noted that different weights are assigned to discrete and continuous matching degrees. These weights are determined based on the degree to which discrete attributes (such as type and classification) are more critical in the domain defined by the experimenters, or by evaluating the contribution of discrete and continuous attributes to the matching results through data analysis. For example, the correlation between two types of attributes on the matching results is calculated, which will not be elaborated here.

[0065] From the above, we can see that the weights for different entities are also different;

[0066] The relationship level refers to the minimum number of hops (i.e., level depth) required to connect two entities through a relationship (relationship edge) in the knowledge graph. It is used to measure the strength of the structural association between them. The smaller the level, the closer the connection between the two entities. The logic for obtaining it is to construct a path in the knowledge graph and use breadth-first search to calculate the shortest path. Starting from the starting entity node, it expands outward layer by layer until the target entity node is found. The path length is the relationship level. ;

[0067] Specifically, breadth-first search is an algorithm that traverses a graph, visiting all nodes in the graph in hierarchical order (starting from the starting node and expanding layer by layer). BFS first visits nodes closer to the starting node, and then gradually visits nodes farther away from the starting node until all reachable nodes in the graph are traversed. The specific implementation steps are as follows: starting from the starting node in the graph, first visit the node, then visit all unvisited nodes directly connected to the current node (i.e., adjacent nodes). For each visited node, its adjacent nodes are added to the queue to be visited, and the nodes are visited in sequence according to the nodes taken out of the queue until the queue is empty. This process ensures that the path from the starting node to the target node is the shortest by expanding the nodes of the graph layer by layer.

[0068] It should be noted that the starting entity node is the entity node that is compared with the other entity nodes in the above content, and the target entity node is the other corresponding entity nodes being compared;

[0069] Among them, the relationship connection level is the length of the shortest path from one entity to another;

[0070] The number of entity interactions refers to the number of interactions (or connections) between two entities in the knowledge graph through direct or indirect relationships. The number of interactions is an important indicator to measure the strength of the connection between two entities. The more interactions, the stronger the association between the two entities. The acquisition logic is to distinguish between direct and indirect interactions, and record each time an entity interacts through a connection relationship as a direct interaction. By finding all the paths that the current entity passes through and calculating the number of paths, each time it passes through the current entity on each path is recorded as an indirect interaction. The direct and indirect interactions are accumulated and calculated to obtain the number of entity interactions. ;

[0071] It should be noted that the number of direct interactions refers to the number of interactions between two entities through a direct single relationship, and the number of indirect interactions refers to the number of interactions between two entities through multiple relationship chains, that is, indirect connections;

[0072] Optionally, we can add more detailed multi-level interaction rules based on the above. For example, the direct relationship between two entities is set as the first-level interaction, the indirect relationship through an intermediate node is set as the second-level interaction, and the relationship chain through two intermediate nodes is set as the third-level interaction, and so on. The deeper the level of interaction, the weaker the connection, which serves as the basis for evaluating the strength of the connection.

[0073] Furthermore, different weights are set for the second and third level interactions of the indirect relationship of multiple interactions. Different weights are set according to the number of interactions to enhance the possibility of hidden data interaction. The specific weighting rules are not limited and will not be elaborated here.

[0074] Normalize the semantic similarity, entity attribute matching degree, relationship connection level, and entity interaction number. All input variables will be converted to the same range to ensure that each input contributes to the model in a balanced manner. Specifically, the normalization method is to normalize the [0,1] interval. The specific formula is:

[0075] ;

[0076] Where, is the semantic similarity, is the normalized semantic similarity, is the minimum value of semantic similarity, is the maximum value of semantic similarity;

[0077] The above formula is also used to normalize the entity attribute matching degree, the number of relationship connection levels, and the number of entity interactions, which will not be elaborated here.

[0078] After normalization, the values ​​of primary semantic similarity, entity attribute matching, relationship connection level, and entity interaction number are all within the range of [0, 1]. The system can make comparisons and decisions on a unified scale, thereby improving the accuracy and efficiency of the logistic regression model.

[0079] Substitute the main semantic similarity, entity attribute matching degree, relationship connection level, and entity interaction number into the logistic regression formula to calculate the data association coefficient. The specific formula is expressed as follows:

[0080] ;

[0081] Where L is the result of logistic regression, i.e., the data correlation coefficient, e is the natural base, and y is the linear combination term of the logistic regression model. Specifically, y can be set as:

[0082] ;

[0083] Where, is the bias term, 、 、 as well as are the regression coefficients of main semantic similarity, entity attribute matching degree, relationship connection level, and entity interaction number;

[0084] S2: Obtain the data association coefficient. Taking one entity as an individual, count the data association coefficients of all other entities related to it, and compare them with the preset association threshold. Select other entities marked as related entities and count their number as the number of candidate connections. At the same time, count the number of screened entities and compare them with the abnormal threshold. Based on the comparison results, enable the connection adjustment rule, perform weighted calculation based on the collection deviation rate and the change ratio of the number of candidate connections to obtain the adjustment ratio, and determine the standard number of connections.

[0085] In the process of calculating the main semantic similarity, entity attribute matching degree, relationship connection level, and entity interaction number in the above step S1, at least two individuals are required to calculate. Therefore, it can be inferred that, taking one entity as an individual, this entity is the entity node compared with the other entity nodes;

[0086] Count the data association coefficients corresponding to all other entities related to it, and record the number of data association coefficients corresponding to all other entities as n;

[0087] After obtaining the data correlation coefficients corresponding to all other entities, the data correlation coefficients are compared and analyzed with the iterative correlation thresholds;

[0088] If the data association coefficient is greater than or equal to the association threshold, the other entities corresponding to the data association coefficient are marked as associated entities, and an association signal is generated;

[0089] If the data association coefficient is less than the association threshold, the other entities corresponding to the data association coefficient are marked as screened entities, and a screening signal is generated;

[0090] It should be noted that the association threshold is obtained by the data association coefficient corresponding to all other entities and the historical association analysis set data, which will not be elaborated here;

[0091] The correlation coefficients of the data marked as associated entities are counted and the number is determined to be m, which is the number of candidate connections;

[0092] The correlation coefficient of the data marked as screened entities is counted and the number is determined as nm, which is then compared with the abnormal threshold;

[0093] If the number of data correlation coefficients marked as screened entities is greater than or equal to the anomaly threshold, a data anomaly signal is generated, the screened entities are deleted, and the adjustment connection rule is enabled. Otherwise, a data normality signal is generated;

[0094] It should be noted that the abnormal threshold is obtained by analyzing the distribution characteristics of the data in the current experiment based on the distribution law of historical data. At the same time, the abnormal threshold is dynamically updated;

[0095] Specifically, enable and adjust the connection rules. The specific steps are as follows:

[0096] The acquisition deviation rate and the change rate of the number of alternative connections;

[0097] The deviation rate is used to measure the difference between the number of screened entities and the anomaly threshold. The logic for obtaining it is to calculate the deviation rate P by subtracting the anomaly threshold from the number of data association coefficients marked as screened entities and then calculating the ratio of the coefficients to the anomaly threshold.

[0098] The change ratio of the number of candidate connections reflects the degree of fluctuation of the current number of candidate connections compared to the average number of candidate connections. It is used to measure the stability of the number of candidate connections. The logic for obtaining it is to subtract the historical average number of candidate connections from the current number of candidate connections and then calculate the ratio of the change ratio R of the number of candidate connections.

[0099] The historical average number of candidate connections may be based on statistics from a historical database or may be based on historical average number of candidate connections determined in different time domains, etc., which will not be elaborated herein.

[0100] Substitute the deviation rate and the change ratio of the number of alternative connections into the weighted formula to calculate the adjustment ratio and add the corresponding connection rules:

[0101] ;

[0102] Where, To adjust the ratio, and are the weights of the influence of the control deviation rate and the change ratio of the number of alternative connections on the adjustment ratio;

[0103] in, , ensuring that the adjustment ratio does not lead to over-correction due to excessive weighting;

[0104] Multiply the adjustment ratio by the number of alternative connections to obtain the standard number of connections B;

[0105] Furthermore, for data whose correlation coefficients are smaller than the abnormal threshold, the default adjustment ratio is 1. In this case, the default number of candidate connections is the standard number of connections.

[0106] S3: Obtain the standard number of connections and the number of candidate connections, compare them, generate a connection number adjustment plan, sort the data correlation coefficients in the candidate connection numbers related to the individual according to their numerical values, and delete them from small to large until the number of candidate connections reaches the standard number of connections, thereby obtaining the connection relationship of the individual;

[0107] Comparing the number of standard connections with the number of alternative connections, the following results are obtained:

[0108] If the number of standard connections is the same as the number of candidate connections, no further processing is required. The number of candidate connections is directly used as the standard number of connections to complete the adjustment of the individual connection relationship.

[0109] If there is a difference between the standard number of connections and the number of alternative connections, a connection number adjustment plan is generated, specifically:

[0110] Sort the data correlation coefficients in the number of candidate connections related to individuals by numerical value and integrate them into a set expression as follows: ;in, is the maximum value of the data association coefficient among the candidate connection numbers, is the second largest value of the data association degree coefficient among the candidate connection quantities, is the minimum value of the data association degree coefficient in the number of alternative connections; m is the total number of data association degree coefficients in the number of individual related alternative connections;

[0111] Subtract the number of standard connections from the number of candidate connections to obtain the number of difference connections. Delete the data association coefficient set in the candidate connection number from small to large, and obtain the adjusted candidate connection number set:

[0112] ;in, is the data association coefficient in the Bth candidate connection number;

[0113] At this time, the number of candidate connections is the same as the number of standard connections, and the individual connection relationships are connected according to the adjusted number of candidate connections;

[0114] This method accurately obtains data association coefficients through multi-dimensional feature collection and logistic regression model calculations. By comparing the association coefficients with preset thresholds, the selected related entities are ensured to have a high degree of relevance, thereby improving the accuracy of data connections. Simultaneously, the number of candidate connections and the number of excluded entities are counted, and through a weighted calculation of the deviation rate and the change ratio, the connection rules are dynamically adjusted to ensure the scientific nature of the standard number of connections. By sorting and optimizing the association coefficients within the number of candidate connections, precise control of data connections is achieved, which not only avoids the increase of invalid connections but also improves the efficiency of data utilization.

[0115] Example 2

[0116] In Example 1 of the present invention, examples are given to illustrate how to accurately obtain the data correlation coefficient through multi-dimensional feature collection and logistic regression model calculation. By comparing the correlation coefficient with a preset threshold, it is ensured that the screened related entities have a high correlation, thereby improving the accuracy of data connection. At the same time, the number of alternative connections and the number of screened entities are counted, and the connection rules are dynamically adjusted through weighted calculation of the deviation rate and the change ratio, and the operation strategy is implemented by sorting and optimizing the correlation coefficients in the number of alternative connections; but in Example 1, the connection relationship is processed and sorted only from the perspective of connecting individuals. However, it is impossible to accurately adjust the data correlation during dynamic changes, reduce the flexibility and stability of the connection rules, and increase the delay of anomaly detection; in response to the above problems, Example 2 of the present invention is further refined;

[0117] S4: Obtain the connection relationship of individuals, obtain directly connected entities and indirectly connected entities, record the timestamp sequence of interaction events between individuals and each connected entity, and obtain time correlation and data sparsity;

[0118] Specifically, a directly connected entity refers to a set of entities that have a one-level relationship with an individual, and an indirectly connected entity refers to a set of entities that are connected to an individual through a multi-level relationship chain. The specific explanations of direct and indirect are described in the explanation of the number of entity interactions in Example 1 above and are not repeated here.

[0119] The logic for obtaining temporal correlation is to extract the interaction timestamp sequence between an individual and each directly or indirectly connected entity through its connection relationship, calculate the standard deviation of the timestamp sequence, and introduce a normalization formula to map the temporal correlation to the [0, 1] interval to obtain the temporal correlation.

[0120] It should be noted that the smaller the standard deviation, the more concentrated the time distribution and the higher the time correlation;

[0121] Among them, the normalization formula is introduced as follows:

[0122] ;

[0123] Where, is the temporal correlation between the individual and each connected entity, is the standard deviation of the timestamp series, is the maximum standard deviation of the connection time distribution in the entire knowledge graph;

[0124] Specifically, the collection rules of Example 2 are set based on the detection of individuals in Example 1. That is, after processing the individual connection relationship and before the next detection of the individual, the collection of Example 2 is enabled to adjust the calculation weight of the data association coefficient, thereby completing the dynamic closed loop of data anomaly detection on the knowledge graph.

[0125] The logic for obtaining data sparsity is to count the number of direct and indirect connections of an individual, calculate the ratio of the total number of connections of all entities to the number of all entities in the knowledge graph to obtain the average number of connections of entities in the knowledge graph, and calculate the ratio of the number of connections of an individual to the average number of connections of entities in the knowledge graph to obtain data sparsity;

[0126] Optionally, the experimenter can consider the local connection density of the category to which the individual belongs, and perform a weighted calculation of the overall sparsity of the knowledge graph and the average sparsity of the category to which the individual belongs to determine the corrected value of data sparsity;

[0127] S5: Using fuzzy logic to determine the data correlation coefficient and calculate the weight result based on the time correlation and data sparsity;

[0128] For example, "High", "Low", and "Medium" are for temporal correlation, and "Sparse", "Dense", and "Moderate" are for data sparsity;

[0129] Formulate a set of fuzzy rules to describe the impact of different input variables on the output variable. The definition of rules can be based on professional knowledge or obtained through data analysis and experiments. For example:

[0130] Mark the time correlation as X, the data sparsity as U, and the weighted result of the data correlation coefficient calculation as C_results;

[0131] Then we can define:

[0132] Rule 1: IF (X is High) AND (U is Dense) THEN (C_results is High)

[0133] Rule 2: IF (U is Low) AND (U is Sparse) THEN (C_results is Low) ...

[0134] Perform fuzzy reasoning based on fuzzy rules to determine the data association coefficient and calculate the weight result;

[0135] It should be noted that the division of fuzzy sets can be adjusted according to actual conditions. For example, although this embodiment takes three fuzzy sets as an example, in fact, the time correlation and data sparsity can be divided into more than three sets to facilitate better precise adjustment according to different time spans.

[0136] Furthermore, for the judgment of time correlation and data sparsity, thresholds can be set according to actual conditions. For example, when the time correlation exceeds 75%, it is marked as "High", and when the data sparsity is higher than 70%, it is marked as "Sparse", and so on. I will not go into details here.

[0137] The weighted result is calculated based on the data association coefficient and substituted into the next round of step S1 to complete the dynamic closed loop of data anomaly detection on the knowledge graph;

[0138] The present invention obtains individual connection relationships, obtains directly connected entities and indirectly connected entities, records the timestamp sequence of interaction events between individuals and each connected entity, obtains time correlation and data sparsity, formulates a set of fuzzy rules for fuzzy reasoning, determines the data correlation degree coefficient and calculates the weight result, improves the dynamic adaptability of data correlation, enhances the flexibility and stability of connection rules, and increases the accuracy of anomaly detection.

[0139] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0140] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0141] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0148] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data anomaly detection method based on knowledge graph, characterized by: include: S1: Collect the semantic similarity, entity attribute matching degree, relationship connection level and entity interaction number of each entity, and calculate the data association degree coefficient by substituting it into the logistic regression. S2: Obtain the data association coefficient. Taking one entity as an individual, count the data association coefficients of all other entities related to it, and compare them with the preset association threshold. Select other entities marked as related entities and count their number as the number of candidate connections. At the same time, count the number of screened entities and compare them with the abnormal threshold. Based on the comparison results, enable the connection adjustment rule, perform weighted calculation based on the collection deviation rate and the change ratio of the number of candidate connections to obtain the adjustment ratio, and determine the standard number of connections. Enable the connection adjustment rule. The specific steps are as follows: The acquisition deviation rate and the change rate of the number of alternative connections; The deviation rate is used to measure the difference between the number of screened entities and the anomaly threshold. The logic for obtaining it is to calculate the deviation rate P by subtracting the anomaly threshold from the number of data association coefficients marked as screened entities and then calculating the ratio of the coefficients to the anomaly threshold. The change ratio of the number of candidate connections reflects the degree of fluctuation of the current number of candidate connections compared to the average number of candidate connections. It is used to measure the stability of the number of candidate connections. The logic for obtaining it is to subtract the historical average number of candidate connections from the current number of candidate connections and then calculate the ratio of the change ratio R of the number of candidate connections. Substitute the deviation rate and the change ratio of the number of alternative connections into the weighted formula to calculate the adjustment ratio and add the corresponding connection rules: ； Where, To adjust the ratio, and are the weights of the influence of the control deviation rate and the change ratio of the number of alternative connections on the adjustment ratio; in, , ensuring that the adjustment ratio does not lead to over-correction due to excessive weighting; Multiply the adjustment ratio by the number of alternative connections to obtain the standard number of connections B; For data marked as screened entities, if the number of correlation coefficients is less than the abnormal threshold, the default adjustment ratio is 1, and the default number of candidate connections is the standard number of connections; S3: Obtain the standard number of connections and the number of candidate connections, compare them, generate a connection number adjustment plan, sort the data correlation coefficients in the candidate connection numbers related to the individual according to their numerical values, and delete them from small to large until the number of candidate connections reaches the standard number of connections, thereby obtaining the connection relationship of the individual; S4: Obtain the connection relationship of individuals, obtain directly connected entities and indirectly connected entities, record the timestamp sequence of interaction events between individuals and each connected entity, and obtain time correlation and data sparsity; S5: Use fuzzy logic to determine the data association coefficient and calculate the weight result based on the time association and data sparsity, and substitute it into the next round of step S1 to complete the dynamic closed loop of data anomaly detection on the knowledge graph.

2. The data anomaly detection method based on knowledge graph according to claim 1 is characterized in that: Through the unique identification of each entity, we get the entity's name, label, attribute value and annotation, as well as the entity's adjacent nodes and their connection relationships. We convert the entity and its information into vector form, use the pre-trained word embedding model to encode the entity's text description, generate the entity vector, and substitute the entity vector into the cosine similarity to calculate the semantic similarity between the two entities. , represents the i-th alignment; Extract the attributes of each entity from the knowledge graph and define different matching rules based on the attribute type. For discrete and deterministic attribute values, compare whether the attribute values ​​are the same. That is, calculate the ratio of the number of matching attributes to the total number of attributes to obtain the discrete matching degree. For continuous numerical attributes, use the difference to calculate the continuous matching degree. Set different weights for discrete matching and continuous matching, and substitute them into the weighted calculation to obtain the matching degree of entity attributes. ; By constructing the path in the knowledge graph, using breadth-first search to calculate the shortest path, starting from the starting entity node, expanding outward layer by layer until the target entity node is found, the path length is the number of relationship levels ; By distinguishing direct interactions from indirect interactions, and recording each time entities interact through connection relationships, it is recorded as a direct interaction. By finding all the paths that the current entity passes through, the number of paths is calculated. On each path, each time it passes through the current entity, it is recorded as an indirect interaction. The direct interactions and indirect interactions are accumulated and calculated to get the number of entity interactions. ; The main semantic similarity, entity attribute matching degree, relationship connection level and entity interaction times are substituted into the logistic regression formula to calculate the data association coefficient.

3. The data anomaly detection method based on knowledge graph according to claim 2 is characterized in that: Calculate the data correlation coefficient corresponding to all other related entities; After obtaining the data correlation coefficients corresponding to all other entities, the data correlation coefficients are compared and analyzed with the iterative correlation thresholds; If the data association coefficient is greater than or equal to the association threshold, the other entities corresponding to the data association coefficient are marked as associated entities, and an association signal is generated; If the data association coefficient is less than the association threshold, the other entities corresponding to the data association coefficient are marked as screened entities, and a screening signal is generated; The correlation coefficient of the data marked as associated entities is counted as the number of candidate connections; The correlation coefficient of the data marked as the screened entity is counted and the quantity is determined, and compared with the abnormal threshold; If the number of data association coefficients marked as screened entities is greater than or equal to the anomaly threshold, a data anomaly signal is generated. After the screened entities are deleted, the adjustment connection rules are enabled. Otherwise, a data normal signal is generated.

4. The data anomaly detection method based on knowledge graph according to claim 1 is characterized in that: If the number of standard connections is the same as the number of candidate connections, the number of candidate connections is directly used as the standard number of connections to complete the adjustment of the individual connection relationship; If there is a difference between the standard number of connections and the number of alternative connections, a connection number adjustment plan is generated. The standard number of connections is subtracted from the number of alternative connections to obtain the difference number of connections. The set of data association degree coefficients in the number of alternative connections is deleted from small to large so that the number of alternative connections is the same as the standard number of connections. The individual connection relationships are connected according to the adjusted number of alternative connections.

5. The data anomaly detection method based on knowledge graph according to claim 4 is characterized in that: Through the individual connection relationship, the interaction timestamp sequence with each directly or indirectly connected entity is extracted. The standard deviation of the timestamp sequence is calculated, and a normalization formula is introduced to map the time correlation to the interval [0, 1] to obtain the time correlation degree. Count the number of direct and indirect connections of an individual, calculate the ratio of the total number of connections of all entities to the number of all entities in the knowledge graph to obtain the average number of connections of entities in the knowledge graph, and calculate the ratio of the number of connections of an individual to the average number of connections of entities in the knowledge graph to obtain data sparsity.

6. The data anomaly detection method based on knowledge graph according to claim 5 is characterized in that: The temporal correlation and data sparsity are defined as input variables and divided into different fuzzy sets respectively; The weighted result of the data association coefficient calculation is defined as the output variable and divided into fuzzy sets; Formulate fuzzy rules to describe the influence of time correlation and data sparsity on the weight calculation results of data correlation coefficient; Fuzzy reasoning is performed based on fuzzy rules to determine the data association degree coefficient and calculate the weight result.

Images