Data quality assessment method and system

Abstract

The invention provides a data quality evaluation method and system, and relates to the technical field of data quality evaluation.The data quality evaluation method is used for coupling type data quality evaluation of knowledge graph construction and specifically comprises the following steps that an entity historical version set of a knowledge graph and attribute time sequence data of the entity historical version set are obtained, calculating a redundancy index and a timeliness entropy change index of each entity, and calculating a coupling quality index based on the coupling coefficient; constructing a global quality evaluation matrix; and outputting a quality evaluation result and a cleaning strategy according to the distribution characteristics of the singular values of the global quality evaluation matrix. According to the method, the accuracy, adaptability and operability of knowledge graph data quality evaluation are remarkably improved, and an effective technical means is provided for constructing and maintaining a high-quality, low-redundancy and high-timeliness knowledge graph.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality assessment, and in particular to a data quality assessment method and system. Background Art

[0002] Currently, in dynamically updated knowledge graphs, highly time-sensitive entities need to be updated frequently to ensure accuracy. However, frequent updates will lead to the accumulation of redundant entities (for example, residual historical versions of entities). In existing related technologies, redundancy or timeliness are generally evaluated independently, and the contradiction caused by the coupling of the two is not resolved: excessive cleaning of redundant entities may mistakenly delete the historical valid versions of highly time-sensitive entities, while retaining redundant entities will lead to distortion of timeliness evaluation. Summary of the Invention

[0003] In order to solve the technical problems in the related art, the present invention provides a data quality assessment method and system.

[0004] In order to achieve the above object, the technical solution adopted by the present invention includes: According to a first aspect of the present invention, a data quality assessment method is provided for coupled data quality assessment of knowledge graph construction, comprising the following steps: Step S1: Get the entity history version set of the knowledge graph and its attribute time series data, the entity attributes of the knowledge graph change over time and have version redundancy problems; Step S2: Calculate the redundancy index of each entity and time-sensitive entropy change index , wherein the redundancy index and the time-sensitive entropy change index The calculation formulas are: Where, The total number of historical versions of the entity, which is used to reflect the update frequency of the entity. is the i-th historical version, For the latest version, is the probability of using the i-th historical version in the query log, It is the information entropy calculation item, which is used to measure the chaos of version distribution. is the attenuation factor used to control the intensity of the similarity effect, is the semantic similarity function based on BERT; is the steepness coefficient of the curve, which is used to control the mutation threshold of time-effect attenuation. is the time difference, that is, the difference between the current time and the last updated time, is the domain-sensitive threshold, corresponding to the validity period of data in different fields, is the entity information entropy, is the absolute value of the rate of change of entity information entropy; Step S3: Calculate the coupling quality index based on the coupling coefficient β : Where, is the domain-adaptive diagonal matrix; Step S4: Constructing a global quality assessment matrix : Where, , is the singular vector matrix, is a diagonal matrix of singular values, is a singular value; Step S5: According to the singular value The distribution characteristics of the output quality assessment results and cleaning strategies.

[0005] Optionally, the attenuation factor and the steepness coefficient of the curve The setting rules are: Where, is the average similarity of the entity's historical versions, The standard deviation of the interval between entity updates.

[0006] Optionally, the dynamic coupling coefficient β is calculated by the following steps: First select a neighborhood point set in the entity vector space ,in, For domain entities, is the set of neighborhood entities, is the Euclidean distance in entity embedding space, is the neighborhood distance threshold; Then calculate the mixed difference: ,in, , is the redundancy index of the current entity and the neighboring entity, , is the timeliness index of the current entity and the neighboring entity, To prevent division by zero constant; Calculate the indicator difference: , Calculate the coupling coefficient: , where is a smoothing constant to prevent the denominator from being zero.

[0007] Optionally, in step S3, the domain adaptive diagonal matrix The generation method is: Where, , are the matrix diagonal weights, is the redundancy index gradient norm, and , To prevent division by zero constant, is the gradient norm of the timeliness index, and .

[0008] Optionally, in step S4, the singular value The calculation method is: Where, is the coupling quality index of the i-th entity, is the spatial average absolute gradient, and , To prevent division by zero constant.

[0009] Optionally, in step S5, the generation rule of the cleaning strategy includes: when is less than a first preset threshold and When the value is greater than a second preset threshold, the execution entity is forced to update; when is greater than a third preset threshold and When it is less than a fourth preset threshold, historical version merging is performed.

[0010] According to a second aspect of the present invention, there is further provided a data quality assessment system for executing the data quality assessment method according to any one of the technical solutions in the first aspect of the present invention, the data quality assessment system comprising: Redundancy analysis module, used to calculate entity semantic similarity and information entropy; Time monitoring module for dynamic tracking and and change; Coupling module for executing and Real-time calculation; Matrix decomposer for Perform SVD decomposition to extract singular values.

[0011] Optionally, the data quality assessment system further includes: The self-learning feedback module is configured to adjust the query hit rate of the cleaned entity and , and when Parameter recalibration is triggered when is the singular value drift threshold.

[0012] According to the third aspect of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data quality assessment method described in any technical solution in the first aspect of the present invention can be implemented.

[0013] According to the fourth aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data quality assessment method described in any one of the technical solutions in the first aspect of the present invention can be implemented.

[0014] Beneficial effects: 1. Through the above technical solution, first, existing related technologies usually evaluate redundancy or timeliness in isolation, which may lead to one-sided evaluation results (for example, excessive cleaning of valid historical versions or retention of too many redundant versions). However, the method of the present invention uses the designed coupling quality index to combine the redundancy index with the timeliness index. and time-sensitive entropy change index By combining them dynamically and nonlinearly, when evaluating the quality of an entity, its redundancy and timeliness status can be considered simultaneously and interrelatedly, thereby realizing a dynamic coupling evaluation of redundancy and timeliness.

[0015] Second, the method of the present invention provides a more refined and accurate quality quantification. Specifically, the coupling quality index designed by the method of the present invention is It is calculated based on information entropy, semantic similarity, logical function, norm, dynamic parameters, etc., and can more accurately capture the complexity of entity version evolution and domain characteristics (for example, the validity period of data in different fields ), which can better reflect the true quality status of an entity in a specific context than simple threshold judgment or weighted summation of independent indicators.

[0016] Third, the method of the present invention supports quality pattern recognition from a global perspective. Specifically, the method of the present invention uses the coupling quality index of each entity as the Organized into a global quality assessment matrix ( ) and perform singular value decomposition (SVD), which can reveal the overall distribution characteristics and potential correlation patterns of entity quality issues in the knowledge graph. ) can indicate the severity and concentration of global quality issues, as well as potential structural problems (e.g., widespread redundancy or timeliness issues in a subgraph), which is a perspective that single-entity evaluation cannot provide.

[0017] Fourth, the method of the present invention can guide a more intelligent and targeted cleaning strategy. Specifically, the method of the present invention is based on (single entity mass) and Distribution (global quality mode), the output cleaning strategy can no longer be a simple one-size-fits-all (for example, delete all old versions) but can be based on the results of coupled evaluation (for example, as described below) Low and fast-changing timeliness, or high redundancy and low timeliness) to trigger the most appropriate and least side-effect specific cleaning operation, which can effectively solve the problem of timeliness distortion caused by accidental deletion of valid historical versions or retention of redundancy mentioned in the background technology.

[0018] Fifth, the method of the present invention can effectively enhance the adaptability and robustness of the evaluation system. Specifically, the method of the present invention introduces multiple dynamic calculation or domain adaptation elements, enabling the evaluation model to automatically adjust parameter weights based on the historical status of specific entities, the neighborhood environment, and the characteristics of the domain, effectively ensuring adaptability and robustness in different scenarios.

[0019] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] in: Figure 1 is a schematic flow chart of steps of a data quality assessment method provided by an exemplary embodiment of the present invention; Figure 2 Schematic diagram of a three-layer coupling evaluation model provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0024] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices. It should also be noted that in the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0025] In order to facilitate relevant technical personnel to have a clearer and more accurate understanding of the technical solutions of the present invention, the technical problems existing in the existing related technologies are first described in more detail below.

[0026] In existing technologies, time-sensitive entities in dynamically updated knowledge graphs require frequent updates to ensure accuracy. However, these frequent updates can lead to the accumulation of redundant entities. These technologies typically evaluate redundancy or timeliness independently, without resolving the conflict caused by the coupling of the two: the dynamic coupling conflict between redundancy and timeliness.

[0027] From the perspective of the split evaluation dimension, in existing related technologies, redundancy evaluation (for example, version similarity) and timeliness evaluation (for example, update frequency) are performed independently. In this case, entities with high update frequency may be misjudged as "redundant", while entities that have not been updated for a long time may have inflated timeliness scores.

[0028] From the perspective of dynamic coupling imbalance, frequently updated entities will inevitably produce historical versions, but existing technologies cannot quantify "how many historical versions are optimal to retain". In this way, excessive cleaning will lead to the loss of high-value historical information, while retaining too many versions will drag down real-time query performance.

[0029] From the perspective of policy conflicts, in existing related technologies, redundant cleaning strategies and time optimization strategies constrain each other. For example, forcibly deleting old versions may destroy the time evolution chain, while retaining redundant versions may cause response delays for time-sensitive entities.

[0030] To summarize, the fundamental pain point of existing technologies lies in the lack of a coupled evaluation mechanism, that is, the inability to achieve a dynamic trade-off between "preserving historical integrity" and "ensuring real-time accuracy." In particular, they have poor adaptability to domain-sensitive entities (for example, medical guidelines need to retain historical versions, while news events require the rapid elimination of old versions).

[0031] Based on this, the present invention provides a brand-new solution. The technical idea of ​​the present invention is to establish a redundancy-time coupling dynamic model and achieve dynamic balance through fourth-order control. The fourth-order control includes an indicator layer (the redundancy indicator introduces semantic decay, and the timeliness entropy change indicator is embedded in the domain time window), a coupling layer (coupling coefficient and domain adaptive diagonal matrix realize neighborhood adaptive weight distribution), a decision layer (SVD decomposition separates local / global quality problems) and a feedback layer (cyclic optimization parameters according to the query hit rate after cleaning). The present invention transforms traditional static evaluation into a dynamic system control problem, and achieves a breakthrough in the quantitative trade-off between "historical version value" and "real-time accuracy" in the knowledge graph. It is particularly suitable for high-precision and high-tech equipment, strategic equipment, medical care, finance and other highly sensitive fields.

[0032] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 and Figure 2 As shown, according to a first aspect of the present invention, a data quality assessment method is provided for coupled data quality assessment of knowledge graph construction, comprising the following steps: Step S1: Get the entity history version set of the knowledge graph And its attribute time series data, the entity attributes of the knowledge graph change over time and there is version redundancy problem; Step S2: Calculate the redundancy index of each entity and time-sensitive entropy change index , where the redundancy index and time-sensitive entropy change index The calculation formulas are: Where, The total number of historical versions of the entity, which is used to reflect the update frequency of the entity. is the i-th historical version, For the latest version, is the probability of using the i-th historical version in the query log, It is the information entropy calculation item, which is used to measure the chaos of version distribution. is the attenuation factor used to control the intensity of the similarity effect, is the semantic similarity function based on BERT; is the steepness coefficient of the curve, which is used to control the mutation threshold of time-effect attenuation. is the time difference, that is, the difference between the current time and the last updated time, is the domain-sensitive threshold, corresponding to the validity period of data in different fields, is the entity information entropy, is the absolute value of the rate of change of entity information entropy; Step S3: Calculate the coupling quality index based on the coupling coefficient β : Where, is the domain-adaptive diagonal matrix; Step S4: Constructing a global quality assessment matrix : Where, , is the singular vector matrix, is a diagonal matrix of singular values, is a singular value; Step S5: According to the singular value The distribution characteristics of the output quality assessment results and cleaning strategies.

[0034] Through the above technical solution, first, the existing related technologies usually evaluate redundancy or timeliness in isolation, which will lead to a one-sided evaluation result (for example, excessive cleaning of valid historical versions or retaining too many redundant versions). The method of the present invention uses the designed coupling quality index to combine the redundancy index and time-sensitive entropy change index By combining them dynamically and nonlinearly, when evaluating the quality of an entity, its redundancy and timeliness status can be considered simultaneously and interrelatedly, thereby realizing a dynamic coupling evaluation of redundancy and timeliness.

[0035] Second, the method of the present invention provides a more refined and accurate quality quantification. Specifically, the coupling quality index designed by the method of the present invention is It is calculated based on information entropy, semantic similarity, logical function, norm, dynamic parameters, etc., and can more accurately capture the complexity of entity version evolution and domain characteristics (for example, the validity period of data in different fields ), which can better reflect the true quality status of an entity in a specific context than simple threshold judgment or weighted summation of independent indicators.

[0036] Third, the method of the present invention supports quality pattern recognition from a global perspective. Specifically, the method of the present invention uses the coupling quality index of each entity as the Organized into a global quality assessment matrix ( ) and perform singular value decomposition (SVD), which can reveal the overall distribution characteristics and potential correlation patterns of entity quality issues in the knowledge graph. ) can indicate the severity and concentration of global quality issues, as well as potential structural problems (e.g., widespread redundancy or timeliness issues in a subgraph), which is a perspective that single-entity evaluation cannot provide.

[0037] Fourth, the method of the present invention can guide a more intelligent and targeted cleaning strategy. Specifically, the method of the present invention is based on (single entity mass) and Distribution (global quality mode), the output cleaning strategy can no longer be a simple one-size-fits-all (for example, delete all old versions) but can be based on the results of coupled evaluation (for example, as described below) Low and fast-changing timeliness, or high redundancy and low timeliness) to trigger the most appropriate and least side-effect specific cleaning operation, which can effectively solve the problem of timeliness distortion caused by accidental deletion of valid historical versions or retention of redundancy mentioned in the background technology.

[0038] Fifth, the method of the present invention can effectively enhance the adaptability and robustness of the evaluation system. Specifically, the method of the present invention introduces multiple dynamic calculation or domain adaptation elements, enabling the evaluation model to automatically adjust parameter weights based on the historical status of specific entities, the neighborhood environment, and the characteristics of the domain, effectively ensuring adaptability and robustness in different scenarios.

[0039] In this embodiment, it should be noted that, first, the redundancy index It is used to quantify the redundancy of historical versions of an entity. It not only considers the number of versions, but also emphasizes the uncertainty of version distribution (i.e., chaos) and the semantic difference between the previous version and the current version.

[0040] Among them, first, information entropy ( ) is the core of entropy in information theory. (The probability of using historical versions in query logs) reflects the frequency distribution of different versions being accessed. A larger entropy value indicates a more even and chaotic version distribution (i.e., no one version is clearly dominant), suggesting the possibility of redundant or invalid versions. A smaller entropy value indicates that a certain version is frequently used (perhaps the current version or a valid historical version), and the redundancy problem may be less severe. Secondly, semantic similarity weighting ( ) is to use advanced models such as BERT to calculate historical versions With the current version The semantic similarity of , the information entropy is weighted by an exponential function: the historical version with high semantic similarity (which may still be valuable or close to the current version) will have its corresponding entropy item weight amplified ( ), making these versions have a greater negative impact on redundancy calculations ( At the same time, the entropy weight of historical versions with low semantic similarity (which may represent an important historical state) will be reduced ( ), reducing its negative impact on redundancy calculation ( Relatively less negative, less redundant), thus avoiding misjudging valid historical versions with large semantic differences as redundant. Finally, the normalization term ( ) Normalize the entropy value to the range of 0-1; the attenuation factor Control the strength of the influence of semantic similarity, The larger the value, the more significant the impact of similarity difference on the weight.

[0041] Second, the time-sensitive entropy change index It is used to quantify the timeliness of entity information, not only considering the time interval ( ), which emphasizes the rate at which information content itself changes over time (the rate of change of information entropy). ).

[0042] Among them, first, the logistic function ( ) is a S-type function used to calculate the time difference (Current time - last update time) for nonlinear attenuation. Specifically, when Far less than the domain sensitivity threshold (data validity period), the function value is close to 1 (good timeliness). Approaching or exceeding the domain sensitivity threshold When , the function value drops sharply to close to 0 (poor timeliness). Control the steepness of this decay process (mutation threshold). Secondly, the information entropy change rate ( )middle, is the entity information entropy (reflecting the richness and uncertainty of entity attribute information), is the absolute value of its rate of change. For this term, even if Large (low logic function value), if the information entropy changes very slowly ( Small), indicating that the entity information is relatively stable, and the negative impact of its "outdated" may not be so serious; even if ∆t is small (the value of the logic function is high), if the information entropy is changing rapidly ( large), indicating that the entity information is in a period of drastic changes and requires close attention to its timeliness. Finally, overall, the timeliness entropy change index It is the product of the logistic function (time decay factor) and the rate of change of information entropy (information content dynamic factor). This means that the final timeliness score is affected by both the passage of time and the actual rate of change of information content.

[0043] In other words, the time-sensitive entropy change index By simulating the time decay effect through logical functions and multiplying it by the information entropy change rate, a comprehensive evaluation of the "freshness" and "information dynamics" of entity information is achieved, avoiding the one-sidedness of judging timeliness based solely on update time.

[0044] Third, the coupling quality index It is used to dynamically and nonlinearly fuse redundancy indicators and time-sensitive entropy change index , forming a single quality indicator.

[0045] Specifically, first, the linear weighted part ( ) is the most direct coupling method, Coupling coefficient) control and The relative importance of the product terms. This means that high redundancy and low timeliness will significantly lower the quality score. Secondly, the nonlinear norm part ( ) is one of the key innovative parts of the present invention, which is used to solve the limitations of simple linear addition. The nonlinear norm part includes a vectorization part ( and Composition vector ), domain adaptive weighted part (diagonal matrix Weighting of redundancy and timeliness indicators), L2 norm part ( , which calculates the weighted vector The Euclidean norm of and The "comprehensive strength" or "comprehensive deviation" after domain adaptive weighting. An abnormally high one or both indicators will lead to an increase in the norm value) and the coupling coefficient Rebalancing The control norm term is at the end The proportion in . It is also dynamically calculated based on the current entity and its neighboring entities. and relative differences (i.e., mixed differences) and absolute differences).

[0046] In general, the coupling quality index The calculation formula cleverly combines the product term, weighted norm term, and dynamic coupling coefficient. The product term directly reflects the mutual constraint relationship between redundancy and timeliness, and the weighted norm term reflects the comprehensive strength or abnormality of the indicator in the local neighborhood environment through domain-adaptive weight and norm calculation. Dynamic coupling coefficient According to the difference in indicators between the entity and the neighborhood, the balance between the two coupling modes is dynamically adjusted. It can flexibly adapt to the specific environment (distribution characteristics of neighborhood indicators) of different entities and realize dynamic coupling evaluation in a true sense.

[0047] Fourth, global quality assessment matrix And SVD decomposition is used to analyze the quality structure of the knowledge graph from a system level and identify global problem patterns and potential associations.

[0048] Specifically, for matrix construction, all entities in the knowledge graph are Organized into a matrix , singular value decomposition (SVD) is used to obtain the singular value and its distribution characteristics, where the singular value On the diagonal matrix In descending order, each Represents The "energy" or "importance" of the matrix in the corresponding principal component direction. The distribution characteristics of the singular values ​​(for example, if the first few Very large and significant, indicating that global quality problems are concentrated in a few major patterns, for example, the entire graph has some type of redundancy-timeliness problem, or a large subgraph has serious problems; for example, if The slow decay indicates that the quality problems are relatively dispersed and there is no particularly prominent single pattern) which can reflect the evolution of the global quality status.

[0049] Overall, the SVD analysis The singular value spectrum of the matrix can go beyond the evaluation of a single entity, revealing the structural characteristics of the overall quality status of the knowledge graph and providing a basis for macro-decision-making (such as resource allocation and determination of key cleaning areas).

[0050] The method of the present invention significantly improves the accuracy, adaptability and operability of knowledge graph data quality assessment, and provides an effective technical means for constructing and maintaining high-quality, low-redundancy and high-efficiency knowledge graphs.

[0051] In one embodiment of the present invention, the attenuation factor of the present invention is and the steepness coefficient of the curve The setting rules can be: Where, is the average similarity of the entity's historical versions, The standard deviation of the interval between entity updates.

[0052] In this embodiment, by ) and the dynamic parameter rules of the update behavior, so that the above technical solution and With adaptive capabilities, it solves the coupling contradiction between redundancy and timeliness from the source of evaluation, and provides a basis for the subsequent coupling quality index. Lay the foundation for accuracy.

[0053] For example, In terms of adaptive adjustment, when entities are highly similar (easy to mistakenly delete valid historical versions), reduce → Reduce the penalty weight of similar versions → Protect valid historical versions (avoid excessive cleaning of redundancies and mistakenly delete historical valid versions of high-age entities). When entities have low similarity (easy to leave redundant versions), increase → Increase the penalty weight for similar versions → Accurately clean up redundancies (avoiding distortion of timeliness evaluation caused by retaining redundant entities).

[0054] For example, In terms of domain and behavior adaptation, when the entity has high volatility + short validity period (time sensitivity), increase → Determine if expired → Avoid redundant accumulation leading to distortion of timeliness (avoid distortion of timeliness evaluation due to retention of redundant entities); when the entity has low volatility + long validity period (redundancy sensitive), reduce → Allow reasonable expiration → Avoid accidental deletion of historical versions due to forced updates (avoid accidental deletion of historical valid versions of high-age entities due to excessive cleaning of redundancies).

[0055] In one embodiment of the present invention, the dynamic coupling coefficient β of the present invention can be calculated by the following steps: First select a neighborhood point set in the entity vector space ,in, For domain entities, is the set of neighborhood entities, is the Euclidean distance in entity embedding space, is the neighborhood distance threshold; Then calculate the mixed difference: ,in, , is the redundancy index of the current entity and the neighboring entity, , is the timeliness index of the current entity and the neighboring entity, To prevent division by zero constant; Calculate the indicator difference: , Calculate the coupling coefficient: , where is a smoothing constant to prevent the denominator from being zero.

[0056] In this embodiment, first, this embodiment is based on the neighborhood relationship of entities in the vector space (rather than static parameters) calculation , so that the coupling mechanism has spatial context awareness and can achieve dynamic neighborhood awareness. Second, this implementation method is based on the redundancy in the field and timeliness The relative difference intensity of the dynamic allocation product term ( ) and the norm term ( ) weights, which can achieve adaptive weight distribution. Third, this implementation can automatically identify whether the main contradiction in the local environment of the current entity is redundancy or timeliness, and tilt the evaluation weight. Fourth, this implementation uses mixed difference, smoothing constant ( 、 ) and neighborhood averaging operations to suppress local fluctuation interference.

[0057] Specifically, by selecting a neighborhood point set in the entity vector space , we can locate neighboring entities with similar semantics / structure to the current entity (such as entities in the same industry or the same event type).

[0058] At the same time, by calculating the mixed difference To quantify the relative proportion of redundancy difference and time difference within the neighborhood (where the numerator Indicates the absolute difference in redundancy between the current entity and its neighbors, the denominator Indicates the absolute difference in timeliness between the current entity and its neighbors, the ratio Indicates the redundancy difference corresponding to the unit time difference. If the ratio is greater than 1, the redundancy difference is dominant and should be increased. , focusing on the product term , severely punish double low, if the ratio is less than 1, the time difference is dominant, and should be reduced , focusing on the norm term , detect anomalies. In this process, the key point is to use relative differences (rather than absolute values) to identify local major contradictions and avoid deviations caused by different indicator dimensions.

[0059] In addition, the absolute fluctuation intensity of redundancy and timeliness in the neighborhood is measured by calculating the mean of index difference. Provides a normalized baseline to prevent The absolute difference is small and invalid. The principle is: when When it is large, the neighborhood redundancy problem becomes prominent or the distribution diverges. When the problem is large, the timeliness issues in the field will be prominent or distributed divergently.

[0060] Finally, the mixed difference Normalized to available weight coefficients Among them, if the absolute difference of the neighborhood is small ( , If the absolute difference in the neighborhood is large (such as a drastic fluctuation in a certain indicator), the denominator is large, and β is reduced, which strengthens the role of the norm term (due to local instability, outlier detection is required).

[0061] In general, this implementation enables the coupling quality assessment to be adaptive to the environment through the dynamic β calculation driven by the neighborhood. A product penalty is applied to prevent redundant accumulation. In time-critical areas (e.g., real-time financial data), β is reduced to enable norm-term anomaly detection and prevent time-sensitive data distortion. In stable areas (e.g., gender attribute data), β is increased to strictly couple the two indicators and avoid accidental deletion of valid versions. This effectively overcomes the flaw in existing technologies: independent evaluation cannot handle coupled contradictions.

[0062] In one embodiment of the present invention, in step S3 of the present invention, the domain adaptive diagonal matrix The generation method can be: Where, , are the matrix diagonal weights, is the redundancy index gradient norm, and , To prevent division by zero constant, is the gradient norm of the timeliness index, and .

[0063] In this embodiment, first, this embodiment is based on the redundancy index within the neighborhood and timeliness indicators The local change intensity of the indicator is used to adaptively allocate the weights of the two in the norm term, which can effectively realize dynamic weight allocation. Second, this implementation method can automatically amplify the weight of the indicator in the area where the indicator changes drastically, so that the norm term can capture the main contradiction (redundant anomaly or time anomaly) more sensitively. Third, this implementation method uses the indicator gradient norm and Quantifying local volatility can achieve unsupervised domain knowledge injection. Fourth, this implementation method uses the denominator Achieve weight normalization to avoid calculation deviation caused by differences in indicator dimensions.

[0064] In this embodiment, it should be noted that, first, for As for the structural design of the matrix, it adopts the form of a diagonal matrix, which can ensure and In the norm calculation, they are weighted independently and do not interfere with each other. Control redundancy index The contribution weight in the norm, Control timeliness indicators Contribution weights in the norm, diagonal zeros can be cut off and The cross-influence of the indicators focuses on the volatility of a single indicator.

[0065] Second, for weight calculation, the redundancy index gradient norm The physical meaning of the neighborhood redundancy is Fluctuation intensity (Euclidean distance and); gradient norm of timeliness index The physical meaning of the neighborhood timeliness The intensity of fluctuations.

[0066] The calculation principle is: for molecules (based on For example), The larger → the more significant the difference in redundancy within the neighborhood → the current regional redundancy problem is the main contradiction → it should be given a higher weight , so that the norm term focuses on detecting redundant anomalies. (For example, if a financial entity has The difference is huge. ,and Relatively stable, ,but , → The 96% weight of the norm term focuses on redundancy) For the denominator, Achieve weight normalization to ensure , while eliminating the and Weight bias caused by absolute numerical differences.

[0067] In one embodiment of the present invention, in step S4 of the present invention, the singular values The calculation method is: Where, is the coupling quality index of the i-th entity, is the spatial average absolute gradient, and , To prevent division by zero constant.

[0068] In this embodiment, first, this embodiment converts the singular values ​​of the mathematical level Transformed into a quality stability indicator with clear business meaning ( and local consistency). Second, this implementation uses the spatial average absolute gradient Capturing the consistency of neighborhood redundancy and suppressing the interference of local noise on global evaluation. Third, this implementation distinguishes the quality of the entity itself and its stability in the neighborhood ( ), to avoid misjudgment of high-quality entities in high-volatility areas. Fourth, the denominator Ensure that the singular value dimensions are uniform and give protective amplification to low gradient areas.

[0069] In this embodiment, it should be noted that for the singular values In terms of the calculation method, its numerator Reflects the quality of the entity itself, with higher values ​​indicating better quality (low redundancy and high timeliness). ) measures the redundancy stability of the neighborhood where the entity is located; if the neighborhood redundancy features are highly consistent ( →0) → denominator is taken → Zoom in (The physical meaning is: the local environment is stable and the entity quality is highly reliable.) The neighborhood redundancy characteristics are significantly different ( Large) → denominator increases → suppression (The physical meaning is: the local environment is turbulent and the uncertainty of the entity quality assessment is high.) The final physical meaning is: ,high Entity = high quality and stable environment, low Entity = low quality or turbulent environment (even if High also needs to be evaluated with caution).

[0070] In general, this embodiment The design gives high quality entities in stable areas , making it the core backbone of the knowledge graph, giving low , marking it as a cleaning priority, giving medium , avoid accidental injury, so that the global quality assessment matrix can be ensured The singular value spectrum can accurately reflect the substantial quality structure of the knowledge graph and provide a reliable basis for subsequent cleaning strategies.

[0071] In one embodiment of the present invention, in step S5 of the present invention, the generation rules of the cleaning strategy may include: when is less than a first preset threshold and When the value is greater than a second preset threshold, the execution entity is forced to update; when is greater than a third preset threshold and When it is less than a fourth preset threshold, historical version merging is performed.

[0072] In this embodiment, first, accurate operation triggering can be achieved. Specifically, based on the coupling evaluation results ( 、 and ) and the aging rate of change ( ), triggering two core cleaning operations differently. Second, forced updates can avoid the problem of "retaining redundancy and causing timeliness distortion," and version merges can avoid the problem of "excessive cleaning and inadvertently deleting valid history." Third, multi-condition joint decision-making can avoid misjudgments based on a single indicator.

[0073] In this embodiment, it can be understood that the first preset threshold, the second preset threshold, the third preset threshold and the fourth preset threshold can be adjusted and modified according to actual conditions and needs, and the present invention does not specifically limit this.

[0074] According to the second aspect of the present invention, a data quality assessment system is provided for executing the data quality assessment method according to any one of the technical solutions in the first aspect of the present invention. The data quality assessment system includes a redundancy analysis module, a timeliness monitoring module, a coupling module, and a matrix decomposer. The redundancy analysis module is used to calculate the semantic similarity and information entropy of entities; the timeliness monitoring module is used to dynamically track and and Changes; coupling modules are used to perform and Real-time calculation of matrix decomposer; Perform SVD decomposition to extract singular values.

[0075] In this embodiment, it is understood that for the redundancy analysis module, it can calculate the semantic similarity through the built-in BERT model and calculate the semantic similarity through the distributed query log analyzer. , using GPU to accelerate information entropy calculation; for the timeliness monitoring module, it can store entity change history through the time series database and update it in real time through the streaming computing engine For coupled modules, it can have a neighborhood searcher (e.g., based on Faiss vector indexing) that is computed by a gradient coprocessor , accelerate matrix operations through FPGA; for the matrix decomposer, it can support sparse matrix decomposition through the SVD chip and be used to output singular values ​​and distribution characteristics.

[0076] In one embodiment of the present invention, the data quality assessment system of the present invention may further include a self-learning feedback module. The self-learning feedback module is configured to adjust the query hit rate of the cleaned entity according to the query hit rate of the cleaned entity. and , and when Parameter recalibration is triggered when is the singular value drift threshold.

[0077] In this embodiment, first, this embodiment dynamically adjusts the (Semantic Decay), (time-dependent steepness), instead of manual parameter adjustment, it can achieve parameter self-optimization. Second, this implementation method uses singular value drift ( ) triggers global recalibration, enabling system-level anomaly detection. Third, this implementation feeds back the operation results (query hit rate) to the front-end evaluation stage, forming a closed loop for decision optimization.

[0078] In this embodiment, for and The adjustment logic (or rule) can be: if the hit rate increases after cleaning → increase (Enhanced redundant cleaning) Reduce (Relax the time limit); if the hit rate drops after cleaning → reduce (Protect historical versions) Increase (Tighten timeliness requirements).

[0079] also, is the singular value drift threshold, which can be set by historical fluctuation statistics or based on experience, and is not specifically limited in the present invention.

[0080] According to the third aspect of the present invention, there is also provided a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it can implement the steps of the data quality assessment method in any technical solution in the first aspect of the present invention.

[0081] In this embodiment, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an application-specific integrated circuit (ASIC). In embodiments of the present invention, computer-readable storage media may be any tangible media that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0082] According to the fourth aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data quality assessment method in any one of the technical solutions in the first aspect of the present invention can be implemented.

[0083] It is understood that in this embodiment, the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and further, the memory may include a combination of the aforementioned types of memory. The present invention is not specifically limited to this.

[0084] Similarly, a processor may implement or execute the various exemplary logical steps described in conjunction with the present disclosure. The processor may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may implement or execute the various exemplary logical steps described in conjunction with the present disclosure. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0085] The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data quality assessment method, characterized in that: The coupled data quality assessment for knowledge graph construction includes the following steps: Step S1: Get the entity history version set of the knowledge graph and its attribute time series data, the entity attributes of the knowledge graph change over time and have version redundancy problems; Step S2: Calculate the redundancy index of each entity and time-sensitive entropy change index , wherein the redundancy index and the time-sensitive entropy change index The calculation formulas are: Where, The total number of historical versions of the entity, which is used to reflect the update frequency of the entity. is the i-th historical version, For the latest version, is the probability of using the i-th historical version in the query log, It is the information entropy calculation item, which is used to measure the chaos of version distribution. is the attenuation factor used to control the intensity of the similarity effect, is the semantic similarity function based on BERT; is the steepness coefficient of the curve, which is used to control the mutation threshold of time-effect attenuation. is the time difference, that is, the difference between the current time and the last updated time, is the domain-sensitive threshold, corresponding to the validity period of data in different fields, is the entity information entropy, is the absolute value of the rate of change of entity information entropy; Step S3: Calculate the coupling quality index based on the coupling coefficient β : Where, is the domain-adaptive diagonal matrix; Step S4: Constructing a global quality assessment matrix : Where, , is the singular vector matrix, is the singular value diagonal matrix, is a singular value; Step S5: According to the singular value The distribution characteristics of the output quality assessment results and cleaning strategies.

2. The data quality assessment method according to claim 1, characterized in that: The attenuation factor and the steepness coefficient of the curve The setting rules are: Where, is the average similarity of the entity's historical versions, The standard deviation of the interval between entity updates.

3. The data quality assessment method according to claim 1, characterized in that: The dynamic coupling coefficient β is calculated by the following steps: First select a neighborhood point set in the entity vector space ,in, For domain entities, is the set of neighborhood entities, is the Euclidean distance in entity embedding space, is the neighborhood distance threshold; Then calculate the mixed difference: ,in, , is the redundancy index of the current entity and the neighboring entity, , is the timeliness index of the current entity and the neighboring entity, To prevent division by zero constant; Calculate the indicator difference: , Calculate the coupling coefficient: , where is a smoothing constant to prevent the denominator from being zero.

4. The data quality assessment method according to claim 3, characterized in that: In step S3, the domain adaptive diagonal matrix The generation method is: Where, , are the matrix diagonal weights, is the redundancy index gradient norm, and , To prevent division by zero constant, is the gradient norm of the timeliness index, and .

5. The data quality assessment method according to claim 3, characterized in that: In step S4, the singular value The calculation method is: Where, is the coupling quality index of the i-th entity, is the spatial average absolute gradient, and , To prevent division by zero constant.

6. The data quality assessment method according to claim 1, characterized in that: In step S5, the generation rules of the cleaning strategy include: when is less than a first preset threshold and When the value is greater than a second preset threshold, the execution entity is forced to update; when is greater than a third preset threshold and When it is less than a fourth preset threshold, historical version merging is performed.

7. A data quality assessment system for executing the data quality assessment method according to any one of claims 1 to 6, characterized in that: The data quality assessment system comprises: Redundancy analysis module, used to calculate entity semantic similarity and information entropy; Time monitoring module for dynamic tracking and and change; Coupling module for executing and Real-time calculation; Matrix decomposer for Perform SVD decomposition to extract singular values.

8. The data quality assessment system according to claim 7, characterized in that: The data quality assessment system further includes: The self-learning feedback module is configured to adjust the query hit rate of the cleaned entity and , and when Parameter recalibration is triggered when is the singular value drift threshold.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data quality assessment method according to any one of claims 1 to 6 can be implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data quality assessment method according to any one of claims 1 to 6 can be implemented.

Images