A knowledge graph-based data intelligent labeling method

Abstract

This invention relates to the field of knowledge graph creation and management technology, and particularly to a data intelligent annotation method based on knowledge graphs. The method includes periodically updating the model, preprocessing the collected entity data, determining the status of the annotation results by comparing a comprehensive annotation quality index with a preset threshold, adjusting the standard entity extraction confidence level and entity verification scaling factor based on the determination results, and determining the external evidence fusion strength. This invention achieves unified processing and optimization of annotation quality analysis performance in intelligent annotation detection systems, reduces system energy consumption, and effectively improves data validity assurance and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph creation and management technology, and in particular to a data intelligent annotation method based on knowledge graphs. Background Technology

[0002] The management and utilization of knowledge graphs involve many fields, and high-quality knowledge graph annotation presents significant challenges for researchers. This leads to static work strategies, making it impossible for the system to optimize based on feedback, and compromising the reliability and stability of annotation quality. To avoid potential risks, quantify the detection quality of different annotation methods, optimize system technical routes, and place higher demands on the scientific rigor and comparability of the collected and adjusted results. Many technical solutions provide graph embedding techniques, which can promptly detect anomalies and significantly improve the overall quality of annotation results, becoming a necessary and effective means. Therefore, traditional graph embedding analysis techniques are currently one of the detection methods for intelligent annotation.

[0003] However, existing detection modes mostly adopt a unified and simple detection method, which fails to consider the multi-data modes of complex application scenarios. These modes involve multiple key parameters such as alignment confidence, recognition coverage, and annotation consistency. Existing technologies are mostly static, single-modal, and isolated detection methods. Most devices can only be used to transmit a large amount of invalid and low-quality map data, resulting in a waste of resources and an inability to systematically and dynamically capture the dynamic correlation between various parameters.

[0004] Meanwhile, existing analysis models have many shortcomings. Key parameters involved in the system often rely on empirically set or fixed standard values, lacking the ability to perceive the system's own state in real time, and failing to form a dynamic adjustment mechanism, thus reducing the efficiency of intelligent annotation and detection.

[0005] Chinese Patent Publication No. CN119322852A discloses a method and system for generating knowledge graph annotation models, including generating a knowledge graph annotation model based on data expressions and using multi-level labels to optimize the model and complete the training task.

[0006] Therefore, although the proposed solution can adjust labels in real time to complete model construction, it still has the following problems: 1. This solution only uses a single intelligent calculation method for data processing. When there are outliers in the collected data, it cannot intelligently optimize the calculation method, which leads to the inability to effectively collect data from each terminal, thereby reducing the annotation quality of the map.

[0007] 2. This scheme uses fixed detection instructions. When the labeled entities change, there will be delays or deviations in data collection, which will affect the working status of the subsequent state detection module and further reduce the labeling quality of the map. Summary of the Invention

[0008] To address this issue, the present invention provides a knowledge graph-based intelligent data annotation method to overcome the problem of low efficiency in data annotation result quality detection and processing caused by the inability to generate corresponding update instructions based on data state changes in existing technologies.

[0009] To achieve the above objectives, this invention provides a knowledge graph-based intelligent data annotation method, comprising: The user log terminal is connected to the corresponding data acquisition module to build a data transmission channel. After the data transmission channel is built, each user log terminal transmits the corresponding entity data. The log parameters generated by each user log terminal during operation are sent to the data processing module for data preprocessing. The entity data for a single user log terminal includes the number of entities in the knowledge graph, the number of entity alignments, the number of labeled entities, and the number of recalled entities. The detection system calculates a comprehensive annotation quality index for each of the received entity data. Based on the comprehensive annotation quality index of each user log terminal during the transmission process, it determines whether the data annotation results for each entity data in a single running cycle meet the standard. If it is determined that the data does not meet the standard, it generates corresponding processing instructions according to the determined reasons, including adjusting the standard entity extraction confidence if the reason is insufficient entity recognition coverage, or adjusting the entity verification scaling factor if the reason is insufficient entity alignment accuracy. The comprehensive annotation quality index is a weighted average of the average of annotation accuracy and annotation recall and annotation consistency. In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy. The first repeated detection condition is that after adjusting the confidence of standard entity extraction, the data annotation results for each entity data in a single running cycle are determined to be non-compliant with the standard based on the newly obtained comprehensive annotation quality index. Based on the obtained adjusted entity verification scaling factor, a re-detection is performed. If the standard is still not met, the corresponding processing method is selected, including adjusting the external data fusion strength. Continuous testing is performed after confirming that the data annotation results for each entity data in a single running cycle meet the standards.

[0010] Furthermore, the process of determining whether the data annotation results for each entity data in a single running cycle meet the standard includes: The ratio of correctly labeled entities to the total number of entities labeled by the system is denoted as the labeling accuracy rate. The ratio of correctly labeled entities to the total number of entities that should actually be labeled is denoted as the labeling recall rate. Calculate the harmonic mean of annotation precision and annotation recall, and record it as the measure value; The ratio of the number of entities in the intersection to the number of entities in the union of two independently labeled sets is denoted as label consistency. The normalized weighted average of the consistency between the measurement value and the annotation is calculated and denoted as the comprehensive annotation quality index. Compare the obtained comprehensive annotation quality index with the preset comprehensive annotation quality index; Based on the comparison results, determine whether the data annotation results for each entity data in a single running cycle meet the standards; If it is determined that each annotation result does not meet the standard, the reason for non-compliance is determined based on the mean confidence score of entity alignment, where the mean confidence score of entity alignment is the average of the ratios of the confidence scores of successfully linked entities to the total number of successfully linked entities. Continuous testing is performed after confirming that the data annotation results for each entity data in a single running cycle meet the standards.

[0011] Furthermore, the process of determining the reasons why the data annotation results for each entity data in a single running cycle do not meet the standard based on the mean entity alignment confidence score includes: Calculate the mean confidence score for entity alignment; The calculated mean confidence score of the entity alignment is compared with the preset mean confidence score of the entity alignment. Based on the comparison results, determine the reasons why the data annotation results for each entity data in a single running cycle do not meet the standards; If the cause is determined to be insufficient entity recognition coverage, the standard entity extraction confidence level is adjusted according to the entity confidence distribution parameter; wherein, the entity confidence distribution parameter is the ratio of the number of candidate entities accepted by the system to the total number of candidate entities, and the standard entity extraction confidence level is the acceptance threshold of the entity recognition confidence level; If the cause is determined to be insufficient entity alignment accuracy, the entity verification scaling factor is adjusted based on the graph information entropy; wherein, the graph information entropy is the weighted sum of the frequencies of each attribute, and the entity verification scaling factor is a function based on the graph information entropy.

[0012] Furthermore, the process of adjusting the standard entity extraction confidence level based on the entity confidence distribution parameter when the cause is determined to be insufficient entity recognition coverage includes: Calculate the entity confidence distribution parameters; The obtained entity confidence distribution parameters are compared with the preset entity confidence distribution parameters; The confidence level of the standard entity extraction is increased based on the comparison results, and the increase in the confidence level of the standard entity extraction is positively correlated with the entity confidence distribution parameter.

[0013] Furthermore, the process of adjusting the confidence level of the standard entity extraction also includes: Calculate the semantic similarity between a single candidate entity and its corresponding entity in the knowledge graph; The maximum value of the semantic similarity is taken as the best matching degree between the entity and the knowledge graph; The average ratio of the total number of matching scores to the total number of entities is denoted as the average knowledge graph semantic similarity. The obtained average semantic similarity of the knowledge graph is compared with the preset average semantic similarity of the knowledge graph. The confidence level of the adjusted standard entity extraction is increased based on the comparison results, and the increase in the confidence level of the adjusted standard entity extraction is positively correlated with the average semantic similarity of the knowledge graph.

[0014] Furthermore, the process of determining whether the annotation result conforms to the standard based on the adjusted re-evaluation includes: In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy, and the entity verification scaling factor is adjusted according to the graph information entropy.

[0015] Furthermore, the process of adjusting the entity verification scaling factor based on the graph information entropy includes: An attribute set is formed based on the attributes of each knowledge graph entity; Calculate the frequency of occurrence of a single attribute; Calculate the weighted sum of the frequencies of each attribute, denoted as the spectral information entropy; The obtained map information entropy is compared with the preset map information entropy; The entity verification scaling factor is increased based on the comparison results, and the increase in the entity verification scaling factor is positively correlated with the graph information entropy.

[0016] Furthermore, the process of adjusting the entity verification scaling factor also includes: Extract the neighbor subgraph of the entities that are successfully aligned in the current period in the knowledge graph; Compare the structural similarity between two subgraphs, and denote it as structural similarity. Calculate the average value of the similarity of each structure, and denote it as the domain structure overlap degree; The obtained overlap of the domain structure is compared with the preset overlap of the domain structure. The adjusted entity verification scaling factor is increased based on the comparison results, and the increase in the adjusted entity verification scaling factor is positively correlated with the overlap of the domain structure.

[0017] Furthermore, the process of determining whether the annotation result conforms to the standard based on the adjusted re-evaluation includes: In response to the second repeated detection condition, the reason why the annotation result does not meet the standard is determined based on the error pattern deviation. Collect feature vectors of non-conforming batches; Calculate the distribution difference of each feature vector in each pattern, and denote it as the distribution dispersion. The distribution dispersion corresponding to the pattern with the largest difference is denoted as the error pattern deviation. The obtained error pattern deviation is compared with the preset error pattern deviation. Based on the comparison results, it was determined that the reason for the non-compliance of the annotation results with the standard was a sudden decline in the quality of the external data source, and the credibility of the external evidence was adjusted according to the time series fluctuation index of the data source. The second repeated detection condition is that after the system completes the adjustment of the entity verification scaling factor, it determines, based on the newly obtained comprehensive annotation quality index, that the data annotation results for each entity data in a single running cycle do not meet the standard.

[0018] Furthermore, the process of adjusting the credibility of external evidence based on the time-series fluctuation index of the data source includes: Select the set of external data feature vectors for the current period; Calculate the similarity of feature distributions between adjacent periods; The ratio of the difference in the characteristic distribution degree between two adjacent periods to the number of periods is calculated and denoted as the data source time series fluctuation index. The obtained time series volatility index of the data source is compared with the preset time series volatility index; The intensity of external evidence fusion is reduced based on the comparison results, and the reduction in the intensity of external evidence fusion is positively correlated with the adjustment range of the time series fluctuation index of the data source. The external evidence fusion strength is the product of the measured entity alignment number and the absolute value of the difference between the data source time-series fluctuation index and 1.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: by constructing a comprehensive annotation quality index, which analyzes the annotation results in each detection cycle, the present invention can effectively determine whether there are acquisition deviations or processing deviations in the cycle. At the same time, it can also output corresponding processing adjustment methods according to the determined actual situation, thereby effectively eliminating the situation where the annotation result quality is too low due to insufficient entity recognition coverage and insufficient entity alignment accuracy during data transmission. While effectively improving the detection efficiency of each annotation result, it effectively avoids the influence of data stability of different modes, thereby effectively improving the processing efficiency of the annotation result quality of the scheme described in the present invention.

[0020] Furthermore, the present invention compares the obtained comprehensive annotation quality index with the preset comprehensive annotation quality index, and determines whether the data annotation results for each entity data in a single running cycle meet the standard based on the comparison results. This allows for intuitive quantification of the collected entity data, thereby quickly determining the actual status of data annotation within the current cycle. Through the rapidly determined results, dynamic evaluation of the collected data can be achieved, thereby effectively outputting corresponding processing decisions and further improving the annotation quality of entity data in the present invention.

[0021] Furthermore, the present invention compares the obtained average entity alignment confidence score with the preset average entity alignment confidence score. Based on the comparison result, it determines the reason why the data annotation result of each entity data in a single running cycle does not meet the standard. This can quickly determine whether the detection of each entity data in the cycle is qualified, thereby ensuring that more accurate data can be obtained after processing decision updates. While further improving accuracy, it further avoids the occurrence of calculation errors, thereby further improving the annotation quality of entity data in the present invention.

[0022] Furthermore, in the case where the cause is determined to be insufficient entity recognition coverage, the present invention compares the obtained entity confidence distribution parameters with the preset entity confidence distribution parameters, and increases the standard entity extraction confidence based on the comparison result. This further improves the completeness of the data processing process and makes the judgment results that meet the standards more scenario-based and intelligent, thereby further improving the annotation quality of entity data in the present invention.

[0023] Furthermore, the present invention compares the obtained average semantic similarity of the knowledge graph with the preset average semantic similarity of the knowledge graph, and reduces the adjusted standard entity extraction confidence based on the comparison result. This makes the adjustment result more refined and accurate, further improving data integrity while avoiding calculation errors, thereby further improving the annotation quality of entity data in the present invention.

[0024] Furthermore, in response to the first repeated detection condition, the present invention determines that the cause is insufficient entity alignment accuracy, which can intuitively identify the corresponding cause and make corresponding processing decisions based on the identified cause, thereby further improving the annotation quality of entity data in the present invention.

[0025] Furthermore, the present invention compares the obtained graph information entropy with a preset graph information entropy, and adds an entity verification scaling factor based on the comparison result. By adjusting the graph information entropy to the corresponding value, a more accurate result is obtained, and a corresponding processing decision is generated, thereby ensuring the acquisition of more accurate data. While further improving the response speed, the present invention further avoids deviations in the comparison results, thereby further improving the annotation quality of entity data in the present invention.

[0026] Furthermore, the present invention compares the obtained domain structure overlap with the preset domain structure overlap, and adds an adjusted entity verification scaling factor based on the comparison result. This can effectively improve the decision-making efficiency of the present invention for different situations, thereby effectively avoiding the processing obstacles caused by excessive interference, further improving the compatibility of decisions for different domain structure overlaps, and thus further improving the annotation quality of entity data by the present invention.

[0027] Furthermore, in response to the second repeated detection condition, the present invention determines the reason why the annotation result does not meet the standard based on the error pattern deviation, thereby intuitively determining whether the error pattern deviation affects the annotation result, thus determining the reason for non-compliance with the standard, accurately adjusting the range, and further generating corresponding processing decisions, thereby ensuring more accurate data is obtained. While further improving the response speed, it further avoids the situation of deviation in the comparison result, thereby further improving the annotation quality of entity data of the present invention.

[0028] Furthermore, the present invention compares the obtained time-series fluctuation index of the data source with a preset time-series fluctuation index, and reduces the fusion strength of external evidence based on the comparison result, thereby effectively avoiding deviations during data collection. While ensuring the rationality of the data during the collection process, the present invention effectively improves the accuracy of the data, thereby further improving the annotation quality of entity data. Attached Figure Description

[0029] Figure 1 This is a flowchart of the knowledge graph-based intelligent data annotation method described in this invention; Figure 2 This is a flowchart illustrating the process of determining whether data annotation results meet the standards based on a comprehensive annotation quality index, as described in this invention. Figure 3 This is a flowchart of the adjustment process for insufficient entity recognition coverage as described in this invention; Figure 4 This is a flowchart illustrating the adjustment process for insufficient entity alignment accuracy described in this invention. Detailed Implementation

[0030] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0031] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0032] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0033] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0034] It should be noted that the data in this embodiment are all derived from a comprehensive analysis and evaluation of historical test data and corresponding historical test results from the system described in this invention over the three months prior to this test. Before this test, the system described in this invention comprehensively determines the preset values ​​stored in the database based on the analysis results of 25,863 cumulative tests over the previous three months and the processing results after handling 19,584 specific cases. Those skilled in the art will understand that the system described in this invention can determine the above-mentioned parameters for a single item by selecting the value with the highest proportion based on the data distribution as the preset standard parameter, using weighted summation to obtain the value as the preset standard parameter, substituting each historical data point into a specific formula and using the value obtained by the formula as the preset standard parameter, or other selection methods, as long as the system described in this invention can clearly define different specific situations in the single-item judgment process through the obtained values.

[0035] Please see Figure 1 As shown, it is a flowchart of the knowledge graph-based intelligent data annotation method of the present invention, wherein, The user log terminal is connected to the corresponding data acquisition module to build a data transmission channel. After the data transmission channel is built, each user log terminal transmits the corresponding entity data. The log parameters generated by each user log terminal during operation are sent to the data processing module for data preprocessing. The entity data for a single user log terminal includes the number of entities in the knowledge graph, the number of entity alignments, the number of labeled entities, and the number of recalled entities. The detection system calculates a comprehensive annotation quality index for each of the received entity data. Based on the comprehensive annotation quality index of each user log terminal during the transmission process, it determines whether the data annotation results for each entity data in a single running cycle meet the standard. If it is determined that the data does not meet the standard, it generates corresponding processing instructions according to the determined reasons, including adjusting the standard entity extraction confidence if the reason is insufficient entity recognition coverage, or adjusting the entity verification scaling factor if the reason is insufficient entity alignment accuracy. The comprehensive annotation quality index is a weighted average of the average of annotation accuracy and annotation recall and annotation consistency. In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy. The first repeated detection condition is that after adjusting the confidence of standard entity extraction, the data annotation results for each entity data in a single running cycle are determined to be non-compliant with the standard based on the newly obtained comprehensive annotation quality index. Based on the obtained adjusted entity verification scaling factor, a re-detection is performed. If the standard is still not met, the corresponding processing method is selected, including adjusting the external data fusion strength. Continuous testing is performed after confirming that the data annotation results for each entity data in a single running cycle meet the standards.

[0036] Specifically, in the detection process of knowledge graph annotation results described in this embodiment of the invention, initial parameters are input to the model for periodic updates, the entity data is preprocessed, and the preprocessed entity data is aligned and compared to generate preliminary annotation results. It is determined whether the comprehensive annotation quality index of the annotation process meets the standard, and if it is determined that it does not meet the standard, a corresponding processing instruction is generated according to the determined reason, and the system performs the corresponding operation according to the received instruction.

[0037] It is understandable that the model will automatically update itself based on the detected comprehensive annotation quality index when performing annotation result detection; Specifically, the system described in this invention stores and manages the collected data and processing results, and constructs a database that can be continuously updated.

[0038] Please see Figure 2As shown, this is a flowchart illustrating the process of determining whether data annotation results meet standards based on a comprehensive annotation quality index, as described in this invention. The processing method includes: Specifically, the ratio of correctly labeled entities to the total number of entities labeled by the system is denoted as the labeling accuracy. The ratio of correctly labeled entities to the total number of entities that should actually be labeled is denoted as the labeling recall. The harmonic mean of the labeling accuracy and the labeling recall is denoted as the measure value. The ratio of the intersection of two independent labels to the union of two independent labels is denoted as the labeling consistency. The normalized weighted average of the measure value and the labeling consistency is denoted as the comprehensive labeling quality index. The obtained comprehensive labeling quality index is compared with the preset comprehensive labeling quality index. Based on the comparison results, it is determined whether the data labeling results for each entity data in a single running cycle meet the standard. If it is determined that each labeling result does not meet the standard, the reason for non-compliance is determined based on the mean entity alignment confidence score. The mean entity alignment confidence score is the average of the ratio of the confidence score of successfully linked entities to the total number of successfully linked entities. If it is determined that the data labeling results for each entity data in a single running cycle meet the standard, continuous detection is performed. In this embodiment, the preset comprehensive labeling quality index H0 = 0.85. If the comprehensive annotation quality index H is greater than the preset comprehensive annotation quality index H0, the detection system determines that the annotation result for the entity data meets the standard. If the comprehensive annotation quality index H is less than or equal to the preset comprehensive annotation quality index H0, the detection system determines that the annotation result for the entity data does not meet the standard, and determines the reason for not meeting the standard based on the mean confidence score of the entity alignment.

[0039] It is understood that the database pre-stores the minimum allowable comprehensive annotation quality index threshold for different types of data. Therefore, the above-mentioned assignment of the preset comprehensive annotation quality index is only a preferred embodiment of the system of the present invention. The present invention does not impose specific restrictions on the value of the preset comprehensive annotation quality index, as long as the comparison result between the obtained comprehensive annotation quality index and the preset comprehensive annotation quality index can directly characterize the analysis of the detection system.

[0040] Specifically, the process described in this embodiment of the invention for determining the reasons why the data annotation results for each entity data in a single running cycle do not meet the standard based on the mean entity alignment confidence score includes: The mean confidence score for entity alignment is calculated, and then compared with a preset mean confidence score for entity alignment. Based on the comparison results, the reasons why the data annotation results for each entity data in a single running cycle do not meet the standard are determined. If the reason is insufficient entity recognition coverage, the standard entity extraction confidence score is adjusted according to the entity confidence score distribution parameter. Here, the entity confidence score distribution parameter is the ratio of the number of candidate entities accepted by the system to the total number of candidate entities, and the standard entity extraction confidence score is the acceptance threshold for entity recognition confidence. If the reason is insufficient entity alignment accuracy, the entity verification scaling factor is adjusted according to the graph information entropy. Here, the graph information entropy is the weighted sum of the frequencies of each attribute, and the entity verification scaling factor is a function based on the graph information entropy. In this embodiment, the preset mean confidence score for entity alignment is C0 = 0.75. If the mean confidence level C of the entity alignment is greater than the mean confidence level C0 of the preset entity alignment, the detection system determines that the reason for not meeting the standard is insufficient entity recognition coverage, and generates an instruction to optimize the confidence level of standard entity extraction. If the mean confidence level C of the entity alignment is less than or equal to the preset mean confidence level C0 of the entity alignment, the detection system determines that the reason for not meeting the standard is insufficient entity alignment accuracy, and generates an instruction to optimize the entity verification scaling factor.

[0041] Please see Figure 3 The diagram shows a flowchart of the adjustment process for insufficient entity recognition coverage as described in this invention. The process includes: Specifically, the process of adjusting the standard entity extraction confidence level according to the entity confidence level distribution parameter when the cause is determined to be insufficient entity recognition coverage, as described in the embodiments of the present invention, includes: Calculate the entity confidence distribution parameters, compare the obtained entity confidence distribution parameters with the preset entity confidence distribution parameters, and increase the standard entity extraction confidence based on the comparison results. The increase in the standard entity extraction confidence is positively correlated with the entity confidence distribution parameters. In this embodiment, the first preset entity confidence distribution parameter θ1 = 0.60, the second preset entity confidence distribution parameter θ2 = 0.85, the first extraction adjustment coefficient n1 = 0.35, the second extraction adjustment coefficient n2 = 0.55, and the third extraction adjustment coefficient n3 = 0.75. If the entity confidence distribution parameter θ is greater than the second preset entity confidence distribution parameter θ2, the detection system determines to use the third extraction adjustment coefficient n3 to adjust the standard entity extraction confidence B. If the entity confidence distribution parameter θ is less than or equal to the second preset entity confidence distribution parameter θ2 and greater than the first preset entity confidence distribution parameter θ1, the detection system determines to use the second extraction adjustment coefficient n2 to adjust the standard entity extraction confidence B. If the entity confidence distribution parameter θ is less than or equal to the first preset entity confidence distribution parameter θ1, the detection system determines to use the first extraction adjustment coefficient n1 to adjust the standard entity extraction confidence B. When the detection system uses the a-th extraction adjustment coefficient na to adjust the standard entity extraction confidence level B, a=1, 2, 3, the adjusted standard entity extraction confidence level B'=B×na is set.

[0042] Understandably, when it is necessary to improve the coverage of entity recognition, the filtering should be appropriately relaxed, that is, the entity confidence distribution parameter should be reduced to allow more low-confidence entities to enter in order to expand the coverage range, thereby improving the confidence of standard entity extraction. Therefore, an appropriate adjustment coefficient should be selected to improve the confidence of standard entity extraction, and the increase in the confidence of standard entity extraction is positively correlated with the decrease in the entity confidence distribution parameter.

[0043] It is understood that the detection system of the present invention re-determines whether it meets the standard based on the adjusted comprehensive labeling quality index, and issues an instruction to adjust the reported value during the transmission process if it is determined that it does not meet the standard.

[0044] Specifically, the standard entity extraction confidence adjustment process described in this embodiment of the invention further includes: The semantic similarity between a single candidate entity and its corresponding entity in the knowledge graph is calculated. The maximum value of the semantic similarity is taken as the best matching degree between the entity and the knowledge graph. The average ratio of the total number of matching degrees to the total number of entities is calculated and taken as the average knowledge graph semantic similarity. The obtained average knowledge graph semantic similarity is compared with the preset average knowledge graph semantic similarity. Based on the comparison result, the adjusted standard entity extraction confidence is increased, and the increase in the adjusted standard entity extraction confidence is positively correlated with the average knowledge graph semantic similarity. In this embodiment, the first preset average knowledge graph semantic similarity S1=0.7, the second preset average knowledge graph semantic similarity S2=0.9, the first extraction correction coefficient k1=1.15, the second extraction correction coefficient k2=1.25, and the third extraction correction coefficient k3=1.35. If the average knowledge graph semantic similarity S is greater than the second preset average knowledge graph semantic similarity S2, the detection system determines that the standard entity extraction confidence B' is corrected and adjusted using the third extraction correction coefficient k3. If the average knowledge graph semantic similarity S is less than or equal to the second preset average knowledge graph semantic similarity S2 and greater than the first preset average knowledge graph semantic similarity S1, the detection system determines that the standard entity extraction confidence B' is corrected and adjusted using the second extraction correction coefficient k2. If the average knowledge graph semantic similarity S is less than or equal to the first preset average knowledge graph semantic similarity S1, the detection system determines that the standard entity extraction confidence B' is corrected and adjusted using the first extraction correction coefficient k1. When the detection system uses the first Extracting correction coefficients When correcting the adjusted standard entity extraction confidence level B', =1, 2, 3, set the adjusted standard entity extraction confidence level B'' = B' × .

[0045] Specifically, the process of determining whether the annotation result conforms to the standard based on the adjusted re-judgment, as described in the embodiments of the present invention, includes: In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy, and the entity verification scaling factor is adjusted according to the graph information entropy.

[0046] Please see Figure 4 The diagram shows a flowchart of the adjustment process for insufficient entity alignment accuracy described in this invention. The process includes: Specifically, the process of adjusting the entity verification scaling factor based on the graph information entropy described in this embodiment of the invention includes: An attribute set is formed based on the attributes of each knowledge graph entity. The frequency of occurrence of a single attribute is calculated, and the weighted sum of the frequencies of each attribute is calculated and denoted as the graph information entropy. The obtained graph information entropy is compared with a preset graph information entropy. Based on the comparison result, an entity verification scaling factor is increased, and the increase in the entity verification scaling factor is positively correlated with the graph information entropy. In this embodiment, the first preset graph information entropy g1=1.5, the second preset graph information entropy g2=3.0, the first factor adjustment coefficient m1=1.35, the second factor adjustment coefficient m2=1.55, and the third factor adjustment coefficient m3=1.75. If the entropy of the map information g is greater than the second preset map information entropy g2, the detection system determines to use the third factor adjustment coefficient m3 to adjust the entity verification scaling factor ρ. If the graph information entropy g is less than or equal to the second preset graph information entropy g2 and greater than the first preset graph information entropy g1, the detection system determines to use the second factor adjustment coefficient m2 to adjust the entity verification scaling factor ρ. If the graph information entropy g is less than or equal to the first preset graph information entropy g1, the detection system determines to use the first factor adjustment coefficient m1 to adjust the entity verification scaling factor ρ. When the detection system adjusts the entity verification scaling factor ρ using the d-th factor adjustment coefficient md, d=1, 2, 3, the adjusted entity verification scaling factor ρ'=ρ×md is set.

[0047] Specifically, the entity verification scaling factor adjustment process described in this embodiment of the invention further includes: Extract the neighbor subgraphs of successfully aligned entities in the knowledge graph during the current period, compare the structural similarity of the two subgraphs, and record it as the structural similarity. Calculate the average of the structural similarities and record it as the domain structural overlap. Compare the obtained domain structural overlap with the preset domain structural overlap. Based on the comparison result, increase the adjusted entity verification scaling factor, and the increase in the adjusted entity verification scaling factor is positively correlated with the domain structural overlap. In this embodiment, the first preset domain structural overlap R1=0.3, the second preset domain structural overlap R2=0.7, the first factor correction coefficient η1=1.015, the second factor correction coefficient η2=1.025, and the third factor correction coefficient η3=1.035. If the overlap of the domain structure R is greater than the second preset overlap of the domain structure R2, the detection system determines that the entity verification scaling factor ρ' after adjustment is corrected using the third factor correction coefficient η3. If the overlap of the domain structure R is less than or equal to the second preset overlap of the domain structure R2 and greater than the first preset overlap of the domain structure R1, the detection system determines that the entity verification scaling factor ρ' after adjustment is corrected using the second factor correction coefficient η2. If the overlap degree of the domain structure R is less than or equal to the first preset overlap degree of the domain structure R1, the detection system determines that the entity verification scaling factor ρ' after adjustment is corrected using the first factor correction coefficient η1. When the detection system uses the j-th factor correction coefficient ηj to correct the adjusted entity verification scaling factor ρ', j=1, 2, 3, the corrected adjusted entity verification scaling factor ρ''=p'×ηj is set.

[0048] Specifically, the process of determining whether the annotation result conforms to the standard based on the adjusted re-judgment, as described in the embodiments of the present invention, includes: In response to the second repeated detection condition, the reason why the labeling result does not meet the standard is determined according to the error mode deviation. The feature vectors of the unqualified batches are collected, and the distribution difference of each feature vector in each mode is calculated and recorded as the distribution dispersion. The distribution dispersion corresponding to the mode with the largest difference is recorded as the error mode deviation. The obtained error mode deviation is compared with the preset error mode deviation. According to the comparison result, it is determined that the reason for the labeling result not meeting the standard is a sudden quality decline of the external data source. The credibility of the external evidence is adjusted according to the time series fluctuation index of the data source. The second repeated detection condition is that after the system completes the adjustment of the entity verification scaling factor, it determines, based on the newly obtained comprehensive annotation quality index, that the data annotation results for each entity data in a single running cycle do not meet the standard.

[0049] In this embodiment, the preset error mode deviation e0 = 0.35; If the error mode deviation e is greater than the preset error mode deviation e2, the detection system determines that the reason for non-compliance with the standard is a sudden decline in the quality of external data, and issues an adjustment instruction for the fusion strength of external evidence. If the error mode deviation e is less than or equal to the preset error mode deviation e2, the detection system determines that the reason for non-compliance with the standard is a common internal error mode and issues an adjustment command to return to the upper layer for repair.

[0050] Specifically, the process of adjusting the credibility of external evidence based on the time-series fluctuation index of the data source as described in this embodiment of the invention includes: Select the set of external data feature vectors under the current period, calculate the feature distribution similarity of adjacent periods, calculate the ratio of the difference in feature distribution degree between two adjacent periods to the number of periods, and denot it as the data source time series fluctuation index. Compare the obtained data source time series fluctuation index with the preset time series fluctuation index, and reduce the external evidence fusion intensity according to the comparison result. The reduction of the external evidence fusion intensity is positively correlated with the adjustment amplitude of the data source time series fluctuation index. The external evidence fusion intensity is the product of the measured entity alignment number and the absolute value of the difference between the data source time series fluctuation index and 1.

[0051] In this embodiment, the first preset data source time series fluctuation index z1=0.25, the second preset data source time series fluctuation index z2=0.60, the first intensity adjustment coefficient λ1=1.25, the second intensity adjustment coefficient λ2=1.35, and the third intensity adjustment coefficient λ3=1.45; If the time series fluctuation index z of the data source is greater than the second preset time series fluctuation index z2 of the data source, the detection system determines to use the third intensity adjustment coefficient λ3 to adjust the intensity w of the external evidence fusion. If the time series fluctuation index z of the data source is less than or equal to the second preset data source time series fluctuation index z2 and greater than the first preset data source time series fluctuation index z1, the detection system determines to use the second intensity adjustment coefficient λ2 to adjust the intensity w of the external evidence fusion. If the time series fluctuation index z of the data source is less than or equal to the first preset time series fluctuation index z1 of the data source, the detection system determines to use the first intensity adjustment coefficient λ1 to adjust the intensity w of the external evidence fusion. When the detection system uses the r-th intensity adjustment coefficient λr to adjust the external evidence fusion intensity w, r=1,2,3, the adjusted external evidence fusion intensity w'=w×λr is set.

[0052] It is understood that the database pre-stores the minimum allowable data source time series fluctuation index threshold for different types of parameters. Therefore, the above-mentioned assignment of values ​​for the first preset data source time series fluctuation index and the second preset data source time series fluctuation index is only a preferred embodiment of the system of the present invention. The present invention does not impose specific restrictions on the values ​​of each preset data source time series fluctuation index, as long as the comparison results between the obtained data source time series fluctuation index and each preset data source time series fluctuation index can directly characterize the analysis of each parameter by the detection system. The detection system also adjusts the fusion strength of the corresponding external evidence, and if the annotation result for the entity data is found to be inconsistent with the standard based on the adjusted fusion strength of external evidence, the detection system sends an update command to the database.

[0053] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge graph-based intelligent data annotation method, characterized in that, include: The user log terminal is connected to the corresponding data acquisition module to build a data transmission channel. After the data transmission channel is built, each user log terminal transmits the corresponding entity data. The log parameters generated by each user log terminal during operation are sent to the data processing module for data preprocessing. The entity data for a single user log terminal includes the number of entities in the knowledge graph, the number of entity alignments, the number of labeled entities, and the number of recalled entities. The detection system calculates a comprehensive annotation quality index for each of the received entity data. Based on the comprehensive annotation quality index of each user log terminal during the transmission process, it determines whether the data annotation results for each entity data in a single running cycle meet the standard. If it is determined that the data does not meet the standard, it generates corresponding processing instructions according to the determined reasons, including adjusting the standard entity extraction confidence if the reason is insufficient entity recognition coverage, or adjusting the entity verification scaling factor if the reason is insufficient entity alignment accuracy. The comprehensive annotation quality index is a weighted average of the average of annotation accuracy and annotation recall and annotation consistency. In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy. The first repeated detection condition is that after adjusting the confidence of standard entity extraction, the data annotation results for each entity data in a single running cycle are determined to be non-compliant with the standard based on the newly obtained comprehensive annotation quality index. Based on the obtained adjusted entity verification scaling factor, a re-detection is performed. If the standard is still not met, the corresponding processing method is selected, including adjusting the external data fusion strength. Continuous testing is performed after confirming that the data annotation results for each entity data in a single running cycle meet the standards.

2. The knowledge graph-based intelligent data annotation method according to claim 1, characterized in that, The process of determining whether the data annotation results for each entity data in a single runtime cycle meet the standard includes: The ratio of correctly labeled entities to the total number of entities labeled by the system is denoted as the labeling accuracy rate. The ratio of correctly labeled entities to the total number of entities that should actually be labeled is denoted as the labeling recall rate. Calculate the harmonic mean of annotation precision and annotation recall, and record it as the measure value; The ratio of the number of entities in the intersection to the number of entities in the union of two independently labeled sets is denoted as label consistency. The normalized weighted average of the consistency between the measurement value and the annotation is calculated and denoted as the comprehensive annotation quality index. Compare the obtained comprehensive annotation quality index with the preset comprehensive annotation quality index; Based on the comparison results, determine whether the data annotation results for each entity data in a single running cycle meet the standards; If it is determined that each annotation result does not meet the standard, the reason for non-compliance is determined based on the mean confidence score of entity alignment, where the mean confidence score of entity alignment is the average of the ratios of the confidence scores of successfully linked entities to the total number of successfully linked entities. Continuous testing is performed after confirming that the data annotation results for each entity data in a single running cycle meet the standards.

3. The knowledge graph-based intelligent data annotation method according to claim 2, characterized in that, The process of determining the reasons why the data annotation results for each entity data in a single running cycle do not meet the standard based on the mean entity alignment confidence score includes: Calculate the mean confidence score for entity alignment; The calculated mean confidence score of the entity alignment is compared with the preset mean confidence score of the entity alignment. Based on the comparison results, determine the reasons why the data annotation results for each entity data in a single running cycle do not meet the standards; If the cause is determined to be insufficient entity recognition coverage, the standard entity extraction confidence level is adjusted according to the entity confidence distribution parameter; wherein, the entity confidence distribution parameter is the ratio of the number of candidate entities accepted by the system to the total number of candidate entities, and the standard entity extraction confidence level is the acceptance threshold of the entity recognition confidence level; If the cause is determined to be insufficient entity alignment accuracy, the entity verification scaling factor is adjusted based on the graph information entropy; wherein, the graph information entropy is the weighted sum of the frequencies of each attribute, and the entity verification scaling factor is a function based on the graph information entropy.

4. The knowledge graph-based intelligent data annotation method according to claim 3, characterized in that, The process of adjusting the standard entity extraction confidence level based on the entity confidence level distribution parameter when the cause is determined to be insufficient entity recognition coverage includes: Calculate the entity confidence distribution parameters; The obtained entity confidence distribution parameters are compared with the preset entity confidence distribution parameters; The confidence level of the standard entity extraction is increased based on the comparison results, and the increase in the confidence level of the standard entity extraction is positively correlated with the entity confidence distribution parameter.

5. The knowledge graph-based intelligent data annotation method according to claim 4, characterized in that, The process of adjusting the confidence level of the standard entity extraction also includes: Calculate the semantic similarity between a single candidate entity and its corresponding entity in the knowledge graph; The maximum value of the semantic similarity is taken as the best matching degree between the entity and the knowledge graph; The average ratio of the total number of matching scores to the total number of entities is denoted as the average knowledge graph semantic similarity. The obtained average semantic similarity of the knowledge graph is compared with the preset average semantic similarity of the knowledge graph. The confidence level of the adjusted standard entity extraction is increased based on the comparison results, and the increase in the confidence level of the adjusted standard entity extraction is positively correlated with the average semantic similarity of the knowledge graph.

6. The knowledge graph-based intelligent data annotation method according to claim 5, characterized in that, The process of determining whether the annotation results conform to the standard based on the adjusted re-evaluation includes: In response to the first repeated detection condition, the cause is determined to be insufficient entity alignment accuracy, and the entity verification scaling factor is adjusted according to the graph information entropy.

7. The knowledge graph-based intelligent data annotation method according to claim 6, characterized in that, The process of adjusting the entity verification scaling factor based on the graph information entropy includes: An attribute set is formed based on the attributes of each knowledge graph entity; Calculate the frequency of occurrence of a single attribute; Calculate the weighted sum of the frequencies of each attribute, denoted as the spectral information entropy; The obtained map information entropy is compared with the preset map information entropy; The entity verification scaling factor is increased based on the comparison results, and the increase in the entity verification scaling factor is positively correlated with the graph information entropy.

8. The knowledge graph-based intelligent data annotation method according to claim 7, characterized in that, The process of adjusting the entity verification scaling factor also includes: Extract the neighbor subgraph of the entities that are successfully aligned in the current period in the knowledge graph; Compare the structural similarity between two subgraphs, and denote it as structural similarity. Calculate the average value of the similarity of each structure, and denote it as the domain structure overlap degree; The obtained overlap of the domain structure is compared with the preset overlap of the domain structure. The adjusted entity verification scaling factor is increased based on the comparison results, and the increase in the adjusted entity verification scaling factor is positively correlated with the overlap of the domain structure.

9. The intelligent data annotation method based on knowledge graphs according to claim 8, characterized in that, The process of determining whether the annotation results conform to the standard based on the adjusted re-evaluation includes: In response to the second repeated detection condition, the reason why the annotation result does not meet the standard is determined based on the error pattern deviation. Collect feature vectors of non-conforming batches; Calculate the distribution difference of each feature vector in each pattern, and denote it as the distribution dispersion. The distribution dispersion corresponding to the pattern with the largest difference is denoted as the error pattern deviation. The obtained error pattern deviation is compared with the preset error pattern deviation. Based on the comparison results, it was determined that the reason for the non-compliance of the annotation results with the standard was a sudden decline in the quality of the external data source, and the credibility of the external evidence was adjusted according to the time series fluctuation index of the data source. The second repeated detection condition is that after the system completes the adjustment of the entity verification scaling factor, it determines, based on the newly obtained comprehensive annotation quality index, that the data annotation results for each entity data in a single running cycle do not meet the standard.

10. The knowledge graph-based intelligent data annotation method according to claim 9, characterized in that, The process of adjusting the credibility of external evidence based on the time-series fluctuation index of the data source includes: Select the set of external data feature vectors for the current period; Calculate the similarity of feature distributions between adjacent periods; The ratio of the difference in the characteristic distribution degree between two adjacent periods to the number of periods is calculated and denoted as the data source time series fluctuation index. The obtained time series fluctuation index of the data source is compared with the preset time series fluctuation index; The intensity of external evidence fusion is reduced based on the comparison results, and the reduction in the intensity of external evidence fusion is positively correlated with the adjustment range of the time series fluctuation index of the data source. The external evidence fusion strength is the product of the measured entity alignment number and the absolute value of the difference between the data source time-series fluctuation index and 1.

Images