An AI large model-based knowledge base data fusion association method and system

Abstract

The application relates to the technical field of knowledge base fusion, and particularly relates to a knowledge base data fusion correlation method and system based on an AI large model. First, original data is collected through a connector, then washing and deduplication processing are carried out, then field names are vectorized and aligned, an AI large model is used to extract entities and attributes, entity conflicts are solved through a density clustering algorithm and an entity similarity threshold, relationship triplets are generated, uniform identification is assigned to entities and attributes for standardized processing, data conflicts are detected through a coefficient of variation, and parameters are dynamically adjusted to optimize a fusion strategy, finally, a vector index is constructed through a graph database, and a graph download interface is provided. On the basis of realizing data fusion, the application skillfully solves the problem that in the existing database fusion method, when facing data conflicts, it is difficult to specifically identify the reasons and give solutions, significantly improves the accuracy and reliability of knowledge base data fusion, and has important innovative significance and practical value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge base fusion, and particularly relates to a knowledge base data fusion and correlation method and system based on an AI large model. BACKGROUND

[0002] In today's era, information is growing explosively, and massive data is flooding into our lives and work. The fragmentation, diversity and rapid update of information bring great challenges to people's acquisition and utilization of knowledge. Knowledge base data fusion emerges as the times require, which can effectively eliminate data redundancy and inconsistency, fill in data gaps, and improve data accuracy, thereby improving data quality. This is like opening a clear knowledge channel for people in the ocean of information, allowing the power of knowledge to be fully utilized, and thus promoting the progress and development of society.

[0003] However, in the process of data fusion, knowledge conflict problems lurk like reefs. These conflicts can lead to decreased data reliability, damaged accuracy, reduced usability, increased maintenance costs, worsened user experience, increased legal compliance risks, even interrupted business processes, and increased technical implementation complexity. More troublesome is that these conflicts may be hidden after data fusion and difficult to be discovered at the first time, but gradually exposed during system operation. At present, there is a lack of intelligent conflict detection, resolution and optimization mechanism, and it is difficult to identify the root cause of the problem and handle the problem in the face of complex fusion problems.

[0004] A knowledge base fusion method for encyclopedic websites is disclosed in Chinese patent CN110377747B, which fuses the knowledge cards (infoboxes) of the three most influential encyclopedic websites, namely, Baidu Encyclopedia, Interactive Encyclopedia, and Chinese Wikipedia. The method comprises the following steps: Step 1, obtaining the query results of the encyclopedic websites about the same entity and preprocessing; Step 2, establishing a mapping relationship between the entities in the encyclopedic websites based on the concept similarity, attribute similarity, and context similarity features; Step 3, aligning the attributes of the knowledge cards of the entities with the established mapping relationship with the help of external dictionaries; Step 4, for the attributes with conflicting attribute values, designing a single true value discovery scheme and a multiple true value discovery scheme according to whether the attribute values are single-valued or multi-valued; Step 5, outputting the fused attribute-attribute value pairs. The final obtained attribute-attribute value pairs about the entity with high reliability and redundancy removal of the three encyclopedic knowledge cards. Although this method can reduce redundancy and reduce conflicts to some extent, its conflict processing is still relatively simple. It cannot give a targeted solution, nor can it timely identify the cause of the conflict and solve the source of the conflict, resulting in the fact that this solution cannot truly solve the exception and reduce the probability of conflict. How to optimize the database fusion scheme, timely identify the main contradiction leading to data conflict, and solve the hidden risks behind the data has become a key research issue. SUMMARY

[0005] The present application provides a knowledge base data fusion association method and system based on AI large model, to overcome the problem in the existing database fusion method that it is difficult to determine the cause of data conflict and give a targeted solution to the cause of data conflict.

[0006] In the first aspect, the present application provides a knowledge base data fusion association method based on AI large model, comprising,

[0007] Data acquisition, based on the connector to collect raw data;

[0008] Data preprocessing, cleaning the raw data, and removing duplicate data based on the Jaccard similarity threshold;

[0009] Field alignment, obtaining each field name and generating corresponding vectorized semantics, and aligning each vectorized semantics based on the field name similarity threshold;

[0010] Entity recognition and extraction, using AI large model to extract entities and attributes from the cleaned data and record them as extracted entities and extracted attributes;

[0011] The entity disambiguation and linking generates corresponding vectorized semantics for each of the extracted entities, merges the extracted entities based on a density clustering algorithm, retrieves vectorized entities in a database based on an entity similarity threshold, records a vectorized entity as a candidate entity if the similarity between the extracted entity and the vectorized entity is higher than the entity similarity threshold, constructs a link between the extracted entity and the vectorized entity with the highest similarity if there are several candidate entities, and records the corresponding extracted entity as a linked entity, otherwise, a new entity is constructed;

[0012] The relationship extraction and triple construction generate triples based on an AI large model;

[0013] The standardized identification and normalization assign uniform identifiers to entities and attributes, and obtain fused data;

[0014] The conflict detection and optimization judge whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data, judge the reason for not meeting the requirements based on the proportion of the repeated hash value of the fused data in the total hash value when it is judged that the requirements are not met, and modify the data preprocessing parameters, field alignment parameters and entity disambiguation and linking parameters based on the corresponding reason;

[0015] The vector index construction and output constructs a vector index using a graph database and provides a download interface for the graph.

[0016] Further, the process of judging whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data includes:

[0017] Obtain the original data and the fused data and vectorize them, and align the original data and the fused data;

[0018] Calculate the absolute difference value of each field, generate a difference value sequence, and determine the coefficient of variation of the original data and the fused data based on the difference sequence;

[0019] If the coefficient of variation is less than or equal to a preset coefficient of variation, it is judged that the current data conflict meets the requirements, and the knowledge base data fusion association method parameters remain unchanged;

[0020] If the coefficient of variation is greater than the preset coefficient of variation, it is judged that the current data conflict does not meet the requirements, and the reason for not meeting the requirements is judged based on the proportion of the repeated hash value of the fused data in the total hash value.

[0021] Further, the process of judging the reason for not meeting the requirements based on the proportion of the repeated hash value of the fused data in the total hash value includes:

[0022] Convert the fused data into a hash value;

[0023] determine a hash collision rate based on a proportion of the number of repeated hash values in the total number of hash values in the fused data;

[0024] If the hash collision rate is less than or equal to a preset hash collision rate, determine that the reason is not met based on a proportion of the linked successful entities in the extracted entities;

[0025] If the hash collision rate is greater than the preset hash collision rate, determine that the Jaccard similarity threshold is not reasonable, and correct the Jaccard similarity threshold based on a ratio of the hash collision rate to the preset hash collision rate;

[0026] The Jaccard similarity threshold refers to the lowest Jaccard similarity required between a pair of data attribute values when determining that the data attribute values are repeated.

[0027] Further, the process of correcting the Jaccard similarity threshold based on the ratio of the hash collision rate to the preset hash collision rate includes:

[0028] determine a hash collision rate ratio based on the ratio of the hash collision rate to the preset hash collision rate;

[0029] lower the Jaccard similarity threshold based on the hash collision rate ratio, and the lowering amplitude of the Jaccard similarity threshold is proportional to the hash collision rate ratio.

[0030] Further, the process of determining that the reason is not met based on the proportion of the linked successful entities in the extracted entities includes:

[0031] determine an extracted entity linking success rate based on the proportion of the linked successful entities in the extracted entities;

[0032] If the extracted entity linking success rate is less than or equal to a preset extracted entity linking success rate, determine that the entity similarity threshold is not reasonable, and correct the entity similarity threshold based on a ratio of the preset extracted entity linking success rate to the extracted entity linking success rate;

[0033] If the extracted entity linking success rate is greater than the preset extracted entity linking success rate, determine that the reason is not met based on a proportion of the number of field name similarities distributed in a field name similarity range in the total number of field name similarities;

[0034] The entity similarity threshold refers to the similarity required between the extracted entity and the vectorized entity when the vectorized entity is recorded as a candidate entity, and the field name similarity range refers to a similarity range uniformly distributed on both sides of the field name similarity threshold, including the field name similarity threshold.

[0035] Further, the process of correcting the entity similarity threshold based on the ratio of the preset extraction entity linking success rate and the extraction entity linking success rate comprises:

[0036] determining an entity linking success rate ratio based on the ratio of the preset extraction entity linking success rate and the extraction entity linking success rate;

[0037] decreasing the entity similarity threshold based on the entity linking success rate ratio, and the decreasing amplitude of the entity similarity threshold is proportional to the entity linking success rate ratio.

[0038] Further, the process of determining the reason for not meeting based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities comprises:

[0039] determining a field name similarity distribution proportion based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities;

[0040] if the field name similarity distribution proportion is less than or equal to a preset field name similarity distribution proportion, determining that the density clustering algorithm parameter setting is unreasonable, and correcting the density clustering algorithm field radius based on the ratio of the coefficient of variation and the preset coefficient of variation;

[0041] if the field name similarity distribution proportion is greater than the preset field name similarity distribution proportion, determining that the field name similarity threshold setting is unreasonable, and correcting the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion;

[0042] The field name similarity threshold refers to the minimum similarity required for the field name to be aligned to the corresponding vectorized semantics.

[0043] Further, the process of correcting the density clustering algorithm field radius based on the ratio of the coefficient of variation and the preset coefficient of variation comprises:

[0044] determining a coefficient of variation ratio based on the ratio of the coefficient of variation and the preset coefficient of variation;

[0045] increasing the density clustering algorithm field radius based on the coefficient of variation ratio, and the increasing amplitude of the density clustering algorithm field radius is proportional to the coefficient of variation ratio.

[0046] Further, the process of correcting the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion comprises:

[0047] determine a distribution proportion difference value based on a difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion;

[0048] reduce the field name similarity threshold value based on the distribution proportion difference value, and the reduction amplitude of the field name similarity threshold value is proportional to the distribution proportion difference value.

[0049] In a second aspect, the present application provides an AI large model-based knowledge base data fusion association system, comprising,

[0050] The acquisition module comprises a plurality of connectors for acquiring the original data.

[0051] The data preprocessing module is connected to the acquisition module and is used to clean the data and deduplicate the cleaned data based on a Jaccard similarity threshold value.

[0052] The field alignment module is connected to the data preprocessing module and is used to obtain each field name and generate corresponding vectorized semantics, and align each vectorized semantics based on a field name similarity threshold value.

[0053] The entity recognition and extraction module is connected to the field alignment module and is used to extract entities and attributes from the cleaned data based on an AI large model, and record the extracted entities and attributes as extracted entities and extracted attributes.

[0054] The entity disambiguation and linking module is connected to the entity recognition and extraction module and is used to generate vectorized semantics corresponding to each extracted entity, merge the extracted entities based on a density clustering algorithm, and retrieve vectorized entities in a database based on an entity similarity threshold value. If there are several candidate entities, a link is established between the extracted entity and the vectorized entity with the highest similarity, and the corresponding extracted entity is recorded as a linked successful entity. Otherwise, a new entity is constructed.

[0055] The triple construction module is connected to the entity disambiguation and linking module and is used to generate triples based on an AI large model.

[0056] The normalization module is connected to the triple construction module and is used to assign uniform identifiers to entities and attributes to obtain fused data.

[0057] The conflict detection and optimization module is connected to the normalization module and is used to determine whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data, determine the reason for not meeting the requirements based on the proportion of the repeated hash value of the fused data in the total hash value when it is determined that the requirements are not met, and modify the data preprocessing parameters, field alignment parameters, and entity disambiguation and linking parameters based on the corresponding reason.

[0058] a control module connected with the conflict detection and optimization module, the data preprocessing module, the field alignment module and the entity disambiguation and linking module, to schedule the corresponding modules to execute according to the instructions of the conflict detection and optimization module;

[0059] an output module connected with the normalization module and the conflict detection and optimization module, to construct a vector index based on a graph database and provide a download interface of a graph atlas.

[0060] Compared with the prior art, the present application has the beneficial effects that the present application collects raw data through the connector, first performs cleaning and deduplication processing to improve data quality, then performs vectorization alignment on the field name to solve the inconsistency problem of the field. Subsequently, the entity and attribute are extracted by using the AI large model, and then the entity disambiguation and linking are performed by using the density clustering algorithm and the entity similarity threshold. The relationship triplets are generated based on the AI large model, and the uniform identification is assigned to the entity and attribute, to obtain the fused data. In the conflict detection and optimization link, the present application uses the coefficient of variation to evaluate the fusion stability, and accurately locates the duplicate data problem by using the hash conflict rate. Based on this, the data preprocessing parameters, the field alignment parameters and the entity disambiguation and linking parameters are dynamically adjusted, and the data conflict problem in the data fusion process is corrected. This effectively solves the problem that the conflict reason is difficult to be judged and the solution is difficult to be given in the existing method, and significantly improves the fusion quality and reliability. Finally, the present application constructs a vector index based on a graph database and provides a graph download interface, which is convenient for user query and application. Through multi-dimensional quantitative analysis, the present application accurately locates the data conflict root cause and dynamically adjusts the related parameters to solve the conflict problem, overcomes the defects of the existing method, significantly improves the quality and reliability of the knowledge base data fusion, and has important innovative significance and practical value.

[0061] Further, the coefficient of variation is the ratio of the standard deviation to the mean, which is scale-independent and suitable for the difference comparison of multi-field and different scale data in cross-source data. In the data fusion scene, different sources of data may have differences in scale, unit, field distribution, etc. The coefficient of variation can well adapt to this diversity, and by calculating the coefficient of variation of the difference sequence of the original data and the fused data, the overall fluctuation level of the fusion process can be quantified. Low coefficient of variation usually indicates that the difference distribution of the fused data is uniform and stable, and the conflict solution is reasonable; while high coefficient of variation implies that there may be conflict error processing, information loss or error matching in the fusion process, which needs to be further investigated. The advantage of this method is that it can quickly and quantitatively evaluate the stability and consistency of data fusion, providing a clear direction and basis for subsequent conflict optimization, thereby improving the quality and reliability of data fusion.

[0062] Further, the application further refines the data conflict detection mechanism by judging the reason for not meeting based on the proportion of repeated hash values in the total hash values after fusion, effectively solving the problem that it is difficult to accurately locate the root cause of data conflict relying only on the coefficient of variation. In the data fusion process, repeated data may cause conflicts, and by calculating the hash conflict rate, the proportion of repeated data in the fused data can be intuitively reflected. When the hash conflict rate is greater than the preset hash conflict rate, it means that the Jaccard similarity threshold is not reasonable, resulting in insufficient deduplication. At this time, the Jaccard similarity threshold is corrected based on the ratio of the hash conflict rate to the preset hash conflict rate, which can dynamically adjust the deduplication strategy, thereby optimizing the effect of data fusion. The advantage of this method is that it can efficiently and objectively identify the conflict problem caused by data duplication, and automatically adjust the similarity threshold to achieve adaptive optimization of the system, without frequent manual intervention, improving the automation and efficiency of the system, enabling the system to better adapt to dynamic changes in data, maintaining the stability and reliability of data fusion.

[0063] Further, the application corrects the Jaccard similarity threshold based on the ratio of the hash conflict rate to the preset hash conflict rate, solving the problem of insufficient deduplication caused by unreasonable Jaccard similarity threshold setting in the data fusion process. The hash conflict rate ratio reflects the deviation of the current hash conflict rate from the preset value, and a higher ratio means that there are more repeated data that have not been correctly processed in the data fusion process. By reducing the Jaccard similarity threshold, the probability of data being identified as repeated can be increased, thereby reducing the hash conflict rate and improving the quality of data fusion. Using a hierarchical adjustment mechanism, different correction thresholds are used for adjustment according to different ranges of hash conflict rate ratio, and this dynamic adjustment method can achieve adaptive optimization of the system, avoid misjudgment caused by excessive adjustment, and provide greater flexibility and adaptability to cope with conflict characteristics of different data sets and application scenarios, further improving the accuracy and robustness of data fusion.

[0064] Further, the extraction entity linking success rate intuitively reflects the success degree of matching the extracted entity with the entity in the database in the entity disambiguation and linking process. By correcting the entity similarity threshold based on the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate, the entity similarity threshold can be dynamically adjusted, the linking success rate can be improved, the misjudgment can be reduced, and the system performance can be improved. The advantage of this method is that it provides a quantitative index that can intuitively evaluate the effect of entity disambiguation and linking process, accurately locate the abnormal reason, and further improve the performance and reliability of the entire knowledge base data fusion association method.

[0065] Further, the present application solves the problem of low entity linking success rate caused by unreasonable entity similarity threshold by modifying the entity similarity threshold based on the ratio of the preset extraction entity linking success rate and the extraction entity linking success rate. The entity linking success rate ratio reflects the deviation of the current linking success rate from the expected value. By making the reduction amplitude proportional to the entity linking success rate ratio, dynamic adjustment can be achieved, allowing the system to better adapt to data changes, improve linking success rate, reduce misjudgment, improve system performance, and have high interpretability and operability. A hierarchical adjustment mechanism is adopted, and different correction thresholds are used for adjustment according to different ranges of the entity linking success rate ratio, which can more accurately deal with different degrees of data conflict and avoid misjudgment caused by excessive adjustment, while providing greater flexibility and adaptability to deal with conflict characteristics of different data sets and application scenarios.

[0066] Further, the field name similarity distribution proportion reflects the distribution of each field near the similarity threshold during the field alignment process. When the field name similarity distribution proportion is large, it means that most fields are distributed around the similarity threshold, indicating that the field name similarity threshold may be set unreasonably, resulting in insufficient alignment of field names and affecting the fusion association effect. When the field name distribution proportion is small, it means that most field names are either obviously similar or obviously dissimilar, and the anomaly may be caused by unreasonable clustering parameter settings. By modifying the density clustering algorithm field radius based on the ratio of the coefficient of variation and the preset coefficient of variation, or modifying the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion, adaptive parameter adjustment can be achieved, the problem can be accurately located, and the alignment quality and system stability can be improved. It is simple to operate and easy to implement.

[0067] Further, the coefficient of variation ratio reflects the volatility of the current data. When the coefficient of variation ratio is greater than 1, it means that there is a large fluctuation and conflict in the data fusion process, which may be caused by the too small field radius of the density clustering algorithm. By increasing the field radius, the clustering effect of the field name can be improved, and the conflict in the data fusion process can be reduced. Making the increase amplitude of the field radius proportional to the coefficient of variation ratio can achieve dynamic adjustment, allowing the system to automatically adjust the clustering parameters according to the actual data volatility, thereby better adapting to different data sets and application scenarios. A hierarchical adjustment mechanism is adopted, and different correction thresholds are used for adjustment according to different ranges of the coefficient of variation ratio, which can more flexibly adapt to different data volatility situations, improve the adaptability and robustness of the system, and further improve the quality of field alignment and the stability of data fusion.

[0068] Further, the distribution proportion difference reflects the deviation between the distribution of the boundary scene in the current field name alignment process and the preset target. By reducing the field name similarity threshold, and the reduction amplitude is proportional to the distribution proportion difference, dynamic adjustment can be achieved to improve the clarity and accuracy of alignment. Using a hierarchical adjustment mechanism, different correction thresholds are used for adjustment according to different ranges of the distribution proportion difference, which can more flexibly adapt to different data distribution situations, improve the adaptability and robustness of the system, and further improve the effect of field name alignment and the quality of data fusion.

[0069] Further, the present application provides an AI large model-based knowledge base data fusion association system, which constructs a complete knowledge base data fusion association process through the cooperative work of the acquisition module, the data preprocessing module, the field alignment module, the entity recognition and extraction module, the entity disambiguation and linking module, the triple construction module, the normalization module, the conflict detection and optimization module, the control module and the output module. The modules are closely connected and cooperate with each other, and can efficiently complete a series of operations such as data acquisition, processing, fusion, conflict detection and optimization. Especially the conflict detection and optimization module can dynamically adjust the fusion strategy based on multiple parameters to ensure the quality and consistency of the fused data. In addition, the system uses a graph database to construct a vector index and provides a graph download interface to facilitate users to query and apply the fused knowledge base. This systematic solution not only improves the efficiency and quality of knowledge base data fusion. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 A module block diagram of the AI large model-based knowledge base data fusion association system in the embodiment of the present application;

[0071] Figure 2 A workflow diagram of the AI large model-based knowledge base data fusion association method in the embodiment of the present application;

[0072] Figure 3 A flowchart for determining the reason for not meeting based on the hash conflict rate in the embodiment of the present application;

[0073] Figure 4 A flowchart for correcting the Jaccard similarity threshold based on the hash conflict rate ratio in the embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose and advantages of the present application clearer and more apparent, the present application will be further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0075] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art will appreciate that the embodiments are only used to explain the technical principles of the present application, and are not intended to limit the protection scope of the present application.

[0076] Please refer to Figure 1 The module block diagram of the knowledge base data fusion association system based on an AI large model in the embodiment of the present application is shown, and the knowledge base data fusion association system based on an AI large model comprises a collection module, a data preprocessing module, a field alignment module, an entity recognition and extraction module, an entity disambiguation and linking module, a triple construction module, a normalization module, a conflict detection and optimization module, a control module, and an output module, wherein,

[0077] The collection module comprises a plurality of connectors for collecting the original data.

[0078] The data preprocessing module is connected with the collection module and is used to clean the data and remove duplicate data based on a Jaccard similarity threshold.

[0079] The field alignment module is connected with the data preprocessing module and is used to obtain field names and generate corresponding vectorized semantics, and align the vectorized semantics based on a field name similarity threshold.

[0080] The entity recognition and extraction module is connected with the field alignment module and is used to extract entities and attributes from the cleaned data based on an AI large model, and the extracted entities and attributes are denoted as extracted entities and extracted attributes.

[0081] The entity disambiguation and linking module is connected with the entity recognition and extraction module and is used to generate vectorized semantics corresponding to each extracted entity, and merge the extracted entities based on a density clustering algorithm, and retrieve vectorized entities in a database based on an entity similarity threshold, and record the vectorized entities with a similarity higher than the entity similarity threshold between the extracted entities and the vectorized entities as candidate entities, and if there are a plurality of candidate entities, construct a link between the extracted entity with the highest similarity and the vectorized entity, and record the corresponding extracted entity as a linked successful entity, otherwise, construct a new entity.

[0082] The triple construction module is connected with the entity disambiguation and linking module and is used to generate triples based on an AI large model.

[0083] The normalization module is connected with the triple construction module and is used to assign uniform identifiers to entities and attributes to obtain fused data.

[0084] a conflict detection and optimization module connected with the normalization module, configured to determine whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data, and determine the reason for not meeting the requirements based on the proportion of the repeated hash value of the fused data in the total hash value when it is determined that the requirements are not met, and correct the data preprocessing parameters, field alignment parameters and entity disambiguation and linking parameters based on the corresponding reason;

[0085] a control module connected with the conflict detection and optimization module, the data preprocessing module, the field alignment module and the entity disambiguation and linking module, configured to schedule the corresponding modules to execute according to the instructions of the conflict detection and optimization module;

[0086] an output module connected with the normalization module and the conflict detection and optimization module, configured to construct a vector index based on a graph database and provide a download interface of the graph.

[0087] In principle, the way of cleaning the original data in the data preprocessing stage is not limited, and the skilled person can select different data cleaning methods according to the characteristics of the data, for example, uniform character encoding to UTF-8, removing non-printable characters, replacing special white space characters (multiple spaces, tabs) with single spaces, uniform decimal point representation and date representation, etc., which will not be repeated here.

[0088] The process of removing duplicate data attribute values based on the Jaccard similarity threshold in the data preprocessing stage includes:

[0089] The AI large model is used to extract entities from the cleaned data, and each entity is taken as a set, and then the intersection and union of each entity are calculated;

[0090] The ratio of the size of the intersection (the number of elements in the set) to the size of the union is called the Jaccard similarity;

[0091] When the Jaccard similarity is greater than the set Jaccard similarity threshold, it is considered that the two are repeated, and any one of the two sets is deleted, otherwise the calculation continues;

[0092] The Jaccard similarity threshold refers to the minimum required similarity threshold between any two data attribute values when judging the repetition of the data attribute values. In the field of financial data fusion or medical data fusion, the Jaccard similarity threshold is usually set between 0.7 and 0.9. Therefore, in this scheme, the value range of the Jaccard similarity threshold is set to 0.7-0.9, in order to improve the accuracy, consistency and reliability of data fusion and reduce data conflicts;

[0093] In the field alignment stage and the entity disambiguation and linking stage, the manner of generating corresponding vectorized semantics for each field name and extracted entity is not limited in principle, and the skilled person can convert each word in the field name into a vector through a pre-trained language model (such as Word2Vec, GloVe, etc.), and then aggregate the vectors of all words in the field name to obtain a vector representation of the field name, or directly use natural language processing (NLP) technology to generate corresponding vectorized semantics, which is not described here.

[0094] In the field alignment stage and the entity disambiguation and linking stage, the manner of generating corresponding vectorized semantics for each field name and extracted entity is not limited in principle, and the skilled person can convert each word in the field name into a vector through a pre-trained language model (such as Word2Vec, GloVe, etc.), and then aggregate the vectors of all words in the field name to obtain a vector representation of the field name, or directly use natural language processing (NLP) technology to generate corresponding vectorized semantics, which is not described here.

[0095] In the entity recognition and extraction process, the process of extracting entities and attributes using AI large models includes:

[0096] Any AI large model can be selected, including but not limited to any one or several of the OpenAI series models, the Anthropic Claude series models, the Google Gemini series, xAI Grok-2, DeepSeek, Kimi, Claude, etc., which are not described here.

[0097] The preprocessed data is input into the selected AI large model to extract entities and attributes through the AI large model.

[0098] In the entity disambiguation and linking stage, the extracted entities are merged using a density-based density clustering algorithm. The density-based density clustering algorithm has the advantages of strong ability to handle noise and outliers, discovery of clusters of any shape, no need to specify the number of clusters in advance, adaptation to clusters of different densities, high computational efficiency, and relatively simple parameter selection, etc. These advantages make the density-based density clustering algorithm perform well in processing complex data distribution and large-scale data sets, and can more accurately identify and link entities, thereby improving the overall quality and reliability of knowledge base data fusion and association. The type of density clustering algorithm is not limited in principle, and the skilled person can select any one of DBSCAN and OPTICS. The radius of the density clustering algorithm is not limited in principle, and can be derived according to the actual situation or K-distance diagram.

[0099] The entity similarity threshold is the similarity required between the extracted entity and the vectorized entity when the vectorized entity is recorded as a candidate entity. The value range of the entity similarity threshold is set to 0.8-0.95 to ensure that the link constructed between the extracted entity and the vectorized entity is reasonable enough.

[0100] The process of generating triplets based on the AI large model includes:

[0101] The data after completing entity disambiguation and linking is input into the AI large model;

[0102] The AI large model identifies entities, relationships and attributes;

[0103] The AI large model outputs: entity pair and relationship triplets (format: entity 1, relationship, entity 2) and entity and attribute triplets (format: entity, attribute, attribute value);

[0104] The vector index construction and output stage includes the process of constructing a vector index using a graph database and providing a download interface of the graph atlas:

[0105] Select a graph database, which is not limited in principle, including but not limited to Neo4j, OrientDB, ArangoDB, etc.

[0106] Convert the fused data into a format supported by the graph database and import the data into the database;

[0107] Generate a vector representation for each entity and relationship;

[0108] Use the index function of the graph database or external index tools (such as FAISS, HNSW, etc.) to construct a vector index;

[0109] Develop an API interface to allow users to query and download graph atlas data.

[0110] Further, please refer to Figure 2 As shown in the figure, it is a workflow diagram of the knowledge base data fusion association method based on the AI large model in the embodiment of the application, and the knowledge base data fusion association method based on the AI large model comprises:

[0111] S1: The acquisition module acquires the original data based on the connector;

[0112] S2: The data preprocessing module cleans the original data and removes duplicate data attribute values based on the Jaccard similarity threshold;

[0113] S3: The field alignment module obtains each field name and generates corresponding vectorized semantics, and aligns each vectorized semantics based on the field name similarity threshold;

[0114] S4: The entity recognition and extraction module uses an AI large model to extract entities and attributes from the cleaned data and records the extracted entities and extracted attributes;

[0115] S5: The entity disambiguation and linking module generates the vectorized semantics corresponding to each extracted entity, then merges the extracted entities based on the density clustering algorithm, and retrieves the vectorized entities in the database based on the entity similarity threshold, records the vectorized entities with similarity higher than the entity similarity threshold between the extracted entities and the vectorized entities as candidate entities, if there are several candidate entities, constructs a link between the extracted entity and the vectorized entity with the highest similarity, and records the corresponding extracted entity as a linked entity, otherwise, a new entity is constructed;

[0116] S6: The triple construction module generates triples based on an AI large model;

[0117] S7: The normalization module assigns a uniform identifier to the entity and the attribute, and obtains the fused data;

[0118] S8: The conflict detection and optimization module determines whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data, and determines the reason for not meeting the requirements based on the proportion of the repeated hash value of the fused data in the total hash value when it is determined that the requirements are not met, and modifies the data preprocessing parameters, field alignment parameters and entity disambiguation and linking parameters based on the corresponding reason;

[0119] S9: The control module schedules the corresponding modules for execution according to the instructions of the conflict detection and optimization module;

[0120] S10: The output module constructs a vector index using a graph database and provides a download interface for the graph.

[0121] Further, the process of determining whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data includes:

[0122] Obtain the original data and the fused data and vectorize them, and align the original data and the fused data;

[0123] Calculate the absolute difference value of each field one by one, generate a difference value sequence, and determine the coefficient of variation of the original data and the fused data based on the difference sequence;

[0124] If the coefficient of variation is less than or equal to the set preset coefficient of variation, it is determined that the current data conflict meets the requirements, and the knowledge base data fusion association method parameters remain unchanged;

[0125] If the coefficient of variation is greater than the preset coefficient of variation, it is determined that the current data conflict does not meet the requirements, and the reason for not meeting the requirements is determined based on the proportion of the repeated hash value of the fused data in the total hash value.

[0126] The cross-source data usually has different scales, different units and different field distributions. The coefficient of variation (CV) is the ratio of the standard deviation to the mean, which is scale-independent, and thus is suitable for comparing the differences of multi-field and different scale data in cross-source data. By calculating the CV of the difference sequence of the original data and the fused data, the overall fluctuation level of the fusion process can be quantified. A low coefficient of variation usually indicates that the difference distribution of the fused data is uniform and stable, and the conflict resolution is reasonable. A high coefficient of variation implies that there may be problems such as incorrect conflict processing, information loss or mismatch in the fusion process, which need to be further investigated. Combined with the proportion of repeated hash values, the conflict source can also be located, such as incomplete deduplication and merging conflicts. Using the coefficient of variation to judge whether the conflict of the fused data is reasonable can provide a quick and quantifiable conflict stability evaluation benchmark, thereby quickly judging the quality of the fused result.

[0127] Specifically, the process of determining whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data includes:

[0128] The process of obtaining the original data and the fused data and vectorizing, and aligning the original data and the fused data includes:

[0129] The conflict detection and optimization module obtains the field names of the original data and the fused data and generates corresponding vectorized semantics, and aligns each vectorized semantics based on a field alignment threshold. In principle, the field alignment threshold AT is not limited, and a technician can adjust it according to needs. In this scheme, the value range of the field alignment threshold AT can be set to 0.85-0.95 to avoid misalignment.

[0130] The conflict detection and optimization module extracts the original data entities and attributes and vectorizes them using an AI large model to obtain original data entity vectors and original entity attribute vectors. The original data entity vectors and original entity attribute vectors are aligned with the extracted entities and their corresponding attributes. The value range of the entity vector alignment threshold is 0.85-0.95 to strictly avoid misalignment, and the value range of the entity vector alignment threshold is 0.75-0.85 to balance efficiency and accuracy.

[0131] Absolute difference value AD i = | original data attribute RDA i - fused data attribute FDA i |, where i = 1, 2,..., n.

[0132] The absolute difference value of each attribute is calculated piece by piece to generate a difference value sequence AD1, AD2, · ···, AD n .

[0133] determine a coefficient of variation of the original data and the fused data based on the difference sequence;

[0134] coefficient of variation

[0135] wherein,

[0136]

[0137] The conflict detection and optimization module compares the coefficient of variation CV with a preset coefficient of variation CV1, wherein the preset coefficient of variation CV1 is set to be in the range of [0.05, 0.15], and for the coefficient of variation CV, when the coefficient of variation is less than or equal to 0.15, the risk of data conflict is low, and for financial, medical data and other databases, higher requirements are required, and when the coefficient of variation is 0.05, the embodiment of the application can not only avoid data conflict, but also better play the role of data fusion to provide support for user decision-making;

[0138] If the coefficient of variation CV is less than or equal to the preset coefficient of variation CV1, the conflict detection and optimization module determines that the current data conflict meets the requirements, and maintains the knowledge base data fusion association method parameter unchanged.

[0139] If the coefficient of variation CV is greater than the preset coefficient of variation CV1, the conflict detection and optimization module determines that the current data conflict does not meet the requirements, and the conflict detection and optimization module determines the reason based on the proportion of the repeated hash value of the fused data in the total hash value.

[0140] Further, the process of determining the reason based on the proportion of the repeated hash value of the fused data in the total hash value includes:

[0141] Converting the fused data into hash values;

[0142] Determining a hash conflict rate based on the proportion of the number of repeated hash values in the fused data in the total number of hash values;

[0143] If the hash conflict rate is less than or equal to the preset hash conflict rate, the reason is determined based on the proportion of the linked successful entities in the extracted entities;

[0144] If the hash conflict rate is greater than the preset hash conflict rate, it is determined that the Jaccard similarity threshold is not reasonable, and the Jaccard similarity threshold is corrected based on the ratio of the hash conflict rate to the preset hash conflict rate;

[0145] The Jaccard similarity threshold refers to the lowest Jaccard similarity between the pair of data attribute values when determining that the data attribute values are repeated.

[0146] A high degree of repetition means that there is more inconsistent information and potential conflict among multiple similar or repeated data, making it difficult to eliminate contradictions in the fusion process, and the possibility and severity of conflict will significantly increase. Therefore, controlling and reasonably processing data repetition during data fusion is the key to ensuring fusion quality and accuracy. In the present scheme, the proportion of the repeated hash value of the fused data in the total hash value (hash collision rate) can be calculated to effectively determine the cause of the conflict in the data fusion process. The high and low of the hash collision rate reflects the degree of data repetition or similar records, which can help the present scheme quickly locate the abnormal cause. When the hash collision rate is higher than the preset threshold, it indicates that the Jaccard similarity threshold is set too high, resulting in insufficient deduplication. At this time, the Jaccard similarity threshold needs to be corrected to optimize the deduplication strategy. This method has the advantages of efficient calculation, easy quantification, and direct reflection of data repetition and fusion effect, and can efficiently, objectively and automatically identify and handle fusion conflicts, improve the accuracy and robustness of knowledge base data fusion, and is a reasonable and effective technical solution.

[0147] Referring to Figure 3 The process of determining the cause of non-satisfaction based on the proportion of the repeated hash value of the fused data in the total hash value in the present embodiment includes:

[0148] The conflict detection and optimization module converts the fused data into a hash value through a hash algorithm, wherein the hash algorithm is not limited in principle, and a technician can select any one of MD5, SHA-1, SHA-256, etc. to convert the fused data into a hash value.

[0149] The conflict detection and optimization module determines the hash collision rate HCR based on the proportion of the number of repeated hash values in the fused data in the total number of hash values.

[0150] The conflict detection and optimization module compares the hash collision rate HCR with a preset hash collision rate HCR1, wherein the preset hash collision rate HCR1 is set to be in the range of [3%, 15%], and the hash algorithm characteristics, actual application experience and performance trade-off are considered in determining the preset hash collision rate HCR1. For high-precision knowledge bases (such as medical and financial fields), a lower hash collision rate is required, and the preset hash collision rate HCR1 is usually guaranteed to be in the range of 3%-5% to ensure the result accuracy. For text processing (5%-10%) and image processing databases (10%-15%, because the visual features themselves have a certain fuzziness), a slightly higher hash collision rate can be accepted. Therefore, in this scheme, the preset hash collision rate is set to be in the range of 3%-15%.

[0151] If the hash collision rate HCR is less than or equal to the preset hash collision rate HCR1, the conflict detection and optimization module determines that the reason is not satisfied based on the proportion of the linked successful entities in the extracted entities.

[0152] If the hash collision rate HCR is greater than the preset hash collision rate HCR1, the conflict detection and optimization module determines that the Jaccard similarity threshold is not reasonable, and modifies the Jaccard similarity threshold based on the ratio of the hash collision rate to the preset hash collision rate.

[0153] Further, the process of modifying the Jaccard similarity threshold based on the ratio of the hash collision rate to the preset hash collision rate includes:

[0154] Determining a hash collision rate ratio based on the ratio of the hash collision rate to the preset hash collision rate.

[0155] Reducing the Jaccard similarity threshold based on the hash collision rate ratio, and the reduction amplitude of the Jaccard similarity threshold is proportional to the hash collision rate ratio.

[0156] The hash collision rate refers to the proportion of the number of repeated hash values in the total hash values in the fused data. The hash collision rate ratio refers to the ratio of the hash collision rate to the preset hash collision rate. The hash collision rate ratio reflects the deviation degree of the current hash collision rate from the preset value. A higher ratio means that there are more repeated data that have not been correctly processed in the data fusion process. Therefore, by reducing the Jaccard similarity threshold, the probability of data being identified as repeated can be increased, thereby reducing the hash collision rate and improving the quality of data fusion. At the same time, this dynamic adjustment method based on the hash collision rate ratio can realize self-adaptive optimization of the system without frequent manual intervention, thereby improving the automation degree and efficiency of the system, making the system better adapt to the dynamic changes of data, and maintaining the stability and reliability of data fusion.

[0157] Referring to Figure 4 Fig. 1 is a flowchart of the process of correcting the Jaccard similarity threshold based on the ratio of the hash collision rate according to an embodiment of the present application, and Fig. 2 is a flowchart of the process of correcting the Jaccard similarity threshold based on the ratio of the hash collision rate according to another embodiment of the present application. The process of correcting the Jaccard similarity threshold based on the ratio of the hash collision rate in the present application includes:

[0158] The collision detection and optimization module determines a hash collision rate ratio A based on the ratio of the hash collision rate to the preset hash collision rate, and compares the hash collision rate ratio A with a set first preset hash collision rate ratio A1 and a second preset hash collision rate ratio A2, where the first preset hash collision rate ratio A1 ∈ (1, 2) and the second preset hash collision rate ratio A2 ∈ [2, 3]. Setting the first preset hash collision rate ratio A1 ∈ (1, 2) and the second preset hash collision rate ratio A2 ∈ [2, 3] helps to achieve a hierarchical adjustment mechanism while more finely controlling the adjustment range of the Jaccard similarity threshold according to the actual hash collision severity. When the hash collision rate ratio A obtained by actual calculation is between 1 and 2, only a slight adjustment of the Jaccard similarity threshold is required at this time. When the hash collision rate ratio A reaches 2, it means that the actual hash collision rate has become twice the ideal hash collision rate, and further escalation of the adjustment mechanism is required at this time. When the value is 3, it may mean that more serious data collisions may occur in the system, and greater adjustment is required at this time. This hierarchical setting can more accurately respond to different degrees of data collision, avoid misjudgment due to excessive adjustment, and at the same time provide greater flexibility and adaptability to respond to the collision characteristics of different data sets and application scenarios.

[0159] If the hash collision rate ratio A is less than or equal to the first preset hash collision rate ratio A1, the collision detection and optimization module corrects the Jaccard similarity threshold JST using a first Jaccard similarity correction threshold a1, and the corrected Jaccard similarity threshold JST' = JST x a1, where the first Jaccard similarity correction threshold a1 is set to 0.98.

[0160] If the hash collision rate ratio A is greater than the first preset hash collision rate ratio A1 and less than or equal to the second preset hash collision rate ratio A2, the collision detection and optimization module corrects the Jaccard similarity threshold JST using a second Jaccard similarity correction threshold a2, and the corrected Jaccard similarity threshold JST' = JST x a2, where the second Jaccard similarity correction threshold a2 is set to 0.95.

[0161] If the hash collision rate ratio A is greater than the second preset hash collision rate ratio A2, the conflict detection and optimization module corrects the Jaccard similarity threshold JST by a third Jaccard similarity correction threshold a3, and the corrected Jaccard similarity threshold JST' = JST x a3, where the third Jaccard similarity correction threshold a3 = 0.9 is set.

[0162] Further, the process of determining the reason for not meeting the requirement based on the proportion of the linked successful entities in the extracted entities includes:

[0163] Determining an extracted entity linking success rate based on the proportion of the linked successful entities in the extracted entities.

[0164] If the extracted entity linking success rate is less than or equal to a preset extracted entity linking success rate, it is determined that the entity similarity threshold setting is unreasonable, and the entity similarity threshold is corrected based on the ratio of the preset extracted entity linking success rate to the extracted entity linking success rate.

[0165] If the extracted entity linking success rate is greater than the preset extracted entity linking success rate, the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities is determined as the reason for not meeting the requirement.

[0166] The field name similarity range refers to a similarity range uniformly distributed on both sides of the field name similarity threshold, including the field name similarity threshold.

[0167] The extracted entity linking success rate refers to the proportion of successfully linked vectorized entities in the database among all extracted entities, which reflects the success degree of matching extracted entities with entities in the database in the process of entity disambiguation and linking. If the extracted entity linking success rate is low, it means that many extracted entities have not been successfully linked to vectorized entities in the database in the process of entity disambiguation and linking, which may be because the entity similarity threshold is set too high, resulting in some actually similar entities not being correctly identified as matching entities. The extracted entity linking success rate provides a quantitative index, which can intuitively evaluate the effect of the entity disambiguation and linking process, accurately locate the abnormal reason, and further improve the performance and reliability of the entire knowledge base data fusion association method.

[0168] Specifically, the process of determining the reason for not meeting the requirement based on the proportion of the linked successful entities in the extracted entities includes:

[0169] The conflict detection and optimization module determines an extracted entity linking success rate B based on the proportion of the linked successful entities in the extracted entities.

[0170] The conflict detection and optimization module compares the extraction entity linking success rate B with a set preset extraction entity linking success rate B1, wherein the preset extraction entity linking success rate B1 is set to be in a range of [80%, 90%], according to the KDD 2024 benchmark test, in the medical knowledge base fusion test, a linking success rate of 88% can ensure that the entity disambiguation accuracy is greater than 92%, and for the knowledge fusion test in the social media field, a slightly lower entity linking success rate (80%-85%) can be accepted, therefore, in the present scheme, the preset extraction entity linking success rate B1 is set to be in a range of [80%, 90%];

[0171] If the extraction entity linking success rate B is less than or equal to the preset extraction entity linking success rate B1, it is judged that the entity similarity threshold setting is unreasonable, and the conflict detection and optimization module corrects the entity similarity threshold based on the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate.

[0172] If the extraction entity linking success rate B is greater than the preset extraction entity linking success rate B1, the conflict detection and optimization module judges that the reason is not met based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities.

[0173] Further, the process of correcting the entity similarity threshold based on the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate includes:

[0174] Determining an entity linking success rate ratio based on the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate.

[0175] Lowering the entity similarity threshold based on the entity linking success rate ratio, and the lowering amplitude of the entity similarity threshold is proportional to the entity linking success rate ratio.

[0176] The entity linking success rate ratio refers to the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate, and the entity linking success rate ratio reflects the deviation degree of the current linking success rate from the expected value. When the entity linking success rate ratio is high, it means that the actual linking success rate is much lower than the preset value, which means that the current entity similarity threshold is too high and needs to be lowered to improve the linking success rate. By making the lowering amplitude proportional to the entity linking success rate ratio, dynamic adjustment can be achieved, so that the system can better adapt to changes in data, improve the linking success rate, reduce misjudgment, improve system performance, and has high interpretability and operability.

[0177] Specifically, the process of correcting the entity similarity threshold based on the ratio of the preset extraction entity linking success rate to the extraction entity linking success rate includes:

[0178] The conflict detection and optimization module determines an entity linking success rate ratio C based on a ratio of the preset extraction entity linking success rate to the extraction entity linking success rate;

[0179] The conflict detection and optimization module compares the entity linking success rate ratio C with a set first preset entity linking success rate ratio C1 and a second preset entity linking success rate ratio C2, wherein the first preset entity linking success rate ratio C1 is set to be in a range of [1.05, 1.2), and the second preset entity linking success rate ratio C2 is set to be in a range of [1.2, 1.4]. According to database system running data monitoring, when the entity linking success rate is lower than 75%, significant data conflict problems will be caused, and when the linking rate is lower than 70%, the conflict will be more serious. Since the entity linking success rate ratio C is a ratio of the preset extraction entity linking success rate to the extraction entity linking success rate, in the present scheme, the first preset entity linking success rate ratio C1 is set to be in a range of [1.05, 1.2), and the second preset entity linking success rate ratio C2 is set to be in a range of [1.2, 1.4].

[0180] If the entity linking success rate ratio C is less than or equal to the first preset entity linking success rate ratio C1, the conflict detection and optimization module corrects the entity similarity threshold EST by using a first entity similarity correction threshold β1, and a corrected entity similarity threshold EST' = β1 × EST, wherein the first entity similarity correction threshold β1 is set to be 0.99.

[0181] If the entity linking success rate ratio C is greater than the first preset entity linking success rate ratio C1 and less than or equal to the second preset entity linking success rate ratio C2, the conflict detection and optimization module corrects the entity similarity threshold EST by using a second entity similarity correction threshold β2, and a corrected entity similarity threshold EST' = β2 × EST, wherein the second entity similarity correction threshold β2 is set to be 0.96.

[0182] If the entity linking success rate ratio C is greater than the second preset entity linking success rate ratio C2, the conflict detection and optimization module corrects the entity similarity threshold EST by using a third entity similarity correction threshold β3, and a corrected entity similarity threshold EST' = β3 × EST, wherein the third entity similarity correction threshold β3 is set to be 0.92.

[0183] Further, the process of determining the reason for non-compliance based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities includes:

[0184] The proportion of field name similarity distribution is determined based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities.

[0185] If the field name similarity distribution proportion is less than or equal to the set preset field name similarity distribution proportion, it is judged that the density clustering algorithm parameter setting is unreasonable, and the density clustering algorithm field radius is corrected based on the ratio of the coefficient of variation to the preset coefficient of variation;

[0186] If the field name similarity distribution proportion is greater than the preset field name similarity distribution proportion, it is judged that the field name similarity threshold setting is unreasonable, and the field name similarity threshold is corrected based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion;

[0187] The field name similarity threshold refers to the minimum similarity required for the field name to be aligned to the corresponding vectorized semantics.

[0188] The field name similarity distribution proportion refers to the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities, reflecting the distribution of field name similarities near the similarity threshold in the field alignment process. When the field name similarity distribution proportion is high, it means that most fields are distributed around the similarity threshold, which means that the field name similarity threshold may be set unreasonably, resulting in that the field names are not aligned, and thus affecting the fusion association effect. When the field name distribution proportion is low, it means that most field names are either obviously similar or obviously dissimilar, and at this time the abnormality may come from unreasonable clustering parameter setting, that is, the clustering radius and other parameters do not properly cluster similar field names into the same cluster, resulting in a decline in entity disambiguation effect. By judging the abnormality reason based on the field name similarity distribution proportion, data-driven adaptive parameter adjustment can be realized, the problem can be accurately located, the alignment quality and system stability can be improved, and the operation is simple and easy to implement.

[0189] Specifically, the process of judging the reason for inconsistency based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities includes:

[0190] The conflict detection and optimization module determines the field name similarity distribution proportion D based on the proportion of the number of field name similarities distributed in the field name similarity range in the total number of field name similarities, wherein the field name similarity range = field name similarity threshold FNST ± γ, γ ∈ [0.05, 0.2];

[0191] The conflict detection and optimization module compares the field name similarity distribution proportion D with a preset field name similarity distribution proportion D1, where the preset field name similarity distribution proportion D1 is set to be in the range of [40%, 50%]. If the field name similarity distribution is relatively uniform or concentrated at both ends (0 and 1), it indicates that the field name similarity threshold is set reasonably. When the field name similarity is concentrated on both sides of the field name similarity threshold, it may be that the field name similarity threshold is not set reasonably. When the field name similarity distribution proportion reaches 40%, the number of field names within the field name similarity range accounts for two-fifths of the total, indicating that the field name similarity threshold is set unreasonably. Therefore, in order to ensure flexibility, the preset field name similarity distribution proportion D1 is set to be in the range of [40%, 50%] in the present scheme.

[0192] If the field name similarity distribution proportion D is less than or equal to the preset field name similarity distribution proportion D1, the conflict detection and optimization module determines that the density clustering algorithm parameter setting is unreasonable, and corrects the density clustering algorithm field radius based on the ratio of the coefficient of variation to the preset coefficient of variation.

[0193] If the field name similarity distribution proportion D is greater than the preset field name similarity distribution proportion D1, the conflict detection and optimization module determines that the field name similarity threshold setting is unreasonable, and corrects the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion.

[0194] Further, the process of correcting the density clustering algorithm field radius based on the ratio of the coefficient of variation to the preset coefficient of variation includes:

[0195] Determining a coefficient of variation ratio based on the ratio of the coefficient of variation to the preset coefficient of variation.

[0196] Increasing the density clustering algorithm field radius based on the coefficient of variation ratio, and the increasing amplitude of the density clustering algorithm field radius is proportional to the coefficient of variation ratio.

[0197] The coefficient of variation ratio refers to the ratio of the coefficient of variation to the preset coefficient of variation, which reflects the fluctuation degree of the current data. When the coefficient of variation ratio is greater than 1, it indicates that there is a large fluctuation and conflict in the data fusion process, which may be caused by a too small field radius of the density clustering algorithm. By increasing the field radius, the clustering effect of the field name can be improved, and the conflict in the data fusion process can be reduced. Making the increasing amplitude of the field radius proportional to the coefficient of variation ratio can achieve dynamic adjustment, so that the system can automatically adjust the clustering parameters according to the fluctuation of the actual data, thereby better adapting to different data sets and application scenarios.

[0198] Specifically, the process of correcting the density clustering algorithm field radius based on the ratio of the coefficient of variation to the preset coefficient of variation includes:

[0199] The conflict detection and optimization module determines a coefficient of variation ratio E based on the ratio of the coefficient of variation to the preset coefficient of variation.

[0200] The conflict detection and optimization module compares the coefficient of variation ratio E with a set first preset coefficient of variation ratio E1 and a second preset coefficient of variation ratio E2, where the first preset coefficient of variation ratio E1 is set to be in the range of (1, 2), and the second preset coefficient of variation ratio E2 is set to be in the range of [2, 4]. According to relevant research in the field of data fusion, when the coefficient of variation of original data and fused data exceeds 0.2, there may be a large risk of data conflict, and when the coefficient of variation exceeds 0.3, it may indicate that there is a serious data conflict. Therefore, in this scheme, the first preset coefficient of variation ratio E1 is set to be in the range of (1, 2), and the second preset coefficient of variation ratio E2 is set to be in the range of [2, 4].

[0201] If the coefficient of variation ratio E is less than or equal to the first preset coefficient of variation ratio E1, the conflict detection and optimization module corrects the density clustering algorithm field radius R using a first field radius correction threshold θ1, and the corrected density clustering algorithm field radius R' = R x θ1, where the first field radius correction threshold θ1 is set to be 1.02.

[0202] If the coefficient of variation ratio E is greater than the first preset coefficient of variation ratio E1 and less than or equal to the second preset coefficient of variation ratio E2, the conflict detection and optimization module corrects the density clustering algorithm field radius R using a second field radius correction threshold θ2, and the corrected density clustering algorithm field radius R' = R x θ2, where the second field radius correction threshold θ2 is set to be 1.05.

[0203] If the coefficient of variation ratio E is greater than the second preset coefficient of variation ratio E2, the conflict detection and optimization module corrects the density clustering algorithm field radius R using a third field radius correction threshold θ3, and the corrected density clustering algorithm field radius R' = R x θ3, where the third field radius correction threshold θ3 is set to be 1.09.

[0204] Further, the process of correcting the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion includes:

[0205] Determine a distribution proportion difference based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion.

[0206] The field name similarity threshold is reduced based on the distribution proportion difference value, and the reduction range of the field name similarity threshold is proportional to the distribution proportion difference value.

[0207] The distribution proportion difference value is the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion, reflecting the deviation between the distribution of the boundary scene in the current field name alignment process and the preset target. When the distribution proportion difference value is positive, it means that the current field name similarity threshold is set too low, resulting in a large number of field names being near the similarity threshold, and the alignment determination has a high uncertainty. Therefore, the field name similarity threshold needs to be reduced, and the reduction range is proportional to the distribution proportion difference value, to realize dynamic adjustment and improve the clarity and accuracy of alignment.

[0208] Specifically, the process of correcting the field name similarity threshold based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion includes:

[0209] The conflict detection and optimization module determines the distribution proportion difference value F based on the difference between the field name similarity distribution proportion and the preset field name similarity distribution proportion.

[0210] The conflict detection and optimization module compares the distribution proportion difference value F with the set first preset distribution proportion difference value F1 and the second preset distribution proportion difference value F2, where the first preset distribution proportion difference value F1 is set to be in the range of [0, 10%], and the second preset distribution proportion difference value F2 is set to be in the range of (10%, 20%]. Setting the first preset distribution proportion difference value F1 in the range of [0, 10%] and the second preset distribution proportion difference value F2 in the range of (10%, 20%) can more flexibly adapt to different data distribution situations, improving the adaptability and robustness of the system. Through experimental verification and data distribution characteristic analysis, reasonable F1 and F2 values can be determined, thereby ensuring that the system still maintains good performance when facing different data sets;

[0211] If the distribution proportion difference value F is less than or equal to the first preset distribution proportion difference value F1, the conflict detection and optimization module corrects the field name similarity threshold FNST using the first field name correction threshold λ1, and the corrected field name similarity threshold FNST' = λ1 x FNST, where the first field name correction threshold λ1 is set to be 0.99.

[0212] If the distribution proportion difference value F is greater than the first preset distribution proportion difference value F1 and less than or equal to the second preset distribution proportion difference value F2, the conflict detection and optimization module corrects the field name similarity threshold FNST using the second field name correction threshold λ2, and the corrected field name similarity threshold FNST' = λ2 x FNST, where the second field name correction threshold λ2 is set to be 0.95.

[0213] If the distribution ratio difference F is greater than the second preset distribution ratio difference F2, the conflict detection and optimization module adopts a third field name correction threshold value λ3 to correct the field name similarity threshold value FNST, and the corrected field name similarity threshold value FNST' = λ3 x FNST, wherein the third field name correction threshold value λ3 is set to 0.9.

[0214] 3 = 0.9.

[0215] In summary, the present application provides an AI large model-based knowledge base data fusion association method and system, which realizes efficient fusion of knowledge base data through a series of steps such as data acquisition, preprocessing, field alignment, entity recognition and extraction, disambiguation and linking, relationship extraction and triple construction, normalization, conflict detection and optimization, etc. The method can effectively eliminate data redundancy and inconsistency, fill in missing data, and improve data accuracy, thereby improving data quality. At the same time, the introduction of intelligent conflict detection and optimization mechanism can specifically identify the causes of data conflicts and give corresponding solutions, overcoming the problem of difficult data conflict resolution in existing database fusion methods. Finally, through vector index construction and output, the user is provided with normalized, easy-to-use and shared fused data, promoting the development of knowledge base data fusion technology.

[0216] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the drawings, but those skilled in the art will readily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without deviating from the principles of the present application, and the technical solutions after such changes or replacements will fall within the protection scope of the present application.

Claims

1. A knowledge base data fusion and association method based on an AI large-scale model, characterized in that, include, Data acquisition: Raw data is collected via connectors. Data preprocessing involves cleaning the original data and removing duplicates from the cleaned data based on the Jaccard similarity threshold. Field alignment involves obtaining the names of each field and generating corresponding vectorized semantics, and then aligning the vectorized semantics based on the field name similarity threshold. Entity recognition and extraction: AI large model is used to extract entities and attributes from the cleaned data and record them as extracted entities and extracted attributes. Entity disambiguation and linking: Generate corresponding vectorized semantics for each extracted entity; merge extracted entities based on density clustering algorithm; retrieve vectorized entities from the database based on entity similarity threshold; and record vectorized entities whose similarity with the extracted entity is higher than the entity similarity threshold as candidate entities. If there are several candidate entities, a link is constructed between the extracted entity with the highest similarity and the vectorized entity, and the corresponding extracted entity is recorded as a successfully linked entity; otherwise, a new entity is constructed. Relation extraction and triple construction: generating triples based on a large AI model; Standardization and normalization assign unified identifiers to entities and attributes to obtain merged data. Conflict detection and optimization: Based on the coefficient of variation of the original data and the fused data, determine whether the current data conflict meets the requirements; and when it is determined that the requirements are not met, determine the reason for the non-compliance based on the proportion of duplicate hash values ​​in the total hash values ​​of the fused data; and, based on the corresponding reason, correct the data preprocessing parameters, field alignment parameters, entity disambiguation and linking parameters. Vector index construction and output: The vector index is constructed using a graph database, and a graph download interface is provided.

2. The knowledge base data fusion and association method according to claim 1, characterized in that, The process of determining whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data includes: The original data and the fused data are obtained and quantized, and the original data and the fused data are aligned. The absolute difference value of each field is calculated one by one to generate a difference value sequence, and the coefficient of variation of the original data and the fused data is determined based on the difference sequence; If the coefficient of variation is less than or equal to the preset coefficient of variation, it is determined that the current data conflict meets the requirements, and the parameters of the knowledge base data fusion and association method remain unchanged. If the coefficient of variation is greater than the preset coefficient of variation, it is determined that the current data conflict does not meet the requirements, and the reason for not meeting the requirements is determined based on the proportion of the duplicate hash value of the merged data in the total hash value.

3. The knowledge base data fusion and association method according to claim 2, characterized in that, The process of determining the reasons for dissatisfaction based on the proportion of duplicate hash values ​​in the total hash value after fusion includes: The fused data is then converted into hash values. The hash collision rate is determined based on the proportion of duplicate hash values ​​in the total number of hash values ​​in the merged data. If the hash collision rate is less than or equal to the preset hash collision rate, the reason for not meeting the requirement is determined based on the proportion of successfully linked entities in the extracted entities. If the hash collision rate is greater than the preset hash collision rate, it is determined that the Jaccard similarity threshold setting is unreasonable, and the Jaccard similarity threshold is corrected based on the ratio of the hash collision rate to the preset hash collision rate. The Jaccard similarity threshold refers to the minimum Jaccard similarity required between any two data attribute values ​​when they are considered to be duplicated.

4. The knowledge base data fusion and association method according to claim 3, characterized in that, The process of adjusting the Jaccard similarity threshold based on the ratio of the hash collision rate to the preset hash collision rate includes: The hash collision rate ratio is determined based on the ratio of the hash collision rate to the preset hash collision rate; The Jaccard similarity threshold is reduced based on the hash collision rate ratio, and the reduction in the Jaccard similarity threshold is proportional to the hash collision rate ratio.

5. The knowledge base data fusion and association method according to claim 3, characterized in that, The process of determining the reason for dissatisfaction based on the proportion of successfully linked entities in the extracted entities includes: The success rate of linking entities is determined based on the proportion of successfully linked entities in the extracted entities. If the success rate of extracting entity links is less than or equal to the preset success rate of extracting entity links, it is determined that the entity similarity threshold setting is unreasonable, and the entity similarity threshold is corrected based on the ratio of the preset success rate of extracting entity links to the success rate of extracting entity links. If the success rate of extracting entity links is greater than the preset success rate of extracting entity links, the reason for non-compliance is determined based on the proportion of the number of field name similarities distributed in the field name similarity range to the total number of field name similarities. Wherein, the entity similarity threshold refers to the similarity that the extracted entity and the vectorized entity need to reach when the vectorized entity is recorded as a candidate entity, and the field name similarity range refers to the similarity range that is evenly distributed on both sides of the field name similarity threshold and includes the field name similarity threshold.

6. The knowledge base data fusion and association method according to claim 5, characterized in that, The process of adjusting the entity similarity threshold based on the ratio of the preset entity link success rate to the entity link success rate includes: The entity link success rate ratio is determined based on the ratio of the preset entity link extraction success rate to the entity link extraction success rate. The entity similarity threshold is reduced based on the entity link success rate ratio, and the reduction in the entity similarity threshold is proportional to the entity link success rate ratio.

7. The knowledge base data fusion and association method according to claim 5, characterized in that, The process of determining the reasons for non-compliance based on the proportion of the number of field name similarities distributed within the range of field name similarities to the total number of field name similarities includes: The distribution ratio of field name similarity is determined based on the proportion of the number of field name similarities distributed within the range of field name similarities in the total number of field name similarities; If the proportion of field name similarity distribution is less than or equal to the preset proportion of field name similarity distribution, it is determined that the density clustering algorithm parameter setting is unreasonable, and the density clustering algorithm neighborhood radius is corrected based on the ratio of the coefficient of variation to the preset coefficient of variation. If the proportion of field name similarity distribution is greater than the proportion of preset field name similarity distribution, it is determined that the field name similarity threshold setting is unreasonable, and the field name similarity threshold is corrected based on the difference between the proportion of field name similarity distribution and the proportion of preset field name similarity distribution. The field name similarity threshold refers to the minimum similarity required for the corresponding vectorized semantics when aligning the field names.

8. The knowledge base data fusion and association method according to claim 7, characterized in that, The process of correcting the neighborhood radius of the density clustering algorithm based on the ratio of the coefficient of variation to the preset coefficient of variation includes: The coefficient of variation ratio is determined based on the ratio of the coefficient of variation to the preset coefficient of variation; The radius of the density clustering algorithm is increased based on the coefficient of variation ratio, and the increase in the radius of the density clustering algorithm is proportional to the coefficient of variation ratio.

9. The knowledge base data fusion and association method according to claim 7, characterized in that, The process of adjusting the field name similarity threshold based on the difference between the field name similarity distribution ratio and the preset field name similarity distribution ratio includes: The distribution ratio difference is determined based on the difference between the distribution ratio of the field name similarity and the preset distribution ratio of the field name similarity. The field name similarity threshold is reduced based on the difference in distribution proportions, and the reduction in the field name similarity threshold is proportional to the difference in distribution proportions.

10. A knowledge base data fusion and association system based on an AI large-scale model, characterized in that, Used to perform the knowledge base data fusion and association method based on the AI ​​large model as described in any one of claims 1-9 include, The acquisition module includes several connectors for acquiring the raw data; A data preprocessing module, which is connected to the acquisition module, is used to clean the data and remove duplicates from the cleaned data based on the Jaccard similarity threshold. The field alignment module, which is connected to the data preprocessing module, is used to obtain the field names and generate the corresponding vectorized semantics, and align the vectorized semantics based on the field name similarity threshold. The entity recognition and extraction module, which is connected to the field alignment module, is used to extract entities and attributes from the cleaned data based on the AI ​​large model, and denoted as the extracted entity and extracted attribute. The entity disambiguation and linking module, which is connected to the entity recognition and extraction module, is used to generate vectorized semantics corresponding to each extracted entity, and to merge extracted entities based on density clustering algorithm, and to retrieve vectorized entities in the database based on entity similarity threshold. Vectorized entities whose similarity between extracted entities and vectorized entities is higher than the entity similarity threshold are recorded as candidate entities. If there are several candidate entities, a link is constructed between the extracted entity with the highest similarity and the vectorized entity, and the corresponding extracted entity is recorded as a successfully linked entity. Otherwise, a new entity is constructed. The triplet construction module is connected to the entity disambiguation and linking module to generate triplets based on the AI ​​large model; The normalization module, which is connected to the triplet construction module, is used to assign unified identifiers to entities and attributes to obtain fused data; The conflict detection and optimization module, which is connected to the normalization module, is used to determine whether the current data conflict meets the requirements based on the coefficient of variation of the original data and the fused data, and, when it is determined that the requirements are not met, to determine the reason for the non-compliance based on the proportion of the duplicate hash value of the fused data in the total hash value, and to correct the data preprocessing parameters, field alignment parameters, entity disambiguation and linking parameters based on the corresponding reasons. The control module, which is connected to the conflict detection and optimization module, the data preprocessing module, the field alignment module, and the entity disambiguation and linking module, is used to schedule the corresponding modules to execute according to the instructions of the conflict detection and optimization module. The output module, which is connected to the normalization module and the conflict detection and optimization module, is used to construct a vector index based on the graph database and provide a graph download interface.

Images