Query-driven incremental data fusion method in heterogeneous data environment
By constructing an attribute semantic plane and a temporal index chain, combined with an intelligent heterogeneous data alignment engine and a global truth database, the query-driven incremental fusion problem of heterogeneous data in dynamic data lakes is solved, achieving efficient and reliable data fusion and consistency of query answers.
Patent Information
- Application Number
- CN202611274668.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
In dynamic data lakes, existing technologies struggle to retrieve sufficient evidence for on-demand queries in heterogeneous data environments, dynamically align heterogeneous values, and reuse historical fusion states during data updates, leading to inconsistent query answers, factual conflicts, and model inference biases.
An attribute semantic plane is constructed, and selective evidence expansion and entity cluster merging are achieved through temporal index chains and materialized base views. An intelligent heterogeneous data alignment engine is used for value normalization and evidence combination, and the fusion state is stored and updated in a global truth database, supporting incremental data fusion.
It improves the reliability of truth discovery, reduces redundant calculations and response latency, solves the problems of heterogeneous data format heterogeneity, partial inclusion and approximation conflict, and provides a stable basis for fusion state auditing and confidence interpretation.
Smart Images

Figure CN123115157A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of data management, data lake, data cleaning and truth discovery, and in particular to a query-driven incremental data fusion method in a heterogeneous data environment. Background Technology
[0002] With the development of enterprise-level data lakes and data platforms, multi-source data from relational tables, semi-structured documents, logs, text, and external data services will continuously flow into the same data lake. Values for the same entity or attribute from different sources often exhibit format differences, missing values, spelling errors, copy propagation, and semantic conflicts. Directly using this data for analysis or large model retrieval enhancement may lead to inconsistent query results, factual conflicts, and model inference biases.
[0003] Traditional batch data fusion typically performs a full fusion of the entire dataset offline. This method can utilize relatively complete global evidence, but it requires processing a large amount of data that is irrelevant to the current query. In scenarios where data lakes are continuously updated, each update may trigger a full re-fusion, resulting in high computational overhead and long response latency.
[0004] On-demand data fusion methods typically process only the records matched by the current query at runtime, reducing the initial processing load for a single query. However, these methods are limited by query conditions and are prone to missing evidence that was filtered out but is useful for truth determination. Furthermore, if there is a lack of incremental maintenance mechanisms, subsequent data arrivals or repeated occurrences of similar queries will still require repeated retrieval and fusion of a large amount of historical evidence that has already been processed.
[0005] Furthermore, heterogeneous values in dynamic data lakes are not simply a matter of string similarity. For example, dates, author names, affiliations, amounts, abbreviations, and enumeration attributes may have multiple formats from different sources; some values have inclusion relationships, some are approximate but conflicting erroneous variants, and some correspond to legitimate multiple truth values. Existing fusion methods that rely on general vector similarity or single-source confidence are insufficient to adequately distinguish these relationships, leading to the incorrect acceptance of multiple conflicting values or the incorrect rejection of legitimate multiple truth values.
[0006] Therefore, there is a need for an incremental data fusion scheme that can retrieve sufficient evidence when querying on demand, dynamically align heterogeneous values, and reuse historical fusion states as data is continuously updated. Summary of the Invention
[0007] This invention aims to address three interrelated problems in on-demand data fusion in dynamic data lakes: query predicates only return partial records, resulting in the omission of valid evidence for the same entity; format variations, included values, and approximate conflicts from different sources are not distinguished, leading to inaccurate truth value selection; and the lack of reusable states during data updates or repeated queries results in repeated retrieval and fusion of historical records. To address these issues, a query-driven incremental data fusion method is proposed for heterogeneous data environments.
[0008] To achieve the above objectives, this invention constructs an attribute semantic plane on top of the data lake, unifying heterogeneous sources into record-level attribute value assertions with record identifiers, source identifiers, and access times. For each attribute, a temporal index chain is constructed, consisting of basic index fragments and incremental index fragments. After historical queries are executed, the query signature, directly matched candidate records, and the identifier of the latest merged incremental fragment can be persisted as a materialized base view. When subsequent queries match this view, only incremental index fragments that have not yet been merged are scanned and deduplicated with the view records; if no match is found, the complete index chain is retrieved, thereby reducing redundant searches.
[0009] During the query processing phase, this invention parses natural language or structured queries into a directed acyclic graph composed of atomic query nodes and dependency edges, and executes them in topological order. Each atomic query first obtains candidate records that directly satisfy the query conditions. Subsequently, the system parses known records through the entity registry, supplements similar records outside the query conditions through source-level key attribute nearest neighbor retrieval, and merges existing entities and unparsed prototype clusters across sources according to centroid similarity; merged clusters formed only by unparsed records are assigned new entity identifiers. The resulting entity clusters contain both query-hit records and extended records belonging to the same entity (query-hit records and extended records belonging to the same entity are merged into evidence records).
[0010] During the fusion phase, for newly appearing entity attribute pairs, this invention employs a process of "original value normalization - intra-source evidence grouping and reliability weighting - cross-source evidence grouping and copy attenuation - relation graph construction - component-level truth value selection." Intra-source evidence groups consist of attribute value assertions from different records within the same source that are formally equivalent; cross-source evidence groups consist of intra-source evidence groups with formally equivalent representative values from different sources. The relation-aware evidence arbitrator establishes inclusion edges and conflict edges between cross-source evidence groups, connects evidence groups with inclusion edges to form candidate truth value components, aggregates component support, compares conflicting components, and calculates a dynamic threshold based on the highest current component support. A single truth value attribute outputs one component, while multiple truth value attributes can output multiple pairwise non-conflicting components; the representative value of the selected component is the fused truth value.
[0011] During the incremental maintenance phase, this invention stores the fusion state TR_e,a(t) of entity attribute pairs in the global truth database.<e,a,H,D,T> H stores historical cross-source evidence groups and their supporting records and support levels; D stores inclusion edges and conflict edges between evidence groups; and T stores the current truth components. When the same entity attribute pair is involved again at the current query time t′, the new evidence ΔR_e,a(t,t′) is obtained by filtering records from the current entity cluster according to their access time. It is a subset of records in the entity cluster that have been added since time t and contain attribute a. The system only forms an incremental evidence group from this subset of records, merges it with equivalent historical groups at the group level, creates new groups for unequal values, updates only the relation edges and candidate components affected by the newly added or modified groups, and then revises the truth values and confidence levels; original records already included in the historical state are no longer included in the alignment and grouping process.
[0012] Specifically, the process is performed in the following order: Attribute semantic plane modeling is performed on the multi-source heterogeneous data in the dynamic data lake to obtain a set of attribute value assertions that represent each physical row, semi-structured object, log entry, or text fragment as sharing the same record identifier, and the source identifier and access time are saved for each record; temporal index chains consisting of basic index fragments and incrementally appended index fragments are constructed for each attribute in the attribute semantic plane; user queries are received, and candidate records related to the user query are retrieved based on the temporal indexes and / or materialized views; selective evidence expansion is performed on the candidate records, merging candidate records and related records belonging to the same entity as the candidate records (excluding those specified in the query conditions) into entity clusters; evidence records are extracted for the target attributes in each entity cluster, and then... The identifier and target attribute form an entity attribute pair; the global truth database with the entity attribute pair as the key is queried to determine whether the entity attribute pair has a historical fusion state; if no historical fusion state exists, value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth selection are performed on the evidence record to generate a fused truth value and its confidence level and write it into the global truth database; if a historical fusion state exists, new evidence records that have been added since the corresponding time of the historical fusion state and belong to the entity attribute pair are determined from the current entity cluster, and value alignment and evidence merging are performed only on the new evidence records. The resulting incremental evidence group is merged with the historical evidence group in the historical fusion state at the group level, the evidence group relationship is locally updated, and the fused truth value and its confidence level are revised.
[0013] The query answer is formed by summarizing the fused truth values corresponding to the target attribute queried by the user.
[0014] Furthermore, the attribute semantic plane represents the snapshot L(t) of the data lake at time t as A(t); A(t) consists of records r, each record r is represented as a set or multiple sets of attribute value assertions (a,v), where a is the attribute name and v is the original attribute value without alignment processing when the data is accessed; all attribute value assertions within the same record share the record identifier rid(r), the source identifier sid(r), and the access time ts(r) to preserve the original record boundaries between attribute values; a row in the relation table, an object or nested object in JSON or XML, a log entry, and a continuous text fragment obtained by information extraction are each treated as a record, so that different data forms can be retrieved, parsed, and merged in a unified record-level attribute assertion form; The temporal index maintains an index chain I_a=[I_a^(0)|I_a^(1)|...|I_a^(m)] for each attribute a, where I_a^(0) is the read-only basic index fragment of the initial snapshot, and I_a^(i) is the immutable append-only incremental index fragment corresponding to the i-th batch of newly added data, 1≤i≤m; each index item is associated with at least the attribute value, record identifier, source identifier and incremental fragment identifier, so as to return the corresponding record according to the condition expression, and determine the record added after a certain historical query or historical fusion state according to the incremental fragment identifier; The materialized base view stores the query signatures, candidate record identifier sets, and the latest merged incremental fragment identifiers of historical queries. The materialized base view V_base(Q) is generated after the historical query Q is executed and stores at least: the query signature consisting of conditional attributes, normalized conditional expressions, target attributes, and atomic query dependencies; the candidate record identifier set R_id,base(Q) directly hit by the historical query; the view generation time t_base; and the latest merged incremental fragment identifier k_base at the time of generation; and the conditional attributes and target attributes of the current query and the historical query are the same, the normalized conditional expressions are equivalent, and the original... When the subquery dependency graph is isomorphic, it is determined that the query signatures of the two queries match and hit the materialized base view. If a hit occurs, the corresponding candidate record is read from the attribute semantic plane according to R_id,base(Q). Only the relevant attributes I_a^(k_base+1) to I_a^(m) are probed. The newly added hit record identifier is combined with R_id,base(Q) and deduplicated to obtain the candidate record set at the current query time. If a hit occurs, the complete index chain from I_a^(0) to I_a^(m) is retrieved. After the current query is completed, the newly added candidate record identifier in this round is written to R_id,base(Q), and k_base is updated to m.
[0015] Furthermore, it also includes decomposing user queries into atomic query graphs, specifically: identifying conditional attributes, conditional expressions, target attributes, and node dependencies in the query, and generating atomic query graphs to parse the query (natural language query or structured query) into a directed acyclic graph G(Q) =<U,E_Q> Where U = {q_1, ..., q_n} is the set of atomic query nodes, and E_Q is the set of dependent edges; each atomic query node q_i is represented as<pred_i,a_c^(i),v_c^(i),a_t^(i),v_t^(i)> Pred_i is the set of predecessor nodes, a_c^(i) and v_c^(i) are the conditional attribute and conditional expression, respectively, a_t^(i) is the target attribute, and v_t^(i) is the projection or transformation expression performed on the target attribute retrieval result; the dependency edge (q_j, q_i) indicates that the conditional expression of q_i uses the output view of q_j; the system executes each atomic query node in topological order and executes nodes without dependencies in parallel, and each node outputs a set of candidate records that directly satisfy its conditional expression.
[0016] Furthermore, the selective evidence expansion includes three stages: First, querying the entity registry E_reg={(eid,s,rid)}, mapping registered candidate records to existing entities, and reading all registered records with the same entity identifier from the entity registry as existing context records for that entity; Second, bucketing unregistered records according to their source key attribute set, performing approximate nearest neighbor retrieval within the corresponding bucket, retrieving records from all records of that source that have a similarity of not less than the source retrieval threshold θ_s and were missed by the original query conditions, and performing existing entity attachment or forming unresolved prototype clusters based on whether the retrieved records have entity identifiers; Third, using all registered records and unresolved prototype clusters corresponding to existing entities as clusters to be merged, calculating cluster centroids and performing cross-source nearest neighbor retrieval, merging clusters with centroid similarity not less than the global merging threshold θ_global, to obtain entity clusters containing query-hit records and records related to the query conditions. Furthermore, for unresolved records r in source s, the record embedding emb(r) is calculated based on the key attribute set K_s, r is mapped to bucket B_s,b, and the candidate nearest neighbor set N_s(r) = {r′∈B_s,b | sim(emb(r),emb(r′))≥θ_s} is calculated, where r′ represents a candidate similar record located in the same bucket B_s,b as the unresolved record r, and θ_s is the in-source retrieval threshold of source s. N_s(r) is divided into a set N_s^+(r) with existing entity identifiers and a set N_s^-(r) without entity identifiers. When N_s^+(r) is not empty, the similarity between each entity and r is aggregated, and r is attached to the existing entity with the highest aggregation similarity that reaches the attachment threshold θ_attach; r that does not reach θ_attach continues to be treated as unresolved records. When N_s^-(r) is not empty, an intra-source adjacency graph is constructed with unresolved records as vertices and edges having a similarity of θ_s. Each connected component forms an unresolved prototype cluster. After cross-source merging, if the merged result contains existing entities, the identifiers of those existing entities are retained; if it contains multiple existing entities, the identifier of the entity with the highest or lowest preset priority is retained, and the mappings of other records are updated; if it only contains unresolved prototype clusters, a new entity identifier is assigned from a monotonically increasing entity identifier counter, and all records within the cluster are written to the entity registry.
[0017] Furthermore, the process includes value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth value selection. This includes: using an entity attribute pair (e, a) as the processing unit, extracting evidence records containing attribute a from the entity cluster of entity e, where each evidence record provides an unaligned original value; converting the original values to canonical form and determining value equivalence; merging attribute value assertions that are canonically equivalent and provided by different records within the same source into an intra-source evidence group; and combining intra-source evidence groups with canonically equivalent representative values from different sources. It is then used to construct cross-source evidence groups; the intra-source support is calculated based on the number of deduplicated records and the source reliability of the intra-source evidence groups, and the aggregate support of the cross-source evidence groups is attenuated based on the inter-source copy probability; a cross-source evidence group relationship graph is established to distinguish between inclusion relationships, conflict relationships, and no explicit relationships; cross-source evidence groups connected by inclusion relationships are used as candidate truth value components, and candidate truth value components with conflicts are compared, and one or more candidate truth value components are selected based on a dynamic threshold; the component representative value of the selected component is used as the fusion truth value, and the confidence level and the set of record identifiers supporting the truth value are output.
[0018] Furthermore, an intelligent heterogeneous data alignment engine converts the original values into a canonical form and determines their equivalence. This engine includes an atomic operation library, a global transformation rule library, a rule cache built on the context according to source attributes, an equivalence judgment function, a failure monitoring unit, and a rule generation unit. The source attribute on the context c=(s_i,a_i,s_j,a_j) represents the source and attribute of the two values to be compared. The transformation rules are formed by sequentially combining one or more atomic operations from the following: whitespace trimming, case unification, character replacement, delimiter splitting, field extraction, date normalization, numerical precision normalization, unit conversion, and additional marker deletion. When comparing the original values v_i and v_j, rules are first retrieved from the rule cache according to historical success rates. If the cache is not found, rules are retrieved from the global transformation rule library, resulting in the canonical forms v_i^c and v_j^c respectively. The equivalence judgment function f_equiv(v_i^c,v_j^c) outputs the equivalent or unequal result. The failure monitoring unit calculates the failure rate of context c within each fusion window as the number of failed matches divided by the number of rule calls. Rule generation is initiated only when the failure rate exceeds the threshold τ for n consecutive failure fusion windows and the cumulative number of failed samples is not less than N_min. The rule generation unit constructs context-bearing samples using high-confidence ground truth values and unselected values of the same entity attribute pair, generates candidate rules, and verifies the positive example alignment success rate and negative example false merging rate on validation samples not involved in the generation. Only candidate rules with a positive example alignment success rate not lower than θ_rule and a negative example false merging rate not higher than θ_false are written to the rule cache and remain valid before being written to the global transformation rule library.
[0019] Furthermore, the cross-source evidence set support is calculated by combining source reliability and inter-source copy probability. This includes maintaining Beta distribution parameters α_s,q and β_s,q for source s and atomic query q, initializing α_s,q=κ·w_s^(0)(q) and β_s,q=κ·(1-w_s^(0)(q)) with the initial reliability w_s^(0)(q) obtained from the source profile and the prior strength κ; after each round of fusion, the positive feedback number Δn_s,q^good and the negative feedback number Δn_s,q^bad are obtained according to whether the intra-source evidence set of source s is covered by the final truth component, and updated according to α_s,q←φ·α_s,q+Δn_s,q^good and β_s,q←φ·β_s,q+Δn_s,q^bad, where φ is the forgetting factor; and using w_s(q)=α_s,q / (α_s,q+β_s,q) as the source reliability. For a source-internal evidence group g, calculate the source-internal support σ_g^(1) = |g|·w_s(q) based on the number of different record identifiers it contains |g|. Combine source-internal evidence groups with equivalent normative representative values but different sources into a cross-source evidence group h, resulting in the contributing source set S_h and the original support Σ_h = ∑_(g∈h)σ_g^(1). Maintain the attribute-level copy probability for each source pair and calculate the average copy probability C_h of the source pairs within the cross-source evidence group; calculate the cross-source evidence group support according to σ_h^(2) = ((1-C_h) + C_h / |S_h|)·Σ_h, where C_h = 0 when |S_h|≤1.
[0020] Furthermore, a cross-source evidence group relationship graph is established using a relation-aware evidence arbitrator to distinguish between inclusion relationships, conflict relationships, and no explicit relationships. The relation-aware evidence arbitrator normalizes and segments the canonical representative value v_i of the cross-source evidence group h_i to obtain a term set T_i. When T_j is a proper subset of T_i, an inclusion edge pointing from h_i to h_j is established. For two cross-source evidence groups without inclusion edges, a conflict edge is established based on the embedding similarity reaching a conflict threshold or the existence of only one non-substitutable term difference between the two, provided that the word overlap rate reaches a threshold. The rest are marked as no explicit relationship. When constructing candidate truth components, each connected component formed after ignoring the direction of the inclusion edge is taken as a candidate truth component B. Therefore, cross-source evidence groups that directly include each other and those indirectly connected through inclusion chains belong to the same candidate truth component. The component support S(B) is the sum of the support of all cross-source evidence groups within the component. The component representative value is selected from the canonical representative values with an in-degree of zero, where zero in-degree indicates that the value is not included by other values within the component. When multiple values with an in-degree of zero exist, they are determined sequentially according to their corresponding cross-source support from high to low, the number of contributing sources from high to low, the canonical string encoding from low to high, and the group identifier from low to high. Two components are considered to be in conflict if there is a conflicting edge connecting any cross-source evidence group within the two components. The component with higher support is retained, and the component with lower support is removed. When the support is equal, one is retained according to the preset deterministic order of the component representative value and the component identifier. When the difference in support between the two components is less than the conflict margin δ_rel, the conflict pair is recorded as a low margin conflict pair for stability judgment in subsequent incremental maintenance. For the components retained after conflict elimination, calculate θ_truth=η·max_k S(B_k), where the threshold changes with the highest support of the component in the current round; for single truth attributes, only output the component that reaches the threshold and has the highest support, where a single truth attribute is defined by the attribute pattern as having at most one valid value for the same entity; for multiple truth attributes, output all components that reach the threshold and are not conflicting with each other, and each selected component corresponds to a fused truth value.
[0021] Furthermore, the global truth library stores the structured fusion state TR_e,a(t) for entity attribute pairs (e,a).<e,a,H,D,T> , where t is the last update time of the state; H is the set of historical cross-source evidence groups, each of which at least stores the group identifier, canonical representative value, the source evidence groups it contains, the source support corresponding to each source, the set of contributing sources, the set of supporting record identifiers, the set of original values, the cross-source support, and the most recent update time; D is the evidence group relationship graph with cross-source evidence groups as vertices and containing edges or conflicting edges as edges, and records low-margin conflict pairs; T is the set of currently selected truth components, each of which at least stores the component members, the component representative value, the fused truth value, the component support, the confidence level, the set of supporting record identifiers, and the selection results in the most recent few refreshes; Furthermore, the incremental fusion truth maintenance includes: reading TR_e,a(t) from the global truth database; obtaining the current entity cluster C_e(t′) of entity e at the current query time t′, and selecting records that satisfy ts(r)>t and contain attribute a to form a new evidence record set ΔR_e,a(t,t′). Therefore, ΔR_e,a(t,t′) is a subset of records newly added to the current entity cluster relative to the historical state, rather than the entity cluster itself; performing time decay on the source intra-source support and cross-source support of each historical evidence group according to their most recent update time; and performing the same normalization, source intra-grouping, and cross-source grouping as the first fusion only on ΔR_e,a(t,t′). Source grouping and support calculation yield an incremental cross-source evidence set ΔH. Each incremental cross-source evidence set is matched with H using a normalized representative value equivalence match. When equivalent, the source-internal evidence sets, contributing source sets, supporting record identifier sets, and original value sets of both are merged. For the same source, the decayed historical source-internal support is added to the incremental source-internal support, and then the cross-source support is recalculated based on the merged source set and the current copy probability. When not equivalent, the incremental cross-source evidence set is added to H as a new set. Only the inclusion edges and conflict edges connected to the newly added set or the modified historical set are recalculated. The affected candidate truth components are reconstructed based on the updated H and D, and T and confidence are revised. Finally, the state time is updated to t′.
[0022] Furthermore, after merging and updating the relationships between the incremental evidence group and the historical evidence group, the component support calculation, conflict comparison, and dynamic threshold screening are re-executed for candidate truth components that include newly added evidence groups or modified historical evidence groups. Components newly added to the selected set or whose support changes in this round are called updated candidate components. For multiple truth attributes, when an updated candidate component has an inclusion connection with the current truth component, they are considered the same component, and the component's representative value and confidence are updated; when there is no explicit relationship between them and the updated candidate component reaches the dynamic threshold, it is added to T as a new truth component; when an updated candidate component has a conflict edge with the current truth component, it is called a challenging component. For single truth attributes, all selected components except the current truth component are considered challenging components. The confidence difference between the challenging component and the challenged current truth component is compared. If the challenge component's confidence is at least δ_stab higher than the current truth component's, it is directly replaced, where δ_stab is a pre-set confidence difference threshold between 0 and 1. If the difference is less than δ_stab, the results of the most recent u refreshes are read for stability voting, where u is a positive integer and 1 ≤ u_min ≤ u. A refresh refers to completing a new evidence retrieval, group-level merging, relation update, and truth recalculation for the same entity attribute pair at a new query time. Replacement is only submitted if the challenging component has been selected at least u_min times in the most recent u refreshes and its confidence is higher than the current truth component; otherwise, the current truth is maintained.
[0023] This invention also provides a query-oriented incremental data fusion system in a dynamic data lake, comprising: The attribute semantic plane construction module is used to perform attribute semantic plane modeling on multi-source heterogeneous data in dynamic data lakes. It obtains a set of attribute value assertions that represent each physical row, semi-structured object, log entry or text fragment as sharing the same record identifier, and saves the source identifier and access time for each record. The temporal index evidence retrieval module is used to construct temporal index chains consisting of basic index fragments and incremental index fragments that are only appended to the attributes in the attribute semantic plane; receive user queries, identify conditional attributes, conditional expressions, target attributes, and node dependencies in the query, and generate atomic query graphs; according to the atomic query graphs, retrieve candidate records that directly satisfy the query conditions based on the temporal index chains and materialized base views, wherein the materialized base views store the query signatures of historical queries, the candidate record identifier set, and the identifier of the latest incremental fragment that has been merged; The selective evidence expansion module is used to perform selective evidence expansion on candidate records, merging candidate records and related records belonging to the same entity as the candidate records that are not subject to the query conditions into entity clusters. The global truth database retrieval module is used to extract evidence records for the target attributes in each entity cluster and form entity attribute pairs with entity identifiers and target attributes; it queries the global truth database with entity attribute pairs as keys to determine whether the entity attribute pairs have historical fusion states. The adaptive alignment hierarchical fusion module is used to perform value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth value selection on the evidence records when there is no historical fusion state, and to generate fused truth values and their confidence levels and write them into the global truth value library. In the incremental fusion truth value maintenance module, when there is a historical fusion state, the module determines the new evidence records that are added after the corresponding time of the historical fusion state and belong to the entity attribute pair from the current entity cluster. Only the new evidence records are value aligned and evidence merged. The resulting incremental evidence group is merged with the historical evidence group in the historical fusion state at the group level. The evidence group relationship is locally updated and the fusion truth value and its confidence are revised.
[0024] Compared with existing technologies, the present invention has at least the following beneficial effects: First, it can obtain contextual evidence beyond the query predicate through selective expansion, thereby improving the reliability of truth discovery; Second, it can handle format heterogeneity, partial inclusion, approximate conflict, and multiple truth value problems through dynamic alignment and relation arbitration; Third, it can reuse historical fusion states when data is continuously added, thereby reducing redundant calculations and response delays; Fourth, it can save the fusion state in a structured form, providing a basis for subsequent auditing, confidence interpretation, and stability revision. Attached Figure Description
[0025] Figure 1 This is the overall framework diagram of the present invention.
[0026] Figure 2 This is a flowchart illustrating the intelligent heterogeneous data alignment engine of the present invention.
[0027] Figure 3 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation
[0028] The present invention will now be described in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only for illustrating the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments can be combined with each other.
[0029] like Figure 1 As shown, the system of this invention includes an attribute semantic plane construction module, a temporal index evidence retrieval module, a selective evidence expansion module, an adaptive alignment hierarchical fusion module, a global truth database retrieval module, and an incremental fusion truth maintenance module. The system takes a dynamic data lake L(t) and a user query Q as input. First, it generates candidate records that are directly matched by the query, and then expands these candidate records into one or more entity clusters. Subsequently, for each entity cluster and the query target attribute, it forms an entity attribute pair (e, a). If the global truth database is not matched, the initial fusion is performed and the fusion state is written; if it is matched, only new evidence is processed and the fusion state is updated. The fused truth values of each entity attribute pair are summarized to form the query answer Ans(Q,t).
[0030] To avoid conceptual confusion between processing stages, the following definitions are made: Candidate records are records directly hit by atomic query conditions; Extended records are records that do not directly hit query conditions but are determined by entity parsing to belong to the same entity as candidate records; Entity cluster C_e(t) is the set of records that are assigned to entity e by the system entity parsing result eid_t at time t; Evidence record R_e,a(t) is a subset of records in C_e(t) that contain target attribute a; Attribute value assertion (a,v) is the smallest evidence unit used for fusion in the evidence record. For query Q, if the system performs initial fusion or equivalent incremental maintenance on all records in R_e,a(t) for each entity attribute involved in the query, and the output Ans(Q,t)[e,a] is consistent with the result obtained by performing a full recalculation on the same R_e,a(t), this is called global fusion consistency; Once historical evidence is merged into TR_e,a(t), subsequent updates only use historical group-level states and new evidence, and the historical record is no longer sent to the alignment, grouping, and scoring process, this is called single-touch.
[0031] In one embodiment, the attribute semantic plane construction module represents a data lake snapshot L(t) as A(t). The smallest unit of evidence in A(t) is the attribute value assertion (a, v), where v is the original value saved at the time of data access. To preserve the association between multiple attributes within the same row or document, the system combines assertions belonging to the same physical record into a record r, and saves a unique record identifier rid(r), a source identifier sid(r), and an access time ts(r) for each record. A row from a relational data source, an object or nested object from JSON or XML, and a log entry are converted into records, respectively; for text data sources, entities, attributes, and values are extracted first, and then semantically continuous text fragments are converted into pseudo-records. The attribute semantic plane only maintains attribute associations within records; cross-source entity parsing is deferred until the query time.
[0032] In the temporal index evidence retrieval module, the system maintains an index chain I_a=[I_a^(0)|I_a^(1)|...|I_a^(m)] for each attribute 'a'. I_a^(0) is the batch-built basic index corresponding to the initial snapshot; I_a^(i) is the immutable incremental index built by assertions of attribute 'a' in the i-th batch of newly added records. Each index entry associates the searchable attribute values with the record identifier, source identifier, access batch, and access time. When new data arrives, only new incremental fragments are generated, without rewriting existing fragments; the backend can perform physical merging without changing the logical fragment boundaries to ensure that historical fragment identifiers can still be used to determine the incremental range.
[0033] When the user query Q arrives, the query parsing module identifies the data dependencies between the conditional attribute a_c, the conditional expression v_c, the target attribute a_t, and the subqueries, and generates an atomic query graph G(Q) =<U,E_Q> Each node q_i =<pred_i,a_c^(i),v_c^(i),a_t^(i),v_t^(i)> This represents a conditional retrieval and target attribute projection; `pred_i` lists its predecessor nodes, `v_c^(i)` can be a constant or a user function, or it can reference the result view of the predecessor node, and `v_t^(i)` is used to limit or transform the target attribute output. The system schedules nodes in topological order, and the current node can only be executed after all predecessors have been completed. Nodes without dependencies are executed in parallel.
[0034] For each atomic query, the system first generates a query signature based on the condition attributes, normalized condition expressions, target attributes, and dependency structure. Two queries are considered to match only if their condition attributes are identical, their target attributes are identical, their condition expressions are semantically equivalent after normalization (date format, numeric format, case, and whitespace), and their atomic query dependency graphs are isomorphic. The materialized base view V_base(Q) stores the set of candidate record identifiers R_id,base(Q) that were directly hit by historical queries at time t_base, as well as the latest incremental fragment number k_base that was merged at that time. If the view is hit, historical candidate records are read from the attribute semantic plane based on the record identifiers. The same condition retrieval is performed only in I_a^(k_base+1) to I_a^(m), and the incremental hit record identifiers are merged with R_id,base(Q) and deduplicated. If no match is found, I_a^(0) to I_a^(m) are retrieved. The resulting R_q(t) is called the candidate record set, representing only records that directly satisfy the atomic query conditions, excluding records obtained through subsequent selective expansion. After the query is completed, the system will add the identifier of the newly added candidate record in this round to R_id,base(Q) and update k_base to m, so that the next same query will not repeatedly probe the incremental segment in this round.
[0035] Because query conditions may exclude records with critical context, the selective evidence expansion module performs entity-level expansion on the candidate record set R_q(t). In the first stage, the system queries the entity registry E_reg={(eid,s,rid)}; candidate records with existing mappings directly determine their entity identifiers, and further read all registration records in E_reg containing that entity identifier, thereby filling in known records within the same entity that were not matched by the query conditions; candidate records without existing mappings constitute the unresolved set R_q^. (t). In the second stage, a set of key attributes K_s= is pre-determined for each source s.<k_s^(1),...,k_s^(h)> The average value of the key attribute embedding is defined as the record embedding emb(r) = 1 / |K_s|·∑_(v∈K_s(r))emb(v), and the bucket identifier b is obtained through locality-sensitive hashing. Within bucket B_s,b, obviously dissimilar records are first filtered using LSH, and then approximate nearest neighbor retrieval is performed using HNSW, resulting in N_s(r) = {r′∈B_s,b | sim(emb(r),emb(r′))≥θ_s}. This set is retrieved from all indexed records of source s, and therefore may include relevant records not matched by the original query conditions.
[0036] For each unresolved record r, N_s(r) is divided into a set of existing entity records N_s^+(r) and a set of unresolved records N_s^-(r). When N_s^+(r) is not empty, for each candidate entity identifier eid_x, the attachment score score(eid_x|r) = 1 / |N_eid_x|·∑_(r′∈N_eid_x)sim(emb(r),emb(r′)) is calculated, where N_eid_x is the set of records in N_s^+(r) with entity identifier eid_x. r is attached to the entity with the highest score and score(eid_x|r)≥θ_attach. If the score does not reach the threshold, the record remains unresolved. For all records that remain unresolved, an intra-source adjacency graph is formed with the record as the vertex and edges with similarity reaching θ_s. Each connected component is used as an unresolved prototype cluster. In the third stage, all existing entity registration records and unresolved prototype clusters are considered as clusters C′ to be merged. The centroid is calculated using μ(C′) = 1 / |C′|·∑_(r∈C′)emb(r) and written into the incremental HNSW graph. When the centroid similarity of two clusters reaches the global threshold θ_global, they are merged, iterating until there are no more clusters to merge. θ_s, θ_attach, and θ_global control intra-source candidate recall, existing entity attachment, and cross-source cluster merging, respectively. In one embodiment using the normalized vector inner product as the similarity sim, θ_s, θ_attach, and θ_global are all set to 0.98. When using other similarity measures or data distributions from different sources, adjustments can be made on the entity-annotated validation data to meet the preset precision and maximize the recall. When the merged result contains existing entities, the existing entity identifiers are retained. When it contains multiple existing entities, they are unified according to a preset priority or the smaller identifier. When it consists entirely of unresolved prototype clusters, new entity identifiers are assigned from the monotonic counter, and all records are registered in E_reg. The final entity cluster set is C={C_1,...,C_l}, where l is the number of entities involved in this atomic query, and C_j is the set of records belonging to the j-th entity, and contains at least one candidate record.
[0037] The relationship between entity clusters and subsequent fusion evidence is as follows: In the selective expansion phase, entity boundaries are first determined; in the fusion phase, subsets are selected from the entity clusters according to the target attribute. Specifically, for entity cluster C_j and target attribute a_t, the evidence record set is R_j,a_t(t) = {r∈C_j | r contains an attribute value assertion (a_t,v)}. Records that do not contain the target attribute a_t do not directly form the support for that attribute, but can participate in entity resolution through other attributes such as ISBN, name, and time. The newly added evidence ΔR_j,a_t(t,t′) during a subsequent query is the set of records in R_j,a_t(t′) whose access time is later than the historical state time t.
[0038] In the adaptive alignment hierarchy fusion module, for entity attribute pairs (e, a) that are not found in the global truth pool, the system performs an initial fusion. For example... Figure 2 As shown, the use and expansion of alignment rules follow the order of "calling existing rules - recording this equivalence judgment - counting failures by window - generating candidate rules - independent verification - hierarchical storage". The original value is the attribute value v in the attribute semantic plane that is saved with the record and has not yet undergone format conversion. When comparing two original values v_i and v_j from the source attribute pair context c=(s_i,a_i,s_j,a_j), the system first retrieves the rule with the higher historical success rate from the context rule cache. If the cache is not hit, it then retrieves it from the global conversion rule base R_base. The rules R_i and R_j are applied to v_i and v_j respectively to obtain the canonical forms v_i^c and v_j^c, and the equivalent result is output by f_equiv(v_i^c,v_j^c). When they are equivalent, the two assertions can enter the same evidence group. When they are not equivalent, they are kept separate and the failure context is recorded. Rules are not generated immediately due to a single failure.
[0039] The transformation rules are formed by sequentially combining one or more operations from the atomic operation library. Atomic operations include whitespace trimming, case unification, character replacement, delimiter splitting, field extraction, date format normalization, numerical precision normalization, unit conversion, and additional marker deletion. The system maintains the rule call count N_call(c), the number of successful matches N_succ(c), and the number of failed matches N_fail(c) for each context c, and calculates the failure rate r_fail(c,W) = N_fail(c,W) / N_call(c,W) within each sliding fusion window W. Rule expansion is initiated only when r_fail(c,W) exceeds τ for n_fail windows and the cumulative number of failed samples is not less than N_min; n_fail represents the number of consecutive windows, which can be 3 to 10, preferably 5; τ can be 0.20 to 0.50, preferably 0.30; and N_min can be 10 to 50, preferably 20. Before the new rule verification is completed, the value pairs that trigger failure will still enter the current round of grouping according to the unequal results, without blocking the query.
[0040] After rule expansion is initiated, the system does not directly generate rules based on a single failed value. Instead, it constructs samples using previously completed high-confidence fusion decisions. For the same entity attribute pair, the representative value selected as true by the relation-aware evidence arbitrator and with a confidence ρ not lower than θ_conf, along with its corresponding original value, is used as a high-confidence benchmark. The original values of the same entity attribute pair that were not selected as true and that have equivalent failure or conflict judgments with the benchmark value are used as values to be analyzed, forming samples together with the context c. θ_conf can be between 0.80 and 0.95, preferably 0.90. The rule generation unit aggregates multiple similar samples and searches for or generates candidate rule pairs (R_i, R_j) from the atomic operation library. The goal is to align format variants that have been proven to be the same fact by other high-confidence evidence, while avoiding the erroneous merging of values with substantial conflicts. Rule generation can employ procedural search, large language model-assisted generation, or a combination of both.
[0041] After candidate rules are generated, the system independently verifies them using validation samples not involved in rule generation, calculating the positive example alignment success rate, negative example false merging rate, and rule applicability coverage. Rules can only be enabled when the positive example alignment success rate is not lower than θ_rule and the negative example false merging rate is not higher than θ_false; θ_rule can range from 0.85 to 0.98, preferably 0.90, and θ_false can range from 0 to 0.10, preferably 0.05. Validated rules are first written to the rule cache of the corresponding context, recording the version, applicable context, operation sequence, and validation metrics; after consistently meeting the metrics in multiple subsequent fusion windows or multiple contexts with the same pattern, they are then written to the global transformation rule base R_base. Manually configured rules and stable validation rules take precedence over newly generated rules; new rules are deactivated or rolled back if the subsequent false merging rate exceeds the threshold.
[0042] After completing value alignment and necessary rule expansion, the system uses an entity cluster C_e with a defined entity identifier e and a target attribute a_t as the processing unit, and executes the following seven steps sequentially. Evidence record R_e,a_t(t) is a set of records in C_e that contain attribute a_t; the (a_t,v) assertion in each evidence record provides an original value. All the following groups are limited to the same entity attribute pair (e,a_t) and their member record identifiers are preserved, enabling step-by-step tracing between the original value, intra-source evidence groups, cross-source evidence groups, candidate truth components, and the final truth value.
[0043] First, the target attribute values are extracted and formed into source-internal evidence groups. The system extracts all attribute value assertions from R_e,a_t(t) according to source s. The original value v, record identifier rid, entity identifier e, source identifier s, attribute a_t, and query identifier q of each assertion are input into the intelligent heterogeneous data alignment engine to obtain the canonical form v^c. Within the same source, assertions with the same canonical form or those deemed equivalent by f_equiv are merged into a source-internal evidence group g. Evidence records included in g are called supporting records of that group; identical assertions generated from repeated crawling, repeated parsing, or repeated fields for the same record are counted only once, therefore |g| equals the number of different record identifiers in the supporting record identifier set R_g. Each source-internal evidence group is stored...<gid,e,a_t,s,v_g^c,V_g,R_g,|g|> Where gid is the source evidence group identifier, v_g^c is the normative representative value (any normative form value in the evidence record belonging to g is taken as the normative representative value of g), V_g is the original value set of member assertions, and R_g is the supporting record identifier set.
[0044] Second, calculate the support of the evidence group within the source and update the source reliability. The system maintains Beta distribution parameters α_s,q and β_s,q for each source s and the current atomic query q. For sources without historical feedback, the target column completeness, format consistency, effective record count, and source authority are first normalized to 0 to 1, and then the initial reliability w_s^(0)(q) is calculated with weights of 0.4, 0.3, 0.2, and 0.1, respectively; if a certain profile is missing, the weight of that item is proportionally allocated to the remaining items. α_s,q = κ·w_s^(0)(q) and β_s,q = κ·(1-w_s^(0)(q)) are initialized with the prior strength κ; w_s^(0)(q) can be between 0.5 and 1.0, preferably 0.9. After each round of truth value selection, for each source evidence group in source s, it is determined whether its normative representative value is covered by the final truth component. Coverage is counted as one positive feedback, and non-coverage is counted as one negative feedback. The positive and negative feedback numbers for the current batch are Δn_s,q^good and Δn_s,q^bad, respectively. Updates are made according to α_s,q←φ·α_s,q+Δn_s,q^good and β_s,q←φ·β_s,q+Δn_s,q^bad, where φ can be 0.90 to 0.995, preferably 0.98; the source reliability is w_s(q)=α_s,q / (α_s,q+β_s,q). The support of the source evidence group is calculated according to σ_g^(1)=|g|·w_s(q).
[0045] Third, cross-source evidence groups are formed. The system compares evidence groups from different sources within the same source, and only merges them into the same cross-source evidence group h if their canonical representation values are the same or if they are determined to be equivalent by f_equiv; groups that are not equivalent within the same source are kept separate to preserve the source's own conflicts. If the equivalence judgment is implemented by a similarity model, the equivalence threshold θ_equiv can be between 0.75 and 0.95, preferably 0.85. Each cross-source evidence group corresponds to a candidate canonical value for the entity attribute pair (e, a_t), and is stored.<hid,e,a_t,v_h^c,G_h,S_h,R_h,V_h,Σ_h,σ_h^(2)> Where, hid is the cross-source evidence group identifier, v_h^c is the normative representative value of the cross-source evidence group, which is taken as the normative representative value of the first source evidence group added when the group is formed, and v_h^c remains unchanged when subsequent equivalent source evidence groups are added; G_h is the set of member source evidence groups, S_h is the set of contribution sources, R_h is the set of all supporting record identifiers, V_h is the set of original values, Σ_h is the original support before decay, and σ_h^(2) is the cross-source support; when the incremental evidence group is merged into the historical cross-source evidence group, the v_h^c already saved in the historical group is used.
[0046] Fourth, calculate the support of cross-source evidence groups. First, sum the support of each source evidence group in h to obtain Σ_h=∑_(g∈G_h)σ_g^(1). To estimate source dependencies, maintain a Beta distribution Beta(α_st^(a_t),β_st^(a_t)) for each unordered source pair {s,t} and attribute a_t; when there is no historical information, let α_st^(a_t)=β_st^(a_t)=1, and initialize C_h to 0 in this round of calculation. Only when a cross-source evidence group is ultimately rejected by the arbitrator, the source pair within that group is used as a conservative update sample. If the two sources provide exactly the same original literal value before normalization, then α_st^(a_t) is incremented by one; otherwise, β_st^(a_t) is incremented by one. The attribute-level copy probability is C_st^(a_t) = α_st^(a_t) / (α_st^(a_t) + β_st^(a_t)); the overall copy probability across multiple attributes can be expressed as C_st^( The formula is calculated as ) = 1 - ∏_a(1 - C_st^(a)). Let the set of contribution sources of h be S_h, then the average copy probability of the source pair is C_h = [1 / M_h]·∑_(s <t,s,t∈S_h)C_st^( ), where M_h=max{1,|S_h|(|S_h|-1) / 2}; when |S_h|≤1, let C_h=0. Finally, calculate the cross-source support according to σ_h^(2)=((1-C_h)+C_h / |S_h|)·Σ_h: when C_h is close to 0, retain the sum of support from multiple sources, and when C_h is close to 1, decay to the contribution of a single effective source.
[0047] Fifth, establish a cross-source evidence group relationship graph. The system normalizes and segments the canonical representative value v_i of each cross-source evidence group to obtain a term set T_i. If T_j is a proper subset of T_i, a directed containing edge h_i→h_j is established, indicating that v_i contains v_j, which is applicable to the support relationship between complete enumeration and partial enumeration. For h_i and h_j that do not have a containing relationship, the word overlap rate |T_i∩T_j| / min(|T_i|,|T_j|) is calculated first; when the word overlap rate reaches θ_overlap, and the embedding cosine similarity reaches θ_embed or the symmetric difference |T_i△T_j|=2, an undirected conflict edge is established, indicating that the two values are highly similar in context but incompatible on a keyword term. θ_overlap can be 0.5 to 0.8, preferably 0.6, and θ_embed can be 0.75 to 0.95, preferably 0.85. A group that has neither containing edges nor conflicting edges is marked as having no explicit relation.
[0048] Sixth, construct candidate truth components and perform conflict elimination. Each connected component obtained after ignoring the direction of the containing edge is defined as a candidate truth component B; therefore, whether h_A directly contains h_B and h_C, or h_A contains h_B and h_B contains h_C, as long as the three groups are connected by the containing edge, they belong to the same component. Component support S(B)=∑_(h_i∈B)σ_i^(2). The maximum canonical value in the component is the value with zero in-degree in the containing graph, that is, there is no other member value that contains the value; its term set cannot be further expanded under the containing partial order. If there is only one maximum canonical value, it is used as the representative value of the component; if there are multiple maximum canonical values that do not contain each other, they are selected in order of cross-source group support corresponding to the value from high to low, number of contributing sources from many to few, canonical string encoding from small to large, and group identifier from small to large, so as to ensure the result is determined. If there is a conflicting edge between two components B_i and B_j that connects any of their member groups, it is called a component conflict. Components with higher support are always retained while those with lower support are removed. When support is equal, one component is retained based on its representative value in descending order and the deterministic order of its identifier (component identifier in ascending order). If the support difference is less than δ_rel, the low-margin conflict pair is recorded in D for subsequent incremental maintenance to determine whether a stable replacement has occurred. δ_rel can be set to 5% to 20% of the current highest component support, preferably 10%.
[0049] Seventh, filter and output the final truth value. For the pairwise non-conflicting components retained after the conflict elimination, calculate the dynamic threshold for this round according to θ_truth = η·max_k S(B_k), where max_k S(B_k) is the highest component support in this round. Therefore, as the highest support changes due to new evidence, source reliability, copy probability, or time decay, θ_truth changes synchronously; η can be 0.5 to 0.8, preferably 0.6. A single truth attribute is an attribute pattern that specifies that the same entity has at most one valid value at any given time. Components with support lower than θ_truth are filtered first, and then the component with the highest support is output. A multiple truth attribute allows the same entity to have multiple valid values simultaneously, outputting all components that reach θ_truth. The number of output components is determined by the number of valid truth values that meet the conditions, rather than being predetermined. Each selected component outputs a component representative value v_B^. Supports record identifier sets and confidence levels ρ_B and v_B^. This refers to the final fused truth value corresponding to the component. The confidence level can be calculated as ρ_B=S(B) / ∑_j S(B_j), where the denominator is the sum of the support of all candidate components before conflict resolution, and the result is restricted to 0 to 1.
[0050] After the initial fusion is completed, the system writes all cross-source evidence groups, evidence group relationship graphs, and selected truth components into the global truth database. If the same entity attribute pair is reached by multiple atomic nodes within the same query, it is first deduplicated by entity identifier and attribute, and then written again; subsequent atomic nodes directly reuse this state.
[0051] For entity attribute pairs (e, a) that have already been matched in the global truth database, the system performs incremental truth maintenance. The global truth database is empty during system initialization; after the entity attribute pair completes the above seven steps for the first time, it is written to TR_e,a(t) with (e, a) as the key.<e,a,H,D,T> Each historical cross-source evidence group in H stores the canonical representative value, member in-source groups, in-source support for each source, source set, supporting record identifier, original value, cross-source support, and most recent update time; D stores included edges, conflicting edges, and low-margin conflict pairs; T stores the selected truth component, its component representative value, confidence level, and the results of the most recent updates. This state preserves the group-level decision context, rather than just storing the final string, thus enabling subsequent updates without rereading the original historical records.
[0052] When the current query time t′, which is later than the historical fusion time t, involves the entity attribute pair (e,a) again, the query layer first obtains the current entity cluster C_e(t′) according to the aforementioned process. The newly added evidence ΔR_e,a(t,t′) is defined as {r∈C_e(t′) | ts(r)>t and record r contains attribute a}, that is, the subset of records in the current entity cluster that are newly added after the historical state time and provide the target attribute value; the entity cluster also contains the processed historical history, so the newly added evidence is not the entity cluster itself. Before processing the newly added evidence, a time decay σ_i^(2)(t′)=γ^(t′-τ_i)·σ_i^(2)(τ_i) is applied to each historical cross-source evidence group h_i according to its most recent update time τ_i, where γ∈(0,1] is the decay factor for each preset time unit; the smaller γ is, the faster the weight of historical evidence decreases.
[0053] Subsequently, the system performs the same value normalization, intra-source grouping, source reliability weighting, cross-source grouping, and copy decay on ΔR_e,a(t,t′) as the first fusion, to obtain the incremental cross-source evidence set ΔH={Δh_1,...,Δh_(m_Δ)}. For each Δh, the system uses the normalized representative value and f_equiv to search for an equivalent group h in the historical group H. If an equivalent group exists, the intra-source group, contributing source, supporting record identifier, and original value of Δh are combined with h. For the same source s, the historical intra-source support σ_g,hist^(1) that has been decayed by γ^(t′-τ) is added to the current incremental intra-source support σ_g,Δ^(1) to obtain the updated support of that source. Then, the support of all sources is summed to obtain a new Σ_h, and C_h and σ_h^(2) are recalculated based on the merged source set and the current copy probability. Since the newly added evidence satisfies ts(r)>t, its record identifier does not overlap with historical supporting records. If duplicate record identifiers are found in the actual system, duplicates are removed before calculation. If no equivalent group exists, Δh is added to H as a new historical group. Whether or not the evidence is equivalent determines not only whether to update the original group's support, but also whether the newly added evidence is merged into existing candidate values or forms new candidate values.
[0054] During partial updates to the relationship graph, the fifth step of relationship determination is performed on each newly added group or modified historical group and its relationships with other groups in H: if the set of canonical terms forms a proper subset relationship, then an inclusion edge is added or updated; if no inclusion relationship exists and the conditions for word overlap and conflict are met, then a conflicting edge is added or updated; existing edges are deleted when the canonical representative value of their endpoints changes and no longer meets the corresponding conditions. Thus, the update scope covers all edges that may have changed due to new evidence, while excluding historical records unrelated to the current update.
[0055] After completing group-level merging and partial updates to the relationship graph, the system identifies the components containing newly added groups or modified historical groups, as well as components with conflicting edges to these components, as affected components. Only the members, representative values, and support of affected components are recalculated. Unaffected components retain the component support stored in their historical state. The system takes the global maximum between the "updated affected component support" and the "unaffected component support" to recalculate θ_truth, thus eliminating the need to reprocess the underlying records of unaffected components. Components that meet the selection criteria after this round of recalculation are called update candidate components, and their representative values are the candidate fusion truth values for this round. If an update candidate component is connected to an existing truth component through an containing edge, the two are merged into the same component, and their representative values and confidence are updated simultaneously. If there is no explicit relationship between the two for multiple truth attributes and the update candidate component reaches the threshold, it is added to T as a new truth value. If there are conflicting edges between the two, or if an update candidate component with a single truth attribute differs from the current truth value, the update candidate component is treated as a challenge component. If the confidence level of the challenge component is at least δ_stab higher than that of the current truth component, it is directly replaced. δ_stab is a pre-set confidence difference threshold between 0 and 1. If the difference is less than δ_stab, the results of the most recent u refreshes are queried for voting, where u is the observation window length and u_min is the minimum number of support attempts required to submit the replacement, and 1 ≤ u_min ≤ u. A refresh refers to completing a new evidence retrieval, group-level merging, relation update, and truth recalculation for the same entity attribute at a new query time. The challenge component is only submitted for replacement if it meets the selection criteria at least u_min times in the most recent u refreshes and its confidence level is higher than that of the current truth component; otherwise, the current truth value is maintained. After the revision is completed, the updated H, D, and T are written back to the global truth database, and the state of TR_e,a is updated to t′ to avoid repeating the processing of newly added records in the next refresh.
[0056] In one example, a user requests a query to "find the author of a book published in 1997 with ISBN X12". The query predicate directly hits sources 2 and 3. Through selective evidence expansion, records with the same ISBN and book title but missing publication date from source 1 are retrieved. These three records are merged into the same entity cluster C_1; C_1 represents the first book entity involved in this query. Three evidence records are extracted for the author attribute, with original values of "David V. Hutton", "Hutton, David V. (author)", and "Hutton, David W.". After alignment, the first two are normalized to "Hutton, David V." and form two intra-source evidence groups, which are then merged into the same cross-source evidence group h_1; the third forms a cross-source evidence group h_2. Assuming the source reliability is 0.9, each intra-source group contains one deduplicated record with a copy probability of 0, then the support of h_1 is 1.8, and that of h_2 is 0.9. The two groups have no inclusion relationship, but their terms highly overlap and differ only in the last letter, thus establishing a conflict edge; each forms a candidate ground truth component B_1 and B_2. If δ_rel is 10% of the highest support, i.e., 0.18, then the support difference 0.9 is greater than δ_rel, and B_2 is removed; when η=0.6, the dynamic threshold is 1.08, B_1 reaches the threshold, and its component representative value "Hutton, David V." is used as the final fused ground truth, and the support record identifiers and confidence scores corresponding to source 1 and source 2 are output.
[0057] In this embodiment, dynamic updating refers to append-only access of newly added records, i.e., L(t′) = L(t) ∪ ΔL(t,t′), where ΔL(t,t′) is the set of newly accessed records between time t and t′. The temporal index fragment, the incremental range of the materialized base view, and the newly added evidence ΔR_e,a(t,t′) are all determined based on this append-only semantics.
[0058] It should be noted that the index structure in this invention can be implemented using inverted indexes, columnar indexes, vector indexes, LSH indexes, HNSW indexes, or combinations thereof; the equivalence function can be implemented using rule functions, statistical models, text embedding models, large language model discriminators, or combinations thereof; the global truth database can be stored in a relational database, key-value database, graph database, or columnar storage system. Substitution of the above implementation methods does not affect the essence of this invention.
[0059] This invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described query-driven incremental data fusion method in a heterogeneous data environment.
[0060] An embodiment of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described query-driven incremental data fusion method in a heterogeneous data environment.
[0061] An embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the above-described query-driven incremental data fusion method in a heterogeneous data environment.
[0062] Figure 3 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Please refer to... Figure 3 The electronic device provided in this embodiment includes a memory and a processor. The memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, they implement a query-driven incremental data fusion method in a heterogeneous data environment according to the present invention.
[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0067] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A query-driven incremental data fusion method in a heterogeneous data environment, characterized in that, The following steps are performed in sequence: Attribute semantic plane modeling is performed on the multi-source heterogeneous data in the dynamic data lake to obtain a set of attribute value assertions with records as boundaries; Temporal index chains are constructed for the attributes in the attribute semantic plane. Receive user queries and retrieve candidate records related to the user queries based on the temporal index and / or materialized views; Selective evidence expansion is performed on candidate records, merging candidate records and related records belonging to the same entity as the candidate records (excluding those specified in the query conditions) into entity clusters. Evidence records are extracted for the target attribute in each entity cluster, and entity attribute pairs are formed using the entity identifier and the target attribute. The global truth database with entity attribute pairs as keys is queried to determine whether the entity attribute pairs have historical fusion states. If no historical fusion state exists, value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth selection are performed on the evidence records to generate fused truth values and their confidence levels, which are then written to the global truth database. When a historical fusion state exists, new evidence records belonging to the entity attribute pair that were added after the corresponding time of the historical fusion state are determined from the current entity cluster. Value alignment and evidence merging are performed only on the new evidence records. The resulting incremental evidence group is merged with the historical evidence group in the historical fusion state at the group level. The evidence group relationship is locally updated and the fusion truth value and its confidence level are revised.
2. The method according to claim 1, characterized in that, The attribute semantic plane represents the snapshot L(t) of the data lake at time t as A(t); A(t) consists of records r, each record r represents a set or multiple sets of attribute value assertions (a,v), where a is the attribute name and v is the original attribute value that has not been aligned when the data is accessed. The temporal index maintains an index chain I_a=[I_a^(0)|I_a^(1)|...|I_a^(m)] for each attribute a, where I_a^(0) is the read-only basic index fragment of the initial snapshot, and I_a^(i) is the immutable append-only incremental index fragment corresponding to the i-th batch of newly added data, 1≤i≤m; each index item is associated with at least the attribute value, record identifier, source identifier and incremental fragment identifier, so as to return the corresponding record according to the condition expression, and determine the record added after a certain historical query or historical fusion state according to the incremental fragment identifier; The materialized base view stores the query signature, candidate record identifier set, and the latest merged incremental fragment identifier of the historical query: The materialized base view V_base(Q) is generated after the historical query Q is executed, and stores at least: the query signature consisting of conditional attributes, normalized conditional expressions, target attributes, and atomic query dependency structure; the candidate record identifier set R_id,base(Q) directly hit by the historical query; the view generation time t_base; and the latest merged incremental fragment identifier k_base at the time of generation; When the current query and the historical query have the same condition attributes and target attributes, equivalent normalized condition expressions, and isomorphic atomic query dependency graphs, it is determined that the query signatures of the two queries match and hit the materialized base view; when a hit occurs, the corresponding candidate record is read from the attribute semantic plane according to R_id,base(Q), and only the relevant attributes I_a^(k_base+1) to I_a^(m) are probed. The newly added hit record identifier is combined with R_id,base(Q) and deduplicated to obtain the candidate record set at the current query time; when a hit occurs, the complete index chain from I_a^(0) to I_a^(m) is retrieved. After the current query is completed, the identifier of the newly added candidate record in this round is written to R_id,base(Q), and k_base is updated to m.
3. The method according to claim 1, characterized in that, This also includes decomposing user queries into atomic query graphs, specifically: identifying conditional attributes, conditional expressions, target attributes, and node dependencies in the query, and generating atomic query graphs to parse the query into a directed acyclic graph G(Q) =<U,E_Q> Where U = {q_1, ..., q_n} is the set of atomic query nodes, and E_Q is the set of dependent edges; each atomic query node q_i is represented as<pred_i,a_c^(i),v_c^(i),a_t^(i),v_t^(i)> , pred_i is the set of predecessor nodes, a_c^(i) and v_c^(i) are the conditional attribute and conditional expression, respectively, a_t^(i) is the target attribute, and v_t^(i) is the projection or transformation expression performed on the target attribute retrieval result; the dependency edge (q_j, q_i) indicates that the conditional expression of q_i uses the output view of q_j; The system executes each atomic query node in topological order and performs parallel execution on nodes that have no dependencies. Each node outputs a set of candidate records that directly satisfy its condition expression.
4. The method according to claim 1, characterized in that, The selective evidence expansion includes three stages: In the first stage, the entity registry E_reg={(eid,s,rid)} is queried, the registered candidate records are mapped to existing entities, and all registered records with the same entity identifier in the entity registry are read as existing context records of that entity. In the second stage, unregistered candidate records are bucketed according to the key attribute set of their source. Approximate nearest neighbor search is performed in the corresponding bucket. Records that are not less than the source search threshold θ_s and missed by the original query conditions are retrieved from all records of the source. Based on whether the retrieved records have entity identifiers, existing entities are attached or unresolved prototype clusters are formed respectively. In the third stage, all registration records corresponding to existing entities and unresolved prototype clusters are taken as clusters to be merged. The centroid of the clusters is calculated and cross-source nearest neighbor retrieval is performed. Clusters with centroid similarity not lower than the global merging threshold θ_global are merged to obtain entity clusters containing query hit records and relevant records outside the query conditions. For an unresolved record r in source s, calculate the record embedding emb(r) based on the key attribute set K_s, map r to bucket B_s,b, and calculate the candidate nearest neighbor set N_s(r) = {r′∈B_s,b | sim(emb(r),emb(r′))≥θ_s}, where r′ represents a candidate similar record in the same bucket B_s,b as the unresolved record r, and θ_s is the source retrieval threshold of source s; divide N_s(r) into a set N_s^+(r) with existing entity identifiers and a set N_s^-(r) without entity identifiers; when N_s^+(r) is not empty, aggregate the similarity of each entity to r, and attach r to the existing entity with the highest aggregation similarity that reaches the attachment threshold θ_attach; r that does not reach θ_attach continues to be an unresolved record; when N_s^-(r) is not empty, construct a source adjacency graph with unresolved records as vertices and similarity reaching θ_s as edges, and form an unresolved prototype cluster for each connected component; After cross-source merging, if the merged result contains existing entities, the identifier of the existing entity is retained; if it contains multiple existing entities, the identifier of the entity with the highest or lowest preset priority is retained and the mapping of other records is updated; if it only contains unresolved prototype clusters, a new entity identifier is assigned from the monotonically increasing entity identifier counter and all records in the cluster are written to the entity registry.
5. The method according to claim 1, characterized in that, The process includes value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth value selection. Specifically, it involves: using an entity attribute pair (e, a) as the processing unit, extracting evidence records containing attribute a from the entity cluster of entity e, where each evidence record provides an unaligned original value; converting the original values to canonical form and determining value equivalence; merging attribute value assertions that are canonically equivalent and provided by different records within the same source into an intra-source evidence group; and merging intra-source evidence groups with canonically equivalent representative values from different sources into cross-source evidence groups. Source evidence group; calculate the source support based on the number of deduplicated records and source reliability of the source evidence group, and attenuate the aggregate support of the cross-source evidence group based on the copy probability between sources; establish a cross-source evidence group relationship graph, distinguishing between inclusion relationships, conflict relationships, and no explicit relationships; construct candidate truth value components from cross-source evidence groups connected by inclusion relationships, compare candidate truth value components with conflicts, and select one or more candidate truth value components based on a dynamic threshold; use the component representative value of the selected component as the fusion truth value, and output the confidence score and the set of record identifiers that support the truth value.
6. The method according to claim 5, characterized in that, The intelligent heterogeneous data alignment engine converts the original values into a standardized form and determines their equivalence. This engine includes an atomic operation library, a global transformation rule library, a rule cache built on the context according to source attributes, an equivalence judgment function, a failure monitoring unit, and a rule generation unit. The source attribute context c=(s_i,a_i,s_j,a_j) represents the source and attribute of the two values to be compared. The transformation rules are formed by sequentially combining one or more atomic operations from the following: whitespace pruning, case unification, character replacement, delimiter splitting, field extraction, date normalization, numerical precision normalization, unit conversion, and additional marker deletion. When comparing the original values v_i and v_j, rules are first retrieved from the rule cache according to historical success rates; if the cache is not found, rules are retrieved from the global transformation rule library, resulting in the corresponding rules. The normative forms v_i^c and v_j^c are used, and the equivalence judgment function f_equiv(v_i^c,v_j^c) outputs the equivalent or inequivalent result; the failure monitoring unit calculates the failure rate of context c in each fusion window = number of matching failures / number of rule calls; rule generation is started only when the failure rate of n_fail fusion windows exceeds the threshold τ and the cumulative number of failed samples is not less than N_min; the rule generation unit constructs context samples using high-confidence ground truth values and unselected values of the same entity attribute pair, generates candidate rules, and verifies the positive example alignment success rate and negative example mismerging rate on the verification samples that did not participate in the generation; only candidate rules with a positive example alignment success rate not lower than θ_rule and a negative example mismerging rate not higher than θ_false are written to the rule cache, and are written to the global transformation rule library after they remain valid.
7. The method according to claim 5, characterized in that, The cross-source evidence set support is calculated by combining source reliability and inter-source copy probability. This includes maintaining Beta distribution parameters α_s,q and β_s,q for source s and atomic query q, initializing α_s,q = κ·w_s^(0)(q) and β_s,q = κ·(1-w_s^(0)(q)) with the initial reliability w_s^(0)(q) obtained from the source profile and the prior strength κ. After each fusion round, the positive feedback number Δn_s,q^good and the negative feedback number Δn_s,q^bad are obtained based on whether the intra-source evidence set of source s is covered by the final truth component, and updated according to α_s,q←φ·α_s,q+Δn_s,q^good and β_s,q←φ·β_s,q+Δn_s,q^bad. φ is the forgetting factor; w_s(q)=α_s,q / (α_s,q+β_s,q) is used as the source reliability; for the source evidence group g, the source support σ_g^(1)=|g|·w_s(q) is calculated based on the number of different record identifiers it contains |g|; the source evidence groups with equivalent normative representative values and different sources are combined into a cross-source evidence group h to obtain the contributing source set S_h and the original support Σ_h=∑_(g∈h)σ_g^(1); the attribute-level copy probability is maintained for the source pair, and the average copy probability C_h of the source pair in the cross-source evidence group is calculated; the cross-source evidence group support is calculated according to σ_h^(2)=((1-C_h)+C_h / |S_h|)·Σ_h, where C_h=0 when |S_h|≤1.
8. The method according to claim 5, characterized in that, A relationship graph of cross-source evidence groups is established using a relationship-aware evidence arbitrator to distinguish between inclusion relationships, conflict relationships, and no explicit relationships. The relationship-aware evidence arbitrator normalizes and segments the canonical representative value v_i of cross-source evidence group h_i to obtain a term set T_i. When T_j is a proper subset of T_i, an inclusion edge pointing from h_i to h_j is established. For two cross-source evidence groups without inclusion edges, a conflict edge is established based on the embedding similarity reaching a conflict threshold or the existence of only one non-substitutable term difference, provided the word overlap rate reaches a threshold; the rest are marked as no explicit relationship. When constructing candidate truth components, each connected component formed after ignoring the direction of the inclusion edge is considered as a candidate truth component B. Therefore, cross-source evidence groups that directly include each other and those indirectly connected via inclusion chains belong to the same candidate truth component. The component support S(B) is the sum of all cross-source evidence groups within the component. The component representative value is selected from the canonical representative values with an in-degree of zero, where zero in-degree indicates that the value is not included by other values within the component. When multiple values have zero in-degree, they are determined sequentially according to their corresponding cross-source support from high to low, the number of contributing sources from high to low, the canonical string encoding from low to high, and the group identifier from low to high. Two components are considered to be in conflict if there is a conflicting edge connecting any cross-source evidence groups within the two components. Components with higher support are retained, while those with lower support are removed. When support is equal, one component is retained according to a preset deterministic order of component representative value and component identifier. When the difference in support is less than the conflict margin δ_rel, the conflict pair is recorded as a low-margin conflict pair for stability assessment during subsequent incremental maintenance. For the components retained after conflict elimination, θ_truth = η·max_k is calculated. S(B_k), the threshold value changes with the highest component support in the current round; for a single truth value attribute, only the component that reaches the threshold and has the highest support is output. A single truth value attribute is an attribute that specifies that the same entity has at most one valid value for the attribute according to the attribute pattern; for multiple truth value attributes, all components that reach the threshold and do not conflict with each other are output. Each selected component corresponds to a fused truth value.
9. The method according to claim 1, characterized in that, The global truth database stores the structured fusion state TR_e,a(t) for entity attribute pairs (e,a).<e,a,H,D,T> , where t is the last update time of the state; H is the set of historical cross-source evidence groups, each cross-source evidence group at least stores the group identifier, canonical representative value, the source evidence groups contained therein, the source support corresponding to each source, the set of contributing sources, the set of support record identifiers, the set of original values, the cross-source support and the most recent update time; D is the evidence group relationship graph with cross-source evidence groups as vertices and containing edges or conflicting edges as edges, and records low margin conflict pairs; T is the set of currently selected truth components. Each truth component must store at least the component members, the component representative value, the merged truth value, the component support, the confidence, the set of support record identifiers, and the selection results in the most recent refreshes. Incremental fusion truth maintenance includes: reading TR_e,a(t) from the global truth database; obtaining the current entity cluster C_e(t′) of entity e at the current query time t′, and selecting records that satisfy ts(r)>t and contain attribute a to form a new evidence record set ΔR_e,a(t,t′); performing time decay on the source intra-source support and cross-source support of each historical evidence group according to their most recent update time; performing the same normalization, source intra-source grouping, cross-source grouping, and support calculation as the first fusion only on ΔR_e,a(t,t′) to obtain the incremental cross-source evidence group set ΔH; Each incremental cross-source evidence group is matched with H for equivalent normalized representative values. When they are equivalent, the source evidence groups, contributing source sets, supporting record identifier sets, and original value sets of the two are merged. For the same source, the decayed historical source support is added to the incremental source support, and then the cross-source support is recalculated based on the merged source set and the current copy probability. When they are not equivalent, the incremental cross-source evidence group is added to H as a new group. Only the included edges and conflicting edges connected to the new group or the modified historical group are recalculated. The affected candidate truth components are reconstructed based on the updated H and D, and T and confidence are revised. Finally, the state time is updated to t′. After merging and updating the relationships between the incremental evidence group and the historical evidence group, the component support calculation, conflict comparison, and dynamic threshold screening are re-executed for candidate truth components that include newly added evidence groups or modified historical evidence groups. Components newly added to the selected set or whose support changes in this round are called updated candidate components. For multiple truth attributes, when an updated candidate component has an inclusion connection with the current truth component, they are considered the same component, and the component's representative value and confidence are updated. When there is no explicit relationship between them and the updated candidate component reaches the dynamic threshold, it is added to T as a new truth component. When an updated candidate component has a conflict edge with the current truth component, it is called a challenging component. For a single truth attribute, except for the current truth component, the components are considered as follows: All selected components are used as challenge components. The confidence difference between the challenge component and the challenged current truth component is compared. When the confidence of the challenge component is at least δ_stab higher than that of the current truth component, it is directly replaced, where δ_stab is a preset confidence difference threshold between 0 and 1. When the difference is less than δ_stab, the results of the most recent u refreshes are read for stability voting, where u is a positive integer and 1≤u_min≤u. A refresh refers to completing a new evidence retrieval, group-level merging, relation update, and truth recalculation for the same entity attribute at a new query time. The challenge component is only submitted for replacement if it has been selected at least u_min times in the most recent u refreshes and its confidence is higher than that of the current truth component; otherwise, the current truth is maintained.
10. A query-oriented incremental data fusion system in a dynamic data lake, characterized in that, include: The attribute semantic plane construction module is used to perform attribute semantic plane modeling on multi-source heterogeneous data in dynamic data lakes. It obtains a set of attribute value assertions that represent each physical row, semi-structured object, log entry or text fragment as sharing the same record identifier, and saves the source identifier and access time for each record. The temporal index evidence retrieval module is used to construct temporal index chains consisting of basic index fragments and incremental index fragments that are only appended to the attributes in the attribute semantic plane; receive user queries, identify conditional attributes, conditional expressions, target attributes, and node dependencies in the query, and generate atomic query graphs; according to the atomic query graphs, retrieve candidate records that directly satisfy the query conditions based on the temporal index chains and materialized base views, wherein the materialized base views store the query signatures of historical queries, the candidate record identifier set, and the identifier of the latest incremental fragment that has been merged; The selective evidence expansion module is used to perform selective evidence expansion on candidate records, merging candidate records and related records belonging to the same entity as the candidate records that are not subject to the query conditions into entity clusters. The global truth database retrieval module is used to extract evidence records for the target attributes in each entity cluster and form entity attribute pairs with entity identifiers and target attributes; it queries the global truth database with entity attribute pairs as keys to determine whether the entity attribute pairs have historical fusion states. The adaptive alignment hierarchical fusion module is used to perform value alignment, intra-source evidence merging, cross-source evidence merging, support calculation, evidence group relationship identification, and truth value selection on the evidence records when there is no historical fusion state, and to generate fused truth values and their confidence levels and write them into the global truth value library. In the incremental fusion truth value maintenance module, when there is a historical fusion state, the module determines the new evidence records that are added after the corresponding time of the historical fusion state and belong to the entity attribute pair from the current entity cluster. Only the new evidence records are value aligned and evidence merged. The resulting incremental evidence group is merged with the historical evidence group in the historical fusion state at the group level. The evidence group relationship is locally updated and the fusion truth value and its confidence are revised.
