Massive data correlation fusion method, device and medium

Abstract

The application provides a mass data association fusion method, device and medium, first, receiving mass data from multiple heterogeneous data sources, preprocessing the data; then integrating into labeled data stream; wherein the labeled data stream includes a two-dimensional semantic vector and a dynamic quality label; finally inputting it into an AI association fusion model to obtain a fusion result; wherein the association fusion model is provided with an association fusion and semantic mapping layer, used for association fusion according to a hierarchical dynamic knowledge graph and a preset fusion rule, and semantic mapping according to a double-tower sub-model and a consistency arbitration rule. The application realizes real-time updating of the association relationship through the hierarchical dynamic knowledge graph, meets the real-time scene demand, reduces the mass data association complexity by using the double-tower model, accelerates the processing speed, improves the conflict resolution accuracy in combination with the arbitration rule. Finally, the efficient, accurate and real-time fusion of mass data is realized, providing a reliable basis for various data analysis and decision-making.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a mass data association fusion method, device and medium. BACKGROUND

[0002] With the development of digital economy, the mass data generated in many fields presents the characteristics of multi-source heterogeneity, and the data sources cover databases, real-time interfaces, log files, etc. Different identifiers are often used for the same entity in different data sources, resulting in a significant semantic gap in cross-source association. The traditional manual configuration of mapping rules is difficult to adapt to the rapid growth of data sources, and cannot meet the association efficiency requirements in dynamic scenarios.

[0003] At the same time, mass data generally have quality problems such as missing values, abnormal values, and duplicate records. Real-time transactions, risk control early warning, and user behavior analysis require strict data processing delay requirements. The traditional batch processing mode is difficult to balance data quality and real-time performance, and is prone to decision bias due to data association lag and inaccurate conflict resolution. In addition, the existing data fusion scheme has the problems of static knowledge graph update, poor adaptability of bucketing strategy, and lack of scene adjustment ability of fusion rules, which further restricts the efficiency and reliability of mass data association fusion. Therefore, it is urgent to build an association fusion technology system that takes into account real-time performance, accuracy, and scene adaptability. SUMMARY

[0004] The embodiments of the present application provide a mass data association fusion method, device and medium to solve the problem that the traditional batch processing mode at the present stage is difficult to balance data quality and real-time performance, and is prone to decision bias due to data association lag and inaccurate conflict resolution.

[0005] In a first aspect, the embodiments of the present application provide a mass data association fusion method, comprising:

[0006] Receiving mass data from multiple heterogeneous data sources, and preprocessing the data;

[0007] Integrating the preprocessed data into labeled data stream; wherein the labeled data stream includes a double-dimensional semantic vector and a dynamic quality label;

[0008] Inputting the labeled data stream into an AI association fusion model to obtain a fusion result;

[0009] The AI association fusion model is provided with an association fusion and semantic mapping layer; the association fusion and semantic mapping layer is used for association fusion according to a hierarchical dynamic knowledge graph and a preset fusion rule, and semantic mapping according to a double-tower sub-model and a consistency resolution rule.

[0010] In a possible implementation, the labeled data stream is input into an AI correlation fusion model to obtain a fusion result, including:

[0011] The two-dimensional semantic vectors in the labeled data stream are subjected to dimension normalization processing to match the vector dimensions with the preset input dimensions of the model; and the dynamic quality labels are subjected to weight initialization based on the credibility levels of the data sources to obtain an adapted data stream;

[0012] The preset fusion rules and the adapted data stream are transmitted to a correlation fusion and semantic mapping layer to obtain a preliminary correlation set and an initial semantic mapping result to determine the fusion result;

[0013] The correlation fusion and semantic mapping layer extracts, based on the hierarchical structure of the hierarchical dynamic knowledge graph, correlation features corresponding to the adapted data stream from each level, and combines the preset fusion rules to preliminarily correlate and aggregate homogenous data and correlated data in the adapted data stream to generate the preliminary correlation set.

[0014] The correlation fusion and semantic mapping layer calls a double-tower submodel to perform semantic mapping processing on the preliminary correlation set; the first subtower is configured to receive two-dimensional semantic vectors in the preliminary correlation set and extract local semantic features through a convolutional neural network, and the second subtower is configured to receive semantic representation vectors of corresponding correlation nodes in the hierarchical dynamic knowledge graph and extract global semantic features through a Transformer encoder; and the initial semantic mapping result is determined according to the cosine similarity of the output features of the two subtowers.

[0015] In a possible implementation, the hierarchical dynamic knowledge graph includes three levels of a basic data layer, a feature correlation layer and a semantic fusion layer, and the levels are dynamically connected through correlation edges; the basic data layer stores original entity and attribute information of each data source, the feature correlation layer stores feature-level correlation relationships between entities, and the semantic fusion layer stores semantic-level correlation relationships between entities and domain knowledge rules; the adapted data stream is transmitted to the correlation fusion and semantic mapping layer to obtain a preliminary correlation set and an initial semantic mapping result, including:

[0016] The reference entity information corresponding to each entity in the adapted data stream is extracted from the basic data layer, an entity recognition algorithm is used to recognize the entity types in the adapted data stream, and an entity alignment algorithm is used to match the to-be-fused entities with the reference entities to mark homogenous entities and suspected correlation entities;

[0017] Based on the preset fusion rules and the feature correlation relationships stored in the feature correlation layer, the marked homogenous entities and suspected correlation entities are subjected to feature-level correlation aggregation to obtain the preliminary correlation set;

[0018] The domain knowledge rules in the semantic fusion layer are called to perform semantic enhancement on the preliminary association set output by the feature association layer, to obtain an initial semantic mapping result.

[0019] In a possible implementation, the preset fusion rule includes a basic rule set and scene adaptation parameters; the basic rule set includes a data priority rule, an attribute fusion rule, and an association strength calculation rule; the scene adaptation parameters include a rule triggering threshold, a feature weight coefficient, and a scene type identifier, and are used to dynamically adjust the execution logic of the basic rule set according to a specific application scenario; based on the preset fusion rule and the feature association relationship stored in the feature association layer, the feature-level association aggregation is performed on the labeled homologous entities and suspected associated entities, to obtain a preliminary association set, including:

[0020] For the labeled homologous entities, the attribute fusion rule in the basic rule set is called according to the attribute type of the homologous entities, to perform fusion processing on the multi-source attribute data, and generate a unified entity attribute description;

[0021] For the labeled suspected associated entities, the association strength calculation rule in the basic rule set and the association relationship stored in the feature association layer are used to calculate the association strength between the entities;

[0022] According to the association threshold set in the scene adaptation parameters, the suspected associated entities whose association strength meets the threshold condition are determined to have an association, and the feature information of the suspected associated entities is aggregated to form the preliminary association set.

[0023] In a possible implementation, the method includes:

[0024] The data-driven semantic vector of the entity is extracted from the preliminary association set as the input of the first sub-tower, and the graph-driven semantic vector corresponding to the entity is extracted from the hierarchical dynamic knowledge graph as the input of the second sub-tower;

[0025] The feature encoding is performed on the input vectors of the first sub-tower and the second sub-tower respectively, to output corresponding semantic feature vectors;

[0026] The similarity calculation is performed on the semantic feature vectors output by the first sub-tower and the semantic feature vectors output by the second sub-tower, to generate an initial semantic mapping result;

[0027] Based on a preset consistency adjudication rule, the initial semantic mapping result is checked and corrected to identify and solve semantic conflicts;

[0028] According to the adjudication result, a final semantic mapping output is generated, and the conflict cases and / or artificial correction results generated in the adjudication process are used to iteratively optimize the double-tower sub-model and / or the consistency adjudication rule.

[0029] In a possible implementation, the initial semantic mapping result is checked and corrected based on a preset consistency adjudication rule to identify and resolve semantic conflicts, including:

[0030] The semantic similarity score is compared with a first threshold value. If the semantic similarity score is higher than the first threshold value, it is directly determined that the semantic mapping is successful, and the result is output. If the semantic similarity score is lower than a second threshold value, it is directly determined that the semantic mapping fails, and the mapping is rejected. If the semantic similarity score is between the first threshold value and the second threshold value, a secondary adjudication is triggered. The first threshold value is higher than the second threshold value.

[0031] For the entity triggering the secondary adjudication, the associated relationship of the entity in each data source is obtained by querying the hierarchical dynamic knowledge graph, and a consistency score of the associated relationship is calculated. The semantic similarity score and the relationship consistency score are weighted and summed to obtain a comprehensive adjudication score. If the comprehensive adjudication score is higher than a third threshold value, it is determined that the semantic mapping is successful, otherwise a tertiary adjudication is triggered.

[0032] For the entity triggering the tertiary adjudication, a conflict detection is performed by using a preset logical rule library. If a logical conflict is detected, it is determined that the semantic mapping fails. If no logical conflict is detected, the comprehensive adjudication score is corrected in combination with a dynamic quality label, and whether the mapping is successful is determined according to the final score after the correction.

[0033] In a possible implementation, the AI association fusion model is further provided with a dynamic optimization and scheduling layer, and the method further includes:

[0034] Through the dynamic optimization and scheduling layer, the running state and output result of the association fusion and semantic mapping layer are monitored in real time to collect performance data.

[0035] Based on preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly.

[0036] The optimization instructions are issued to the association fusion and semantic mapping layer to dynamically adjust and optimize the calculation tasks, model parameters and fusion rules therein.

[0037] In a possible implementation, based on preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly, including:

[0038] Based on the performance data, the key bottlenecks affecting the performance are identified by a preset optimization algorithm, and optimization instructions are generated. The optimization instructions include: adjusting the network weight of the double-tower sub-model, updating the threshold value in the consistency adjudication rule, or performing an add, delete or modify operation on the entity or relationship of the hierarchical dynamic knowledge graph.

[0039] The embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor realizes the above-mentioned mass data association fusion method when executing the computer program.

[0040] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the above-mentioned mass data association fusion method.

[0041] Compared with the prior art, the embodiment of the present application provides a mass data association fusion method, first, receiving mass data from multiple heterogeneous data sources, preprocessing the data; then integrating the preprocessed data into labeled data stream; wherein the labeled data stream includes a two-dimensional semantic vector and a dynamic quality label; finally, inputting the labeled data stream into an AI association fusion model to obtain a fusion result; wherein the AI association fusion model is provided with an association fusion and semantic mapping layer; the association fusion and semantic mapping layer is used for association fusion according to the hierarchical dynamic knowledge graph and the preset fusion rule, and semantic mapping according to the double-tower sub-model and the consistency adjudication rule. The present application realizes real-time update of the association relationship through the hierarchical dynamic knowledge graph, meets the real-time scene demand; the double-tower model is used to reduce the mass data association complexity and speed up the processing speed; the consistency adjudication rule containing the urgency is combined to improve the conflict resolution accuracy. Finally, the mass data is efficiently, accurately and real-timely fused, providing a reliable basis for various data analysis and decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the implementation flowchart of the mass data association fusion method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0043] The embodiment of the present application will be described in detail below with reference to the drawings.

[0044] Figure 1 is the implementation flowchart of the mass data association fusion method provided by the embodiment of the present application. As shown in Figure 1 , the method can comprise:

[0045] S110, receiving mass data from multiple heterogeneous data sources, preprocessing the data;

[0046] S120, integrating the preprocessed data into labeled data stream; wherein the labeled data stream includes a two-dimensional semantic vector and a dynamic quality label;

[0047] S130, inputting the labeled data stream into an AI association fusion model to obtain a fusion result;

[0048] The AI correlation fusion model is provided with a correlation fusion and semantic mapping layer; the correlation fusion and semantic mapping layer is used for correlation fusion according to a hierarchical dynamic knowledge graph and a preset fusion rule, and is used for semantic mapping according to a double-tower submodel and a consistency adjudication rule.

[0049] In the embodiment of the application, first, a stream-batch integrated access mechanism is used to synchronously receive massive heterogeneous data from different systems, specifically covering three types of core data sources:

[0050] Structured data source: such as user table and order table in a relational database, a stable connection is established through a JDBC interface, data is pulled at a preset frequency (such as 1 time per second for real-time transaction data and 1 time per hour for historical data), and the structured data field is ensured to be complete;

[0051] Semi-structured / Unstructured data source: such as JSON format logs returned by an API interface and text log files (such as user behavior logs and system running logs), which are connected through a Kafka, Flink or other stream processing framework, and a'shard parsing + format verification' strategy is adopted. Large volume log files (such as more than 10 GB) are divided into 1 MB shards, and the format legality of each shard is verified;

[0052] Third-party data source: such as Excel data tables and CSV files provided by cooperative institutions, which are accessed through OSS, FTP or other file transmission protocols, and the file encoding (UTF-8, GBK, etc.) and separator (comma, tab, etc.) are automatically identified to realize automatic access without manual configuration of the format.

[0053] The preprocessing stage is carried out around 'data cleanliness improvement' and'subsequent association adaptation', and includes three key operations:

[0054] Preliminary filtering of redundant and abnormal data: a 'rule filtering + statistical analysis' dual mechanism is adopted. First, obviously invalid data is removed through business rules; then, extreme abnormal values in numerical data are identified through the 3sigma criterion, and low-value data is removed through null value proportion statistics to reduce redundancy in subsequent processing;

[0055] Data format standardization: the same type of fields from different data sources are standardized. For example, time fields are uniformly converted to a standard format; for numerical fields, 2 decimal places are uniformly retained to avoid association failure caused by format differences;

[0056] Core feature extraction: for each piece of data, core features required for subsequent semantic modeling and quality evaluation are extracted.

[0057] In the embodiment of the application, a 'general semantic + domain semantic' two-dimensional coding is used to generate a semantic vector that can be associated across sources:

[0058] General semantic vector: Use a pre-trained word vector model (such as Word2Vec, Glove) to encode entity identifiers and attribute values in the data, convert text / numerical information into low-dimensional dense vectors (such as 128 dimensions), and ensure that the vector similarity of "same meaning information" in different data sources is high;

[0059] Domain semantic vector: Introduce a domain-specific vocabulary to "fine-tune" the general semantic vector. Avoid association errors caused by "same name, different meaning";

[0060] Vector integration: Concatenate the general semantic vector and the domain semantic vector according to the weight to form the final two-dimensional semantic vector, each vector carrying "data meaning + domain attribute" dual information, providing a semantic basis for cross-source association.

[0061] In the embodiments of the present application, based on the analysis results of the preprocessed data, dynamic quality labels containing "health status" and "association potential" are generated:

[0062] Health score (1-100 points): calculated from three dimensions of data integrity (number of missing fields / total number of fields), timeliness (current time-data collection time), and volatility (deviation of field value from historical mean).

[0063] Association potential score (1-10 points): based on the integrity of entity identifiers in the data (such as whether it contains a user unique identifier) and the clarity of semantic vectors (vector modulus fluctuation range).

[0064] Label integration: combine the health score and the association potential score into a dynamic quality label (such as "health 92+association 10" and "health 65+association 5"), and the label is refreshed in real time with data updates, providing a quantitative basis for subsequent screening of high-quality data.

[0065] Bind each piece of preprocessed data to the corresponding "two-dimensional semantic vector + dynamic quality label" to form a structured labeled data stream, ensuring that each piece of data carries "associable and assessable" key information.

[0066] In the embodiments of the present application, the two-dimensional semantic vector allows heterogeneous tags from different data sources to be automatically matched through vector similarity, avoiding the tediousness of traditional manual configuration of mapping rules, and the accuracy of cross-source entity association is improved from 65%.

[0067] In some embodiments, the labeled data stream is input into an AI association fusion model to obtain a fusion result, including:

[0068] Dimensional normalization processing is performed on the two-dimensional semantic vector in the labeled data stream to match the vector dimension with the preset input dimension of the model; at the same time, based on the credibility level of the data source, the dynamic quality label is initialized with a weight to obtain an adapted data stream;

[0069] The preset fusion rule and the adapted data stream are transmitted to an associated fusion and semantic mapping layer to obtain a preliminary association set and an initial semantic mapping result, so as to determine a fusion result.

[0070] The associated fusion and semantic mapping layer extracts associated features corresponding to the adapted data stream from each layer based on the hierarchical structure of the hierarchical dynamic knowledge graph, and combines the preset fusion rule to preliminarily associate and aggregate homologous data and associated data in the adapted data stream to generate a preliminary association set.

[0071] The associated fusion and semantic mapping layer calls a double-tower sub-model to perform semantic mapping processing on the preliminary association set; the first sub-tower is used to receive a two-dimensional semantic vector in the preliminary association set and extract local semantic features through a convolutional neural network, and the second sub-tower is used to receive a semantic representation vector of a corresponding associated node in the hierarchical dynamic knowledge graph and extract global semantic features through a Transformer encoder; and the initial semantic mapping result is determined according to the cosine similarity of the output features of the two sub-towers.

[0072] In the embodiment of the application, first, the labeled data stream is processed: on the one hand, the two-dimensional semantic vector is subjected to dimension normalization operation, and the dimension size of the vector is adjusted through a standardization algorithm to strictly match the input dimension requirement of the model, so as to ensure that the vector data can be effectively received and processed by the model; on the other hand, according to the credibility level of the data source (such as the credibility difference of different sources such as core database and third-party interface), the health score and the associated potential score in the dynamic quality label are subjected to weight initialization, and different weights are given to data from different sources, so as to finally form an adapted data stream and improve the matching degree of data and model.

[0073] Then, the preset fusion rule (including scenario-based association logic, conflict processing priority, etc.) and the adapted data stream are transmitted to the associated fusion and semantic mapping layer, and the layer is cooperatively processed to generate a preliminary association set and an initial semantic mapping result, which serves as a basis for determining the final fusion result.

[0074] The processing of the associated fusion and semantic mapping layer includes two core links:

[0075] Preliminary association set generation: This layer relies on the hierarchical structure of the layered dynamic knowledge graph (the hierarchical division of the core relationship layer and the non-core relationship layer) to accurately extract and adapt the association features (such as the hierarchical relationship between entities, attribute association, etc.) corresponding to the adapted data stream from each level of the knowledge graph; combined with the association logic defined in the preset fusion rules, the same source homologous data and associated data with potential association in the adapted data stream are preliminarily associated and aggregated, and the data with direct or indirect association is classified and integrated to form a preliminary association set, reducing the data dispersion of subsequent processing.

[0076] Initial semantic mapping processing: In the association fusion and semantic mapping layer, the double-tower sub-model is called to perform deep semantic mapping on the preliminary association set. Among them, the first sub-tower focuses on processing the two-dimensional semantic vectors in the preliminary association set, and extracts local detailed features (such as specific attributes of entities, local semantic associations) in the vectors through a convolutional neural network; the second sub-tower receives the semantic representation vectors of the associated nodes in the layered dynamic knowledge graph corresponding to the preliminary association set, and captures global semantic features (such as the position of an entity in the overall knowledge network and the global association with other entities) through a Transformer encoder; finally, the cosine similarity between the output features of the two sub-towers is calculated to determine the initial semantic mapping result, quantifying the semantic association degree between data and providing accurate basis for the final fusion result.

[0077] By constructing an AI association fusion system suitable for adapting massive heterogeneous data, the present application can effectively solve the semantic gap problem of cross-domain association of multi-source data, eliminate the identification difference between different data sources relying on two-dimensional semantic vectors, reduce the tedious operation of manually configuring mapping rules, and improve the cross-source association efficiency; the combination of intelligent cleaning and layered knowledge graph technology can accurately identify and handle data quality problems and conflict risks, reducing the decision bias caused by data quality defects; the application of double-tower sub-model and dynamic optimization mechanism greatly improves the efficiency of massive data association fusion, adapts to the rapid response demand in real-time scenarios; the full-link dynamic optimization capability ensures that the model can be continuously adjusted with the growth of data sources and changes in scenarios, adapts to the business characteristics of different fields, and finally realizes the efficient collaborative processing of massive heterogeneous data from access to fusion, providing high-quality and reliable data support for subsequent accurate decision-making, significantly improving the efficiency and accuracy of data fusion, and meeting the core needs of massive data association fusion in various industries.

[0078] In some embodiments, the hierarchical dynamic knowledge graph comprises three levels of a basic data layer, a feature association layer and a semantic fusion layer, and each level is dynamically connected through an association edge; wherein the basic data layer stores original entity and attribute information of each data source, the feature association layer stores feature-level association relationships between entities, and the semantic fusion layer stores semantic-level association relationships between entities and domain knowledge rules; the adapted data stream is transmitted to the association fusion and semantic mapping layer to obtain a preliminary association set and an initial semantic mapping result, including:

[0079] The reference entity information corresponding to each entity in the adapted data stream is extracted from the basic data layer, an entity recognition algorithm is used to recognize the entity type in the adapted data stream, and an entity alignment algorithm is used to match the to-be-fused entity with the reference entity, and mark the homologous entity and the suspected associated entity;

[0080] Based on the preset fusion rules and the feature association relationships stored in the feature association layer, the feature-level association aggregation is performed on the marked homologous entity and suspected associated entity to obtain a preliminary association set;

[0081] The domain knowledge rules in the semantic fusion layer are called to perform semantic enhancement on the preliminary association set output by the feature association layer to obtain an initial semantic mapping result.

[0082] In the embodiment of the application, the hierarchical dynamic knowledge graph adopts a three-level architecture design, including a basic data layer, a feature association layer and a semantic fusion layer, and information interaction and association update are realized between each level through an association edge with dynamic properties, forming a complete knowledge network system. Among them, the basic data layer serves as the bottom support of the knowledge graph, responsible for storing the original entity and its attribute information of each data source without deep processing, covering the basic data such as the basic identification and inherent characteristics of the entity, providing original information basis for upper layer processing; the feature association layer is in the middle layer, focusing on storing the association relationships between entities based on feature dimension, such as direct association of entities in attribute value, feature vector, etc., reflecting the surface feature association between entities; the semantic fusion layer as the top layer, stores the deep association relationship between entities based on semantic level, and integrates professional knowledge rules in the field, provides semantic interpretation and rule constraint for the association relationship.

[0083] The process of transmitting the adapted data stream to the association fusion and semantic mapping layer to obtain a preliminary association set and an initial semantic mapping result, which is specifically divided into three progressive links:

[0084] In a first step, entity matching and marking are completed based on the basic data layer. Reference entity information corresponding to each entity in the adapted data stream (i.e., stored and verified standard entity data) is retrieved and extracted from the basic data layer; each entity in the adapted data stream is subjected to type determination through an entity recognition algorithm to ensure consistency of entity classification; and an entity alignment algorithm is used to accurately compare the to-be-fused entities and the reference entities from multiple dimensions such as attribute similarity and identification consistency, to mark entities of the same nature as homologous entities (such as entities of different data sources that are identified differently but point to the same user), and mark entities with potential association clues as suspected associated entities (such as entities that share part of the characteristics but have not been explicitly confirmed to be associated), thereby laying a foundation for subsequent association and aggregation.

[0085] In a second step, a preliminary association set is generated based on the feature association layer. The preliminary association set is generated based on the preliminary fusion rules and the feature association layer. The preliminary fusion rules define the logic and priority of feature association, and the feature association layer stores the feature-level association relationships between entities. The homologous entities and suspected associated entities that have been marked are subjected to association and aggregation processing. For homologous entities, their attribute information is directly combined to eliminate redundancy. For suspected associated entities, the association strength is determined based on the feature association relationship, and entities that meet the association threshold are grouped into the same association group. Finally, a preliminary association set containing multiple association groups of entities is formed, and the surface feature-level integration of entities is achieved.

[0086] In a third step, semantic enhancement and mapping are achieved with the help of the semantic fusion layer. The preliminary association set output by the feature association layer is subjected to deep semantic processing based on the domain knowledge rules stored in the semantic fusion layer. The rationality of entity association in the preliminary association set is verified, and unreasonable associations are corrected. At the same time, semantic explanations are added to the entity association based on the semantic-level association relationship, so that the preliminary association set is upgraded from feature-level aggregation to semantic-level fusion, and finally an initial semantic mapping result containing semantic association information is obtained, providing deep semantic basis for subsequent accurate fusion.

[0087] In some embodiments, the preliminary fusion rules include a basic rule set and scene adaptation parameters. The basic rule set includes data priority rules, attribute fusion rules, and association strength calculation rules. The scene adaptation parameters include rule trigger thresholds, feature weight coefficients, and scene type identifiers, which are used to dynamically adjust the execution logic of the basic rule set according to specific application scenarios. Based on the preliminary fusion rules and the feature association relationships stored in the feature association layer, the homologous entities and suspected associated entities that have been marked are subjected to feature-level association aggregation, and a preliminary association set is obtained, including:

[0088] For the marked homologous entities, the corresponding attribute fusion rules in the basic rule set are called according to the attribute types of the homologous entities, and the multi-source attribute data is subjected to fusion processing to generate a unified entity attribute description.

[0089] For the marked suspected associated entities, the association strength between entities is calculated according to the association strength calculation rules in the basic rule set and the association relationship stored in the feature association layer;

[0090] According to the association threshold set in the scene adaptation parameter, the suspected associated entities whose association strength meets the threshold condition are determined to exist association, and their feature information is aggregated to form a preliminary association set.

[0091] In the embodiment of the application, the preset fusion rule adopts a double-layer design of "basic rule + scene adaptation", which not only guarantees the generality of the fusion logic, but also adapts to the differentiated needs of different business scenarios:

[0092] The basic rule set: as the core framework of the rule system, it covers three types of key rules to provide general logic support for data association fusion. Among them, the data priority rule is used to define the priority level of different data sources (such as core database data priority higher than third-party interface data), to avoid disordered selection when multiple source data conflicts; the attribute fusion rule defines the fusion method of attribute data for different attribute types of entities (such as text attribute de-duplication and merging, numerical attribute taking mean value / maximum value, time attribute taking earliest / latest value); the association strength calculation rule provides a method to quantify the degree of association between entities, which integrates entity attribute similarity, feature vector matching degree and other indicators to output a comparable association strength value, providing a basis for determining whether the entities are associated.

[0093] Scene adaptation parameters: as a dynamic adjustment component of the rule system, it is used to flexibly optimize the execution logic of the basic rule set according to the characteristics of specific application scenarios. Among them, the rule triggering threshold defines the critical condition for the rule to take effect (such as only when the data health score is higher than a certain threshold, the attribute fusion rule is executed); the feature weight coefficient adjusts the contribution proportion of different features in the association strength calculation; the scene type identifier is directly associated with the scene-specific rule configuration to ensure that the rule system accurately matches the scene requirements.

[0094] For the marked homologous entities and suspected associated entities, the preliminary association set is finally generated through type-based processing and step-by-step calculation, and the specific process includes three core links:

[0095] 1. Attribute fusion processing of homologous entities

[0096] Homologous entities refer to entities that are essentially the same but come from different data sources (such as user entities of the same user in system A and system B), and the core of their association aggregation is to eliminate attribute redundancy and generate a unified description:

[0097] Firstly, the attribute types of the marked homologous entities are identified one by one; then, the corresponding attribute fusion rules are called from the basic rule set according to the attribute types. For example, for the text attribute "user name", if there are slight differences in multi-source data, the text deduplication and merging rule is called to retain the standard expression; for the numerical attribute "account balance", if there are deviations in multi-source data, the numerical attribute mean value rule is called to calculate the unified balance value; for the time attribute "last login time", the latest value rule of the time attribute is called to determine the unique login time; finally, all the fused attributes are integrated to form the unified attribute description of the homologous entity, avoiding the information confusion caused by the dispersion of multi-source attributes.

[0098] 2. Association strength calculation of suspected associated entities

[0099] Suspected associated entities refer to entities that have potential association clues but have not been explicitly confirmed to be associated. The core of association aggregation is to quantify the association strength to determine whether they are truly associated:

[0100] Firstly, the association strength calculation rules are called from the basic rule set to determine the calculation dimensions and formula of the association strength (such as association strength = attribute similarity * 40% + feature vector matching degree * 60%); secondly, the association relationship data of the suspected associated entity pair is extracted from the feature association layer; then, the association relationship data is substituted into the association strength calculation rule, and the adjusted feature weight coefficient in the scene adaptation parameter is combined to calculate the specific association strength value of the entity pair, and to convert "suspected association" into a quantifiable numerical index.

[0101] 3. Association determination and aggregation based on association threshold

[0102] The effective association is screened through the association strength threshold to complete the construction of the preliminary association set:

[0103] Firstly, the association threshold set for the current scene in the scene adaptation parameter is read; then, the association strength value of the suspected associated entity is compared with the threshold. If the association strength is greater than or equal to the threshold, it is determined that the entity pair "has effective association"; if the association strength is less than the threshold, it is determined that "the association is insufficient" and is not included in the association range; finally, for the entity pair determined to "have effective association", the feature information of each entity is aggregated and integrated, and is supplemented to the unified attribute description of the homologous entity to form a preliminary association set containing "homologous entity group + effective associated entity pair", laying a foundation for the subsequent depth fusion in the semantic level.

[0104] In some embodiments, the double-dimensional semantic vector includes a data-driven semantic vector and a graph-driven semantic vector; the labeled data stream is input into an AI association fusion model to obtain a fusion result, including:

[0105] extract the data-driven semantic vector of the entity from the preliminary association set as the input of the first sub-tower, and extract the graph-driven semantic vector corresponding to the entity from the hierarchical dynamic knowledge graph as the input of the second sub-tower;

[0106] The first sub-tower and the second sub-tower respectively encode the input vectors to output corresponding semantic feature vectors;

[0107] The semantic feature vectors output by the first sub-tower and the semantic feature vectors output by the second sub-tower are subjected to similarity calculation to generate an initial semantic mapping result;

[0108] Based on the preset consistency adjudication rule, the initial semantic mapping result is checked and corrected to identify and solve semantic conflicts;

[0109] According to the adjudication result, a final semantic mapping output is generated, and the conflict cases and / or artificial correction results generated in the adjudication process are used to iteratively optimize the double-tower sub-model and / or the consistency adjudication rule.

[0110] In some embodiments, based on the preset consistency adjudication rule, the initial semantic mapping result is checked and corrected to identify and solve semantic conflicts, including:

[0111] The semantic similarity score is compared with the first threshold value, if it is higher than the first threshold value, it is directly determined as semantic mapping success, and the output result is output; if it is lower than the second threshold value, it is directly determined as semantic mapping failure, and the mapping is rejected; if it is between the first threshold value and the second threshold value, the second-level adjudication is triggered; wherein the first threshold value is higher than the second threshold value;

[0112] For the entity triggering the second-level adjudication, the hierarchical dynamic knowledge graph is queried to obtain the association relationship in each data source, and the consistency score of the association relationship is calculated; the semantic similarity score and the relationship consistency score are weighted and summed to obtain a comprehensive adjudication score; if the comprehensive adjudication score is higher than the third threshold value, it is determined as semantic mapping success, otherwise the third-level adjudication is triggered;

[0113] For the entity triggering the third-level adjudication, the conflict is detected by using the preset logic rule library; if a logical conflict is detected, it is determined as semantic mapping failure; if no logical conflict is detected, the comprehensive adjudication score is corrected in combination with the dynamic quality label, and whether the mapping is successful is determined according to the final score after correction.

[0114] In the embodiment of the present application, the process of inputting the labeled data stream into the AI association fusion model to generate a fusion result takes the double-tower sub-model as the core, combines the consistency adjudication and iterative optimization mechanism, realizes the accurate conversion from preliminary association to final fusion result, and is specifically divided into five progressive links:

[0115] 1. Dual-tower sub-model input vector extraction.

[0116] To achieve bidirectional semantic alignment between data and knowledge graph, two types of core vectors need to be extracted as dual-tower inputs:

[0117] First sub-tower input: Extract the "data-driven semantic vector" of the entity from the preliminary association set. This vector is generated based on the entity's own two-dimensional semantic vector (general + domain), directly reflecting the semantic features learned from the original data, without external association information from the knowledge graph;

[0118] Second sub-tower input: Extract the "graph-driven semantic vector" of the entity from the hierarchical dynamic knowledge graph. This vector integrates the hierarchical association relationships related to the entity in the knowledge graph (such as attribute association in the basic data layer, feature association in the feature association layer, and semantic association in the semantic fusion layer), reflecting the semantic positioning of the entity in the global knowledge network.

[0119] 2. Feature encoding of dual-tower sub-model.

[0120] Through the dual-tower sub-model, the two types of input vectors are subjected to deep feature extraction, converting the original semantic vectors into more discriminative semantic feature vectors:

[0121] The first sub-tower focuses on semantic encoding of data itself, extracting local detailed features of the entity (such as exclusive attributes and short-term feature changes) from the data-driven semantic vector through convolutional neural networks and other structures;

[0122] The second sub-tower focuses on semantic encoding of knowledge graph association, extracting global association features of the entity (such as hierarchical relationships with other nodes and semantic positioning under domain rule constraints) from the graph-driven semantic vector through Transformer encoder and other structures;

[0123] Finally, the two sub-towers output the encoded "data-side semantic feature vector" and "graph-side semantic feature vector", providing a high-quality feature basis for subsequent similarity calculation.

[0124] 3. Initial semantic mapping result generation.

[0125] By quantifying the correlation degree of the features output by the two sub-towers, the initial semantic mapping result is generated:

[0126] The "data-side semantic feature vector" output by the first sub-tower and the "graph-side semantic feature vector" output by the second sub-tower are subjected to similarity calculation (such as cosine similarity), resulting in a semantic similarity score. The higher the score, the more consistent the semantic features learned from the data itself and the knowledge graph.

[0127] With the similarity score as the core, an initial semantic mapping result is generated, and it is preliminarily determined whether the semantic association between entities is established.

[0128] 4. Verification and correction based on consistency adjudication rules.

[0129] To solve the semantic conflicts (such as ambiguous similarity scores, inconsistent data and graph semantics) that may exist in the initial mapping result, multi-level verification and correction are required through preset consistency adjudication rules:

[0130] The rationality of the initial semantic mapping result is verified in combination with the association relationship of the hierarchical dynamic knowledge graph, the logic rule library and the dynamic quality label;

[0131] The results with conflicts (such as moderate similarity scores and contradictory association relationships) are corrected, and it is determined whether the semantic mapping is established, to ensure the accuracy of the results.

[0132] 5. Final result generation and model iteration optimization.

[0133] Based on the results of the consistency adjudication, the final semantic mapping result is output as the core component of the fusion result;

[0134] At the same time, the semantic conflict cases (such as misjudged mapping relationships) and artificial correction results generated in the adjudication process are collected as training data and fed back to the system:

[0135] The double-tower sub-model is iteratively trained to optimize the feature encoding capability and improve the accuracy of semantic similarity calculation;

[0136] The threshold, weight and other parameters of the consistency adjudication rules are adjusted to enhance the adaptability of the rules to complex scenarios, and the continuous evolution of the model and the rules is realized.

[0137] In the embodiments of the present application, the verification and correction based on the preset consistency adjudication rules adopt a "three-level adjudication" mechanism, which accurately identifies and solves semantic conflicts through multi-level screening and verification, ensures the reliability of the semantic mapping result, and the specific process is as follows:

[0138] 1. First-level adjudication: preliminary screening based on semantic similarity score.

[0139] Two key thresholds (the first threshold > the second threshold) are set to preliminarily determine the semantic similarity score of the initial semantic mapping result:

[0140] If the similarity score is higher than the first threshold: it indicates that the data side and the graph side are highly consistent in semantics, and there is no obvious conflict, so it is directly determined as "semantic mapping success", and the result is output;

[0141] If the similarity score is below the second threshold: it indicates that the semantic difference between the data side and the graph side is significant, and the conflict risk is extremely high. Directly determine as "semantic mapping failure" and reject the mapping;

[0142] If the similarity score is between the first threshold and the second threshold: it indicates that there is uncertainty in the semantic association (such as partial feature matching but insufficient overall consistency), and triggers a secondary arbitration for further verification.

[0143] 2. Secondary arbitration: comprehensive determination combined with association consistency.

[0144] For the "to be verified" entities in the primary arbitration, introduce the association data in the knowledge graph for deep verification:

[0145] Association consistency score calculation: query the hierarchical dynamic knowledge graph, extract the association of entities in each data source (such as "transaction association" of entity A and entity B in data source 1, "browsing association" in data source 2), and calculate the "association consistency score" by comparing the coincidence degree and logical consistency of multi-source association;

[0146] Comprehensive arbitration score generation: the "semantic similarity score" in the primary arbitration and the "association consistency score" in the secondary arbitration are weighted and summed according to the preset weight (such as 50% each) to obtain the comprehensive arbitration score;

[0147] If the comprehensive arbitration score is higher than the third threshold: it indicates that after combining the association, the reliability of the semantic association meets the standard, and it is determined as "semantic mapping success";

[0148] If the comprehensive arbitration score is lower than the third threshold: it indicates that there is still a significant conflict risk, triggering a tertiary arbitration.

[0149] 3. Tertiary arbitration: final determination based on logical rules and quality labels.

[0150] For the entities still in doubt in the secondary arbitration, introduce logical rules and data quality information for final verification:

[0151] Logical conflict detection: call the preset logical rule library (such as common sense rules and business constraint rules in the field), and perform logical compliance detection on entity association; if obvious logical conflict is detected (such as "entity A and entity B association violates the 'one account corresponds to one user' rule"), directly determine as "semantic mapping failure";

[0152] Quality label correction and final determination: if no logical conflict is detected, the comprehensive determination score is corrected in combination with the dynamic quality label of the entity (such as the health score, the correlation potential score); according to the final score after correction, if it is higher than the third threshold, it is determined that the semantic mapping is successful, otherwise it is determined that the semantic mapping fails.

[0153] Through the layer-by-layer screening and verification of the three-level determination, high-credibility semantic mapping can be quickly confirmed, and conflicting relationships can be accurately identified and excluded, and finally reliable semantic mapping results are output, thereby providing core support for the accurate fusion of massive data.

[0154] In some embodiments, the AI correlation fusion model is also provided with a dynamic optimization and scheduling layer, and the method further comprises:

[0155] Through the dynamic optimization and scheduling layer, the running state and output result of the correlation fusion and semantic mapping layer are monitored in real time to collect performance data;

[0156] Based on the preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly;

[0157] The optimization instructions are issued to the correlation fusion and semantic mapping layer to dynamically adjust and optimize the calculation tasks, model parameters and fusion rules therein.

[0158] In some embodiments, based on the preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly, including:

[0159] Based on the performance data, the key bottlenecks affecting performance are identified through a preset optimization algorithm, and optimization instructions are generated; wherein the optimization instructions include: adjusting the network weight of the double-tower sub-model, updating the threshold parameter in the consistency determination rule, or performing entity or relationship addition, deletion or modification operation on the layered dynamic knowledge graph.

[0160] In the embodiments of the present application, the dynamic optimization and scheduling layer serves as the "intelligent control center" of the AI correlation fusion model, and through real-time monitoring, intelligent analysis and dynamic adjustment, it ensures that the model always maintains efficient and stable operation in the process of processing massive data, and the core process includes three key links:

[0161] 1. Real-time monitoring and performance data collection.

[0162] The dynamic optimization and scheduling layer continuously tracks the whole-link running state of the correlation fusion and semantic mapping layer, and focuses on collecting two types of core data:

[0163] Running state data: including the processing time of each module (such as the feature encoding time of the double-tower sub-model, the consistency arbitration time), resource occupancy rate (such as CPU / GPU utilization, memory occupancy), task queue length (such as the number of associated sets to be processed), etc., reflecting the real-time running load of the system;

[0164] Output result data: including the association accuracy of the fusion result, the conflict resolution success rate, the semantic mapping precision, etc., reflecting the core performance indicators of the model.

[0165] Through real-time collection and summary of these data, comprehensive basis is provided for subsequent analysis and optimization.

[0166] 2. Performance analysis and optimization instruction generation.

[0167] Based on the preset performance indicators (such as association accuracy threshold, processing delay upper limit) and resource constraints (such as maximum computing power allocation, memory limit), multi-dimensional analysis is performed on the collected performance data:

[0168] If it is found that the processing time of a module exceeds the threshold (such as the long feature encoding time of the double-tower sub-model), it is determined as a "performance bottleneck";

[0169] If it is found that the resource utilization is too low (such as high GPU idle rate) or part of the rule adaptability is insufficient (such as the consistency arbitration threshold leading to the increase of misjudgment rate), it is identified as an "optimization opportunity".

[0170] Combined with the preset optimization algorithm (such as resource scheduling algorithm based on reinforcement learning, parameter tuning algorithm based on statistical analysis), precise optimization instructions are generated for bottlenecks or opportunities, and the objects and specific methods that need to be adjusted are clearly defined.

[0171] 3. Dynamic adjustment and full-link optimization.

[0172] The generated optimization instructions are issued to the association fusion and semantic mapping layer to trigger targeted adjustments:

[0173] Scheduling computing tasks (such as assigning high-priority association aggregation tasks to idle computing power nodes);

[0174] Correcting model parameters (such as adjusting the feature weight of the double-tower sub-model);

[0175] Updating fusion rules (such as optimizing the threshold of consistency arbitration).

[0176] Through real-time feedback and adjustment, the model adapts to data distribution changes, resource fluctuations and scene requirements, and always maintains the optimal running state.

[0177] In the embodiment of the application, based on the collected performance data, the core factors affecting the system performance are located through deep mining by a preset optimization algorithm (such as decision tree analysis, regression analysis): if the semantic mapping accuracy decreases, it may be that the feature encoding capability of the double-tower sub-model is insufficient; if the conflict resolution efficiency is low, it may be that the threshold setting of the consistency adjudication rule is unreasonable; if the association aggregation time-consuming is too long, it may be that there is redundancy in the entity / relationship storage structure of the hierarchical dynamic knowledge graph.

[0178] In the embodiment of the application, three types of core optimization instructions are generated for the identified bottlenecks:

[0179] Model parameter adjustment: such as adjusting the network weight of the double-tower sub-model (enhancing the capture ability of key semantic features), optimizing the activation function parameter (improving the distinguishing degree of feature encoding), so as to improve the accuracy of semantic similarity calculation;

[0180] Rule parameter update: such as modifying the first / second / third threshold in the consistency adjudication rule (relaxing or tightening the judgment standard according to the scene requirement), adjusting the feature weight of the association strength calculation (adapting to the association emphasis of different scenes), so as to improve the accuracy of conflict resolution;

[0181] Knowledge graph maintenance: performing entity or relationship addition, deletion and modification operations (such as deleting redundant associations, supplementing high-frequency use of core relationships, and correcting incorrect semantic associations) on the hierarchical dynamic knowledge graph, so as to reduce the association query time-consuming and improve the support efficiency of the knowledge graph.

[0182] Through the above process, the dynamic optimization and scheduling layer realizes the transformation of the model from "passive running" to "active evolution", ensuring that the AI association fusion model always maintains efficient, accurate and stable fusion capability when facing massive heterogeneous data and complex scenes, and providing continuous and reliable technical support for data value mining.

[0183] The embodiment of the application provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor implements the method shown in the above embodiment when executing the computer program.

[0184] The embodiment of the application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method shown in the above embodiment.

[0185] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments. If there is no special description and no logical conflict, the terms and / or descriptions of different embodiments are consistent and can be mutually referenced, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0186] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for correlation and fusion of massive data, characterized in that, include: Receive massive amounts of data from multiple heterogeneous data sources and preprocess the data; The massive amount of data includes structured data sources, unstructured data sources, and third-party data sources; The structured data source includes user tables and order tables in a relational database; The unstructured data sources include JSON format logs and text log files returned by the API interface; the third-party data sources include Excel data tables and CSV files provided by partner institutions. The preprocessed data is integrated into a labeled data stream; The annotated data stream includes a two-dimensional semantic vector and dynamic quality labels; The labeled data stream is input into the AI ​​association fusion model to obtain the fusion result; The AI ​​association fusion model includes an association fusion and semantic mapping layer. The association fusion and semantic mapping layer is used to perform association fusion based on a hierarchical dynamic knowledge graph and preset fusion rules, and to perform semantic mapping based on a dual-tower sub-model and consistency adjudication rules. The annotated data stream is input into the AI ​​association fusion model to obtain the fusion result, including: The two-dimensional semantic vectors in the annotated data stream are normalized to match the preset input dimension of the model; at the same time, the dynamic quality labels are weighted and initialized based on the credibility level of the data source to obtain the adapted data stream. The preset fusion rules and the adapted data stream are transmitted to the association fusion and semantic mapping layer to obtain a preliminary association set and an initial semantic mapping result, so as to determine the fusion result; The association fusion and semantic mapping layer is based on the hierarchical structure of the hierarchical dynamic knowledge graph. It extracts the association features corresponding to the adapted data stream from each level, and combines them with preset fusion rules to perform preliminary association aggregation on the source data and related data in the adapted data stream to generate a preliminary association set. The association fusion and semantic mapping layer calls a dual-tower sub-model to perform semantic mapping processing on the initial association set; wherein the first sub-tower is used to receive the two-dimensional semantic vectors in the initial association set and extract local semantic features through a convolutional neural network, and the second sub-tower is used to receive the semantic representation vectors of the corresponding association nodes in the hierarchical dynamic knowledge graph and extract global semantic features through a Transformer encoder; the initial semantic mapping result is determined based on the cosine similarity of the output features of the two sub-towers.

2. The method for correlation and fusion of massive data according to claim 1, characterized in that, The hierarchical dynamic knowledge graph comprises three layers: a basic data layer, a feature association layer, and a semantic fusion layer, with each layer dynamically connected via association edges. The basic data layer stores the original entity and attribute information from each data source; the feature association layer stores feature-level associations between entities; and the semantic fusion layer stores semantic-level associations between entities and domain knowledge rules. The adapted data stream is transmitted to the association fusion and semantic mapping layer to obtain a preliminary association set and initial semantic mapping results, including: The baseline entity information corresponding to each entity in the adapted data stream is extracted from the basic data layer. The entity recognition algorithm is used to identify the entity type in the adapted data stream. The entity to be merged is matched with the baseline entity through the entity alignment algorithm, and the same-source entities and suspected related entities are marked. Based on the preset fusion rules and the feature association relationships stored in the feature association layer, feature-level association aggregation is performed on the marked homogeneous entities and suspected related entities to obtain a preliminary association set; The domain knowledge rules in the semantic fusion layer are invoked to semantically enhance the preliminary association set output by the feature association layer, resulting in the initial semantic mapping result.

3. The method for correlation and fusion of massive data according to claim 2, characterized in that, The preset fusion rules include a basic rule set and scenario adaptation parameters; wherein, the basic rule set includes data priority rules, attribute fusion rules, and association strength calculation rules; the scenario adaptation parameters include rule trigger thresholds, feature weight coefficients, and scenario type identifiers, used to dynamically adjust the execution logic of the basic rule set according to specific application scenarios; based on the preset fusion rules and the feature association relationships stored in the feature association layer, feature-level association aggregation is performed on the marked homologous entities and suspected related entities to obtain a preliminary association set, including: For the marked entities with the same origin, the corresponding attribute fusion rules in the basic rule set are called according to their attribute types to perform fusion processing on the multi-source attribute data and generate a unified entity attribute description. For the identified suspected related entities, the association strength between the entities is calculated based on the association strength calculation rules in the basic rule set and the association relationships stored in the feature association layer. Based on the association threshold set in the scenario adaptation parameters, suspected associated entities whose association strength meets the threshold condition are determined to be associated, and their feature information is aggregated to form the preliminary association set.

4. The method for correlation and fusion of massive data according to claim 1, characterized in that, The dual-dimensional semantic vector includes data-driven semantic vectors and graph-driven semantic vectors; The annotated data stream is input into the AI ​​association fusion model to obtain the fusion result, including: The data-driven semantic vectors of entities are extracted from the initial association set as the input of the first sub-tower, and the graph-driven semantic vectors corresponding to the entities are extracted from the hierarchical dynamic knowledge graph as the input of the second sub-tower. The first and second sub-towers are used to encode the features of their respective input vectors, and the corresponding semantic feature vectors are output. The semantic feature vectors output by the first sub-tower and the semantic feature vectors output by the second sub-tower are used to calculate the similarity and generate the initial semantic mapping result. Based on preset consistency adjudication rules, the initial semantic mapping results are verified and corrected in order to identify and resolve semantic conflicts; The final semantic mapping output is generated based on the adjudication results, and the dual-tower sub-model and / or consistency adjudication rules are iteratively optimized using conflict cases and / or manual correction results generated during the adjudication process.

5. The method for correlation and fusion of massive data according to claim 4, characterized in that, Based on preset consistency determination rules, the initial semantic mapping results are verified and corrected to identify and resolve semantic conflicts, including: The semantic similarity score is compared with a first threshold. If it is higher than the first threshold, the semantic mapping is directly determined to be successful and the result is output. If it is lower than the second threshold, the semantic mapping is directly determined to be unsuccessful and the mapping is rejected. If it is between the first threshold and the second threshold, a second-level decision is triggered. The first threshold is higher than the second threshold. For entities that trigger a Level 2 ruling, the hierarchical dynamic knowledge graph is queried to obtain their association relationships in their respective data sources, and the consistency score of the association relationship is calculated; the semantic similarity score and the consistency score are weighted and summed to obtain a comprehensive ruling score; if the comprehensive ruling score is higher than a third threshold, the semantic mapping is determined to be successful, otherwise a Level 3 ruling is triggered. For entities that trigger a Level 3 ruling, conflict detection is performed using a pre-defined logical rule base. If a logical conflict is detected, the semantic mapping is deemed to have failed. If no logical conflict is detected, the comprehensive ruling score is corrected based on the dynamic quality label, and the final score after correction is used to determine whether the mapping was successful.

6. The method for correlation and fusion of massive data according to claim 1, characterized in that, The AI ​​correlation fusion model also includes a dynamic optimization and scheduling layer, and the method further includes: The dynamic optimization and scheduling layer monitors the running status and output results of the association fusion and semantic mapping layer in real time to collect performance data. Based on preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly. The optimization instructions are sent to the association fusion and semantic mapping layer to dynamically adjust and optimize the computational tasks, model parameters and fusion rules therein.

7. The method for correlation and fusion of massive data according to claim 6, characterized in that, Based on preset performance indicators and resource constraints, the performance data is analyzed to identify performance bottlenecks or optimization opportunities, and optimization instructions are generated accordingly, including: Based on the performance data, a preset optimization algorithm is used to identify key bottlenecks affecting performance and generate optimization instructions. The optimization instructions include: adjusting the network weights of the dual-tower sub-model, updating the threshold parameters in the consistency adjudication rules, or performing addition, deletion, or modification operations on entities or relationships in the hierarchical dynamic knowledge graph.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images