Multi-system heterogeneous data automatic fusion docking system and method

Abstract

The application relates to the technical field of data processing, and provides a multi-system heterogeneous data automatic fusion docking system and method.The application sends sniffing packets to multiple data source systems, identifies protocol types based on information entropy distribution of data frames, extracts field semantic features of protocol data streams and embeds uniform vector spaces, calculates cross-system semantic similarity in combination with system field types, generates a compensation plan in combination with data source quality by detecting semantic conflicts and conflict probabilities in a mapping process, determines a reliability weight in combination with historical consistency of the data source, verifies whether type identifiers of the semantic conflicts and the compensation plan are consistent, performs weighting fusion after compensation, directly performs weighting fusion and updates a predicted conflict probability when there is no matching, calculates a deviation of a quality index of a fusion result from a preset business constraint, converts the deviation into an adjustment amount of semantic similarity calculation, feeds back an adjusted similarity threshold value and a conflict prediction time window, and improves the efficiency of multi-system heterogeneous data fusion docking.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a system and method for automated fusion and docking of heterogeneous data from multiple systems. Background Technology

[0002] The application scenarios of heterogeneous data from multiple systems are becoming increasingly widespread. There are a large number of data source systems with different architectures, protocols and data formats in fields such as industrial control and government services. Achieving automated integration and docking of heterogeneous data from multiple systems has become a core requirement for breaking down data silos and enhancing the value of data utilization.

[0003] Currently, the existing multi-system heterogeneous data fusion and docking mainly adopts protocol conversion, field mapping and simple fusion. By identifying the data source protocol type through preset protocol templates, cross-system field mapping relationships are configured manually or semi-automatically, and the mapped fields are weighted and fused. Some systems have added simple conflict detection mechanisms to avoid obvious data conflicts, providing technical support for multi-system data collaboration.

[0004] Existing technologies mostly rely on preset protocol templates to identify protocols, which cannot adapt to unknown private protocols or non-standard protocols and cannot meet the docking needs of multiple types of heterogeneous data sources. Conflict detection can only identify obvious field conflicts and cannot accurately predict the probability of conflicts. Furthermore, the generation of compensation plans does not take into account the differences in data source quality, resulting in poor conflict handling and failing to avoid fusion deviations at the root.

[0005] Based on the shortcomings of the existing technology, the technical problem to be solved by this application is how to realize the automated fusion and docking of heterogeneous data from multiple systems, and improve the fusion efficiency and fusion quality. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application provides a method and system for automated fusion and docking of heterogeneous data from multiple systems. This system achieves full-process automation from protocol identification, semantic mapping, conflict detection, compensation execution, data fusion to parameter feedback adjustment, significantly reducing manual operation costs, improving the efficiency of fusion and docking of heterogeneous data from multiple systems, and adapting to the automated fusion needs of large-scale heterogeneous data sources.

[0007] In the first aspect, this application provides a method for automated fusion and docking of heterogeneous data from multiple systems. The method includes: sending sniffing packets to multiple data source systems, identifying the protocol types of different data sources based on the information entropy distribution of data frames, and outputting protocol data streams.

[0008] Extract the semantic features of the fields in the protocol data stream and embed them into a unified vector space. Calculate the cross-system semantic similarity based on the system domain type to generate a mapping relationship and detect semantic conflicts in the mapping process. When the predicted conflict probability is greater than the probability threshold, generate a compensation plan based on the data source quality.

[0009] Align the data source fields according to the mapping relationship, determine the confidence weight according to the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, execute the corresponding compensation plan and then perform weighted fusion to obtain the fusion result; otherwise, directly perform weighted fusion and update the predicted conflict probability.

[0010] The deviation between the quality index of the fusion result and the preset business constraints is calculated, and the deviation is converted into the adjustment amount for semantic similarity calculation. The similarity threshold and conflict prediction time window are then adjusted accordingly.

[0011] As an optional implementation, the output protocol data stream includes:

[0012] The sniffing packets are sent to multiple data source systems and the returned data frames are obtained. After the data frames are fragmented and reassembled, they are divided into feature segments according to the frame header identifier, data segment length and check bit distribution. The information entropy of different feature segments is calculated to determine the information entropy distribution.

[0013] The information entropy distribution is matched with a preset protocol entropy feature library, and the protocol type is identified by dynamic threshold clustering. Feature segments whose information entropy deviates from the standard entropy range are marked as extended feature segments.

[0014] The extended feature segments are organized according to the protocol type and concatenated with the standard feature segments corresponding to the protocol type. The concatenated protocol data is then sorted by field normalization according to the protocol semantic level, and the protocol data stream is output.

[0015] As an optional implementation, the generation of the mapping relationship includes:

[0016] Extract the field semantic features of the standard feature segments and extended feature segments in the protocol data stream, and embed them into a unified vector space to obtain the standard semantic vector and extended semantic vector;

[0017] Based on the system domain type, a cross-system domain weight matrix is ​​constructed. The standard semantic vector and the extended semantic vector are weighted by the domain weight matrix to obtain the semantic similarity between cross-system fields.

[0018] Semantic similarity is clustered and matched, and valid results are filtered by combining similarity thresholds to generate initial mapping relationships. The initial mapping relationships are then corrected by combining the confidence of protocol type to obtain the final mapping relationship.

[0019] As an optional implementation, detecting the semantic conflict includes:

[0020] Semantic deviation features and protocol format adaptation features are extracted from standard semantic vectors and extended semantic vectors. Conflict features are obtained by weighted fusion of semantic deviation features and protocol format adaptation features based on domain weight matrix and protocol type confidence.

[0021] The conflict characteristics of multiple consecutive mapping relationships are statistically analyzed using a conflict prediction time window. The dispersion of the conflict characteristics within the conflict prediction time window is calculated, and the ratio of the dispersion to a preset benchmark value is used as the conflict probability.

[0022] As an optional implementation, the generation compensation plan includes:

[0023] When the probability of conflict exceeds the probability threshold, the conflict type is classified according to the dimensional ratio of semantic deviation features and protocol format adaptation features, and the conflict level is labeled by associating the discreteness of the corresponding conflict features.

[0024] The quality score of the data source is determined based on data integrity, temporal stability and field consistency. The quality score is then weighted by a domain weight matrix to obtain the compensation priority for different data sources.

[0025] Configure a compensation plan that includes type identifier, compensation priority, and execution time, based on the conflict type, conflict level, and compensation priority.

[0026] As an optional implementation, the obtained fusion result includes:

[0027] Align the corresponding fields of standard feature segments and extended feature segments between different data sources according to the mapping relationship, and determine the credibility weight of the fields based on the historical consistency of the data sources.

[0028] If the semantic conflict matches the type identifier in the compensation plan, and the execution time is within the conflict prediction time window, then the corresponding compensation operation will be executed according to the compensation priority.

[0029] The compensated standard feature segments and extended feature segments are weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result.

[0030] As an optional implementation, the updated prediction of conflict probabilities includes:

[0031] If the semantic conflict does not match the type identifier in the compensation plan, or the execution time exceeds the conflict prediction time window, the standard feature segments and extended feature segments between different data sources are directly weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result.

[0032] The actual deviation values ​​of the fields during this weighted fusion process are extracted, and the predicted conflict probability is updated by combining the dispersion of conflict features within the conflict prediction time window.

[0033] As an optional implementation, the feedback adjustment similarity threshold includes:

[0034] The actual deviation value of the field is used as the quality index of the fusion result. The deviation between the quality index and the preset business constraints is calculated, and the deviation is converted into the adjustment amount for semantic similarity calculation.

[0035] The adjustment amount is corrected based on the dispersion of conflict features within the conflict prediction time window, and the similarity threshold is adjusted based on the corrected adjustment amount.

[0036] As an optional implementation, adjusting the conflict prediction time window includes:

[0037] The cumulative amount of actual deviation values ​​of fields within the current conflict prediction time window is statistically analyzed, and the window deviation adaptation coefficient is calculated in combination with the dispersion of conflict characteristics.

[0038] The window bias adaptation coefficient is weighted and corrected based on the domain weight matrix to distinguish the conflict response differences between the standard feature segment and the extended feature segment, and to generate the scaling ratio of the corresponding conflict prediction time window.

[0039] The duration of the conflict prediction time window is dynamically adjusted according to the scaling ratio.

[0040] Secondly, this application provides an automated fusion and docking system for heterogeneous data from multiple systems, which includes: a sensing module, a mapping module, a fusion module, and a feedback module;

[0041] The perception module is used to send sniffing packets to multiple data source systems, identify the protocol type of different data sources based on the information entropy distribution of data frames, and output the protocol data stream.

[0042] The mapping module is used to extract the semantic features of the fields of the protocol data stream and embed them into a unified vector space. It calculates the semantic similarity across systems based on the system domain type to generate mapping relationships and detects semantic conflicts in the mapping process. When the predicted conflict probability is greater than the probability threshold, a compensation plan is generated based on the data source quality.

[0043] The fusion module is used to align data source fields according to the mapping relationship, determine the confidence weight based on the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, the corresponding compensation plan is executed and then weighted fusion is performed to obtain the fusion result; otherwise, the weighted fusion is performed directly and the predicted conflict probability is updated.

[0044] The feedback module is used to calculate the deviation between the quality index of the fusion result and the preset business constraints, convert the deviation into the adjustment amount for semantic similarity calculation, and feed back the adjustment similarity threshold and conflict prediction time window.

[0045] Compared with existing technologies, the beneficial effects of this application are as follows: By sending sniffing packets to multiple data source systems, the protocol type is identified based on the information entropy distribution of data frames, effectively identifying standard protocols, private protocols, and non-standard protocols, adapting to the docking needs of heterogeneous data sources of multiple domains and types, and providing a data foundation for subsequent mapping and fusion processes; by extracting the semantic features of the fields of the protocol data stream and embedding them into a unified vector space, and combining the system domain type to calculate cross-system semantic similarity, the accuracy of the mapping relationship generation is ensured; at the same time, by detecting semantic conflicts and conflict probabilities in the mapping process, and combining the data source quality to generate a compensation plan, the differences in data source quality are distinguished, and the fusion deviation is reduced from the root.

[0046] Alignment of data source fields is achieved based on mapping relationships. Reliability weights are determined by combining the historical consistency of the data source. The semantic conflict is checked to see if it matches the type identifier of the compensation plan. If they match, effective compensation is performed before weighted fusion, avoiding resource waste caused by invalid compensation. If there is no match, weighted fusion is performed directly and the predicted conflict probability is updated, balancing fusion efficiency and fusion quality.

[0047] By calculating the deviation between the quality indicators of the fusion results and the preset business constraints, the deviation is converted into an adjustment amount for semantic similarity calculation. The similarity threshold and conflict prediction time window are dynamically adjusted based on the dynamic changes in data conflicts and the fusion quality. This allows the system to optimize the front-end parameters in real time. This application achieves full-process automation from protocol identification, semantic mapping, conflict detection, compensation execution, data fusion to parameter feedback adjustment, which greatly reduces the cost of manual operation, improves the efficiency of multi-system heterogeneous data fusion and docking, and adapts to the automated fusion needs of large-scale heterogeneous data sources. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0049] Figure 1 This is a flowchart illustrating the automated fusion and docking method for heterogeneous data from multiple systems provided in this application embodiment.

[0050] Figure 2 This is a flowchart illustrating the logic of detecting semantic conflicts provided in an embodiment of this application.

[0051] Figure 3 This is a system architecture diagram of the automated fusion and docking system for heterogeneous data from multiple systems provided in this application embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0053] Example 1:

[0054] like Figure 1 The diagram shows a flowchart of a method for automated fusion and docking of heterogeneous data from multiple systems, as provided in this application embodiment. The method includes:

[0055] S1. Send sniffing packets to multiple data source systems, identify the protocol type of different data sources based on the information entropy distribution of data frames, and output the protocol data stream.

[0056] Specifically, the output protocol data stream includes:

[0057] The sniffing packets are sent to multiple data source systems and the returned data frames are obtained. After the data frames are fragmented and reassembled, they are divided into feature segments according to the frame header identifier, data segment length and check bit distribution. The information entropy of different feature segments is calculated to determine the information entropy distribution.

[0058] The information entropy distribution is matched with a preset protocol entropy feature library, and the protocol type is identified by dynamic threshold clustering. Feature segments whose information entropy deviates from the standard entropy range are marked as extended feature segments.

[0059] The extended feature segments are organized according to the protocol type and concatenated with the standard feature segments corresponding to the protocol type. The concatenated protocol data is then sorted by field normalization according to the protocol semantic level, and the protocol data stream is output.

[0060] The heterogeneous data sources of multiple systems have different communication protocols. Some data sources use non-public private protocols to encapsulate data frames, which cannot be parsed by a preset fixed protocol template. Specifically, a preset format sniffing and probe message is sent to each data source system to be connected. The message adopts a lightweight frame structure to reduce the response load of the data source. After receiving the data frame returned by the data source, the fragmented frame data is reassembled according to the fragmentation identifier and offset parameter of the network transmission protocol to restore the data frame structure.

[0061] Here, the sniffing and detection messages are fixed-length binary short frames with a total length of 8 bytes. They include, in sequence, a frame start identifier, a device addressing bit, a detection function bit, a fragmentation reserved bit, a check bit, and a frame end identifier. Each identifier bit uses a fixed value. The device addressing bit can distinguish the domain of the data source and the device number to achieve targeted detection of multiple heterogeneous data sources. Typical messages can be presented intuitively in hexadecimal form.

[0062] After reassembly, based on the frame structure characteristics of the data frame, the data frame is divided into a frame header identification feature segment, a data segment length feature segment, and a check bit distribution feature segment. The frame header identification feature segment is a fixed identifier field area at the start of the data frame, the data segment length feature segment is a numerical field area marking the length of the valid data payload, and the check bit distribution feature segment is a field area carrying data verification information. For each feature segment, the information entropy value of the binary bit stream within each feature segment is calculated based on Shannon information entropy. The entropy values ​​of each feature segment are combined to form the information entropy distribution of the data frame.

[0063] Taking Modbus protocol data sources and private custom protocol data sources in the industrial IoT scenario as examples, the frame header identifier feature segment is a fixed byte sequence with strong data regularity, and the information entropy value is usually in the range of 0.21 to 0.38; the data segment length feature segment is a variable numerical field with medium data randomness, and the information entropy value ranges from 0.52 to 0.67; the check bit distribution feature segment changes dynamically with the data payload, with high randomness, and the information entropy value can reach the range of 0.83 to 0.95. By calculating the entropy values ​​of the three types of feature segments respectively, a unique information entropy distribution that characterizes the structural features of the data frame can be formed.

[0064] This enables adaptation to data sources with unknown protocol types, eliminating the need to pre-configure protocol parsing parameters for the data source. It achieves quantitative representation of data frame structure based on information entropy, eliminating subjective interference from manual parsing and improving the objectivity of protocol feature extraction.

[0065] The frame structure feature segments of different standard communication protocols have stable information entropy distribution patterns. Heterogeneous data sources have vendor-defined non-standard extended fields, and the information entropy values ​​of these fields deviate from the entropy value range of standard communication protocols. Specifically, the information entropy distribution of the current data frame is compared field by field with the standard communication protocol entropy feature templates stored in the preset protocol entropy feature library. The protocol entropy feature library contains the standard entropy ranges of mainstream industrial and network protocols such as Modbus, MQTT, and TCP private encapsulation. After the comparison is completed, the clustering judgment threshold is dynamically adjusted based on the number of currently accessed data sources and the degree of dispersion of entropy distribution. Data frames whose entropy distribution feature similarity meets the threshold requirements are classified into the same protocol category, thus completing the identification of the protocol type.

[0066] After determining the protocol type, the standard entropy range of each feature segment corresponding to that protocol is retrieved. The information entropy of each feature segment is compared with the standard entropy range. Feature segments whose information entropy exceeds the upper or lower limit of the standard entropy range are determined to be non-standard protocol fields and marked as extended feature segments. Feature segments whose information entropy is within the standard entropy range are identified as standard feature segments. The standard entropy range for the frame header identifier feature segment of the standard MQTT protocol is set to 0.2 to 0.4. The actual calculated entropy value of this feature segment of a certain data source is 0.76, which exceeds the upper limit of the standard entropy range, so this feature segment is marked as an extended feature segment. The actual entropy value of the data segment length feature segment of this data source is 0.61, which is within the standard entropy range of 0.5 to 0.7 of the standard MQTT protocol, so it is classified as a standard feature segment.

[0067] This enables automatic identification of multiple heterogeneous data source protocol types, eliminating the need to configure independent parsing rules for a single data source. It achieves accurate marking of extended feature segments through entropy deviation judgment, effectively distinguishing standard protocol fields from private extended fields, and laying a structural foundation for subsequent data regularization.

[0068] Extended feature segments are vendor-defined fields, which suffer from issues such as inconsistent byte order, redundant padding bits, and chaotic field formats. They cannot be directly used in subsequent semantic feature extraction and mapping operations. Furthermore, the original arrangement order of standard feature segments and extended feature segments lacks a unified rule, which can easily lead to semantic ambiguity in the data. Specifically, based on the identified protocol type, the marked extended feature segments undergo regularization processing. Regularization operations include unifying the byte order format, removing invalid padding bytes, and filling in missing field bits, so that the structure of the extended feature segments conforms to the system's unified parsing standard. After regularization, the extended feature segments and the standard feature segments of the corresponding protocol type are concatenated in a fixed logical order to form the protocol data structure.

[0069] After concatenation, the fields of the protocol data are normalized and sorted according to the inherent semantic hierarchy of the protocol. The sorting rules follow the hierarchical logic of identifier fields, length fields, data payload fields and verification fields to eliminate the differences in the arrangement order of fields from different data sources. After sorting, the standardized structured data is output as a protocol data stream for subsequent processing.

[0070] For example, for an extended feature segment of a custom TCP protocol, the original data is stored in little-endian order and contains 4 invalid padding bytes. During the normalization process, the byte order is converted to the system's unified big-endian order, the 4 invalid padding bytes are removed, and 1 missing flag bit is added. The normalized extended feature segment and the standard feature segment are concatenated in the order of standard feature segment first and extended feature segment last. Then, the fields are sorted according to the protocol semantic level, and finally a protocol data stream with a unified structure is formed.

[0071] This eliminates format differences in extended fields from heterogeneous data sources, enables standardized processing of private extended fields, and avoids semantic ambiguity caused by field arrangement order through semantic hierarchical normalization sorting, thereby improving the uniformity of data structure. The protocol data stream has a standardized structure and a unified semantic hierarchy, providing a stable and standardized data carrier for subsequent extraction of field semantic features of standard feature segments and extended feature segments and embedding into a unified vector space, ensuring the stability of semantic vector generation and mapping relationship construction.

[0072] S2. Extract the semantic features of the fields in the protocol data stream and embed them into a unified vector space. Calculate the cross-system semantic similarity based on the system domain type to generate a mapping relationship. Detect semantic conflicts during the mapping process. When the predicted conflict probability is greater than the probability threshold, generate a compensation plan based on the data source quality.

[0073] Furthermore, generating the mapping relationship includes:

[0074] Extract the field semantic features of the standard feature segments and extended feature segments in the protocol data stream, and embed them into a unified vector space to obtain the standard semantic vector and extended semantic vector;

[0075] Based on the system domain type, a cross-system domain weight matrix is ​​constructed. The standard semantic vector and the extended semantic vector are weighted by the domain weight matrix to obtain the semantic similarity between cross-system fields.

[0076] Semantic similarity is clustered and matched, and valid results are filtered by combining similarity thresholds to generate initial mapping relationships. The initial mapping relationships are then corrected by combining the confidence of protocol type to obtain the final mapping relationship.

[0077] The semantics of the fields in the standard feature segment and the extended feature segment in the protocol data stream are heterogeneous. The same type of field in different data source systems may be expressed in different ways. Although their semantic connotations are related, they cannot be directly compared across systems. At the same time, the semantic levels of the standard feature segment and the extended feature segment are different. If a unified quantification process is not performed, the subsequent semantic similarity calculation will not be able to be implemented accurately. Specifically, the output protocol data stream is first split into fields to separate the standard feature segment and the extended feature segment. For the fields of the two types of feature segments, a combination of semantic word segmentation and feature encoding is used to extract the semantic features of the fields. Semantic word segmentation is used to split the semantic units of the fields, and feature encoding is used to convert the semantic units into quantifiable feature factors to eliminate the semantic ambiguity caused by different expression methods.

[0078] Then, a unified vector space is constructed. This unified vector space is based on the semantic dimension preset by the system and covers all possible field semantic types. The extracted standard feature segment field semantic features and extended feature segment field semantic features are mapped to this unified vector space through a vector embedding algorithm to obtain the standard semantic vector and extended semantic vector corresponding to each field. This enables the field semantics of different data sources and different types of feature segments to have a unified quantitative representation form.

[0079] For example, the protocol data stream contains the standard feature segment "Equipment Operating Temperature" from industrial data sources and the extended feature segment "Environmental Monitoring Temperature" from government data sources. The semantic features of the equipment operating temperature are extracted as equipment, operating status, and temperature parameters, and the semantic features of the environmental monitoring temperature are extracted as environment, monitoring, and temperature parameters. The two types of semantic features are embedded into a unified vector space to obtain the standard semantic vector (0.12, 0.35, 0.87) and the extended semantic vector (0.21, 0.42, 0.87), where each dimension of the vector corresponds to the semantic weight of equipment / environment, operation / monitoring, and temperature parameters, respectively, thus achieving unified quantification of the semantics of the two types of fields.

[0080] This eliminates semantic ambiguity caused by differences in field representation. By embedding a unified vector space, it achieves standardized quantification of the semantics of feature segments from different data sources and of different types, providing a unified basis for subsequent cross-system semantic similarity calculations and avoiding comparison errors caused by inconsistent semantic representations.

[0081] The importance of field semantics varies among data sources in different system domains, resulting in different weightings during cross-system integration. Using a uniform weighting to calculate semantic similarity would lead to results that deviate from actual business needs and fail to accurately reflect the degree of semantic association between fields across systems. System domain type refers to the industry or business domain to which each data source system belongs, such as industrial control, government data, and medical data. Field semantic priorities differ across domains, and the domain weight matrix is ​​used to quantify the weighting of field semantics across different domains, achieving domain adaptation for semantic similarity calculation.

[0082] Specifically, based on the domain type of each data source system, a cross-system domain weight matrix is ​​constructed. The rows and columns of this matrix correspond to the semantic dimensions of fields in different data source systems. The matrix elements are the weight coefficients of the semantics of the corresponding domain fields. The weight coefficients are determined based on the business priority of the system domain. The weight coefficients of core business fields are higher than those of non-core business fields, and the weight coefficients of standard feature segment fields are higher than those of extended feature segment fields in the same domain, ensuring that the weight allocation meets the needs of actual business scenarios.

[0083] After construction, the standard semantic vector and the extended semantic vector are weighted by the domain weight matrix respectively. The weighting method is the dot product operation between the vector and the matrix to eliminate the weight difference of the semantics of the fields in different domains and obtain the semantic similarity between cross-system fields. This is used to characterize the degree of semantic association between fields in different data source systems. The higher the similarity, the more consistent the semantics of the fields are, and the more suitable it is to build a mapping relationship.

[0084] For example, consider data source A in the industrial control domain and data source B in the government data domain. In the constructed domain weight matrix, the weight coefficient of the core field in the industrial control domain is 0.8, the weight coefficient of the core field in the government data domain is 0.7, and the weight coefficient of the extended feature segment field is uniformly 0.5. The standard semantic vector (0.12, 0.35, 0.87) of data source A is multiplied by the domain weight matrix to obtain a weighted standard semantic vector, and the extended semantic vector (0.21, 0.42, 0.87) of data source B is multiplied by the domain weight matrix to obtain a weighted extended semantic vector. The cosine similarity between the two weighted vectors is calculated, and the semantic similarity between the cross-system fields is 0.91, indicating that the two fields are semantically closely related.

[0085] This enables the differentiated allocation of semantic weights for fields in different domains, making semantic similarity calculation more aligned with actual business needs, improving the accuracy of cross-system field semantic comparison, avoiding semantic similarity deviations caused by uniform weights, and taking into account the semantic differences between standard feature segments and extended feature segments.

[0086] After semantic similarity calculation, there are a large number of discrete comparison results, some of which have low semantic similarity. If these results are directly used as mapping relationships, it will lead to data fusion errors. At the same time, the confidence level of the protocol type directly affects the reliability of field semantic matching. If the confidence level of the protocol type is ignored, it may lead to a mismatch between the mapping relationship and the protocol semantics. Specifically, we first perform cluster matching on the semantic similarity between cross-system fields. We use density clustering algorithm to group the comparison results of fields with close semantic similarity into one category, realizing the aggregation of semantic fields of the same type and avoiding the random errors of single comparison results. In the clustering process, semantic similarity is used as the core clustering basis to ensure that the clustering results can reflect the actual semantic relationship of the fields.

[0087] Then, based on the system's business requirements and protocol type characteristics, a similarity threshold is determined to filter valid comparison results. Comparison results with semantic similarity greater than the similarity threshold are considered valid and included in the initial mapping relationship, while comparison results with semantic similarity not greater than the similarity threshold are considered invalid and discarded to ensure the validity of the initial mapping relationship. After generating the initial mapping relationship, the identified protocol type and its confidence level are retrieved. The confidence level of the protocol type refers to the reliability of the protocol identification result. The higher the confidence level, the more accurate the protocol type identification and the higher the reliability of the corresponding field semantic matching.

[0088] The initial mapping relationship is corrected by combining the confidence of the protocol type. If the confidence of the protocol type corresponding to an initial mapping relationship is greater than the confidence threshold, the mapping relationship is retained. If the confidence of the protocol type is not greater than the confidence threshold, the semantic similarity corresponding to the mapping relationship is recalculated and corrected again by combining the domain weight matrix. If the semantic similarity after correction does not reach the similarity threshold, the mapping relationship is removed. If it reaches the similarity threshold, the mapping relationship is retained and updated, and the final mapping relationship is obtained.

[0089] In this embodiment, the preset confidence threshold is 0.7, which is an intermediate threshold that balances fault tolerance and accuracy. It filters out abnormal mappings with low protocol recognition confidence, while avoiding the mis-screening of normal mappings due to excessively high thresholds, and adapts to the recognition accuracy distribution characteristics of heterogeneous protocols in multiple fields. The protocol confidence value is 0 to 1, which is used to characterize the reliability of the data frame protocol recognition results.

[0090] For example, with a preset similarity threshold of 0.8, after clustering and matching, 5 sets of field comparison results are obtained, with semantic similarities of 0.91, 0.85, 0.78, 0.93, and 0.72, respectively. Three sets of results with semantic similarity greater than 0.8 are selected as the initial mapping relationship. The confidence of the protocol type corresponding to each set of comparison results is retrieved. The confidence of the protocol type of one set of comparison results is 0.65, which is not greater than the preset confidence threshold of 0.7. The semantic similarity of this set of comparison results is recalculated and corrected. After correction, the semantic similarity is 0.79, which is still not greater than the similarity threshold, so it is removed. Finally, 2 sets of valid mapping relationships are obtained.

[0091] This reduces the random errors of single comparison results, eliminates invalid comparison results, ensures the validity of the initial mapping relationship, improves the accuracy and reliability of the mapping relationship through protocol type confidence correction, avoids the failure of the mapping relationship due to protocol identification errors, and achieves accurate matching between the mapping relationship and protocol semantics and business requirements. The mapping relationship determines the effect of subsequent data fusion, provides a basic guarantee for the generation of compensation plans and the calculation of fusion results, and ensures the orderly progress of the entire multi-system heterogeneous data fusion process.

[0092] Specifically, such as Figure 2 As shown, detecting semantic conflicts includes:

[0093] Semantic deviation features and protocol format adaptation features are extracted from standard semantic vectors and extended semantic vectors. The two types of features are then weighted and fused based on the domain weight matrix and the confidence of the protocol type to obtain conflict features.

[0094] The conflict characteristics of multiple consecutive mapping relationships are statistically analyzed using a conflict prediction time window. The dispersion of the conflict characteristics within the conflict prediction time window is calculated, and the ratio of the dispersion to a preset benchmark value is used as the conflict probability.

[0095] Semantic conflicts in cross-system heterogeneous data are not only reflected in the differences in the semantic connotations of fields, but also in the differences in protocol format compatibility. Simply detecting semantic deviations alone cannot fully cover the conflict scenarios of heterogeneous data. If protocol format compatibility is ignored, conflicts caused by format incompatibility will be missed. At the same time, the degree of impact of semantic conflicts in different domains varies, and the reliability of protocol identification determines the accuracy of conflict detection. Among them, the semantic deviation feature is a quantitative representation of the semantic difference between standard semantic vectors and extended semantic vectors in a unified vector space, used to reflect the degree of deviation of the semantic connotations of the fields corresponding to the two types of vectors. The protocol format compatibility feature is a quantitative representation of the degree of compatibility between the field protocol format corresponding to the standard semantic vectors and extended semantic vectors and the system's preset protocol format, used to reflect the potential conflict caused by incompatibility of field protocol formats. The two types of features together constitute the core dimensions of semantic conflict detection, and neither can be omitted.

[0096] Specifically, the standard semantic vector and the extended semantic vector are retrieved first. A combination of vector difference calculation and semantic dimension comparison is used to extract semantic deviation features. By calculating the difference between the standard semantic vector and the extended semantic vector in each semantic dimension within a unified vector space, and combining this with the semantic weights of each dimension, the semantic deviation features are integrated to obtain the semantic deviation features. These features quantify the degree of semantic deviation between the two types of vectors; the higher the degree of deviation, the greater the likelihood of semantic conflict. Simultaneously, protocol format adaptation features are extracted. Based on the identified protocol type, the corresponding standard format specification is retrieved. The field protocol formats corresponding to the standard semantic vector and the extended semantic vector are compared to their compatibility with the standard format specification. Protocol format adaptation features are obtained through quantified calculation of format adaptation; the lower the compatibility, the greater the potential for conflict caused by the protocol format.

[0097] The domain weight matrix and protocol type confidence scores are retrieved. The domain weight matrix quantifies the weight ratio of semantic deviation features and protocol format adaptation features in different domains. The feature weights of core domains are higher than those of non-core domains, and the feature weights corresponding to standard semantic vectors are higher than those corresponding to extended semantic vectors. The protocol type confidence score is used to correct the reliability of feature weighting. The higher the confidence score, the higher the weight of the corresponding feature, ensuring that the calculation of conflict features is consistent with the reliability of protocol identification. The semantic deviation features and protocol format adaptation features are weighted and fused. The weighting coefficients are jointly determined by the domain weight matrix and the protocol type confidence score, and finally, conflict features that can comprehensively reflect the potential semantic conflicts are obtained.

[0098] For example, consider a standard semantic vector (0.12, 0.35, 0.87) for data source A in the industrial control domain and an extended semantic vector (0.21, 0.42, 0.87) for data source B in the government data domain. When extracting semantic deviation features, the difference in each dimension of the two types of vectors is calculated, and combined with the weight coefficients of the industrial control and government data domains, the semantic deviation features are obtained. When extracting protocol format adaptation features, the protocol type corresponding to data source A is Modbus, and the protocol type corresponding to data source B is a custom TCP protocol. The adaptation of the protocol formats of the two types of fields to the corresponding standard protocol formats is compared to obtain the protocol format adaptation features. The weight coefficients of the core fields in the industrial control domain (0.8) and the core fields in the government data domain (0.7) are retrieved from the domain weight matrix. The confidence scores of the protocol types are 0.9 (Modbus protocol) and 0.85 (custom TCP protocol), respectively. Based on the above weights and confidence scores, the two types of features are weighted and summed to obtain conflict features, which comprehensively reflect the potential conflicts in the semantics and formats of the fields corresponding to the two types of vectors.

[0099] This enables full-dimensional detection of semantic conflicts, covering two types of conflict scenarios: differences in the semantic connotation of fields and incompatibility of protocol formats. It avoids the omission of conflicts caused by single feature detection. It also makes conflict features fit the business priorities and protocol recognition reliability of different domains, improves the objectivity and accuracy of conflict features, ensures that the actual situation of semantic conflicts is truly reflected, and provides basic data for subsequent conflict probability calculation.

[0100] The conflict characteristics of a single mapping relationship can only reflect the potential conflict of a single mapping and cannot reflect the conflict trend of multiple consecutive mapping relationships. If semantic conflict is judged based on a single conflict characteristic, it is easily affected by accidental factors, leading to misjudgment or omission of conflict judgment. Specifically, the conflict prediction time window and the preset benchmark value are retrieved first. The conflict prediction time window is the statistical interval of multiple consecutive mapping relationships. Its duration is set based on the mapping frequency of the system's business scenario to ensure that a sufficient number of consecutive mapping relationships are covered and the temporal fluctuations of conflict characteristics are captured. The preset benchmark value is the system's preset benchmark for the dispersion of conflict characteristics. It is set based on historical conflict data and business requirements and is used as a criterion for quantifying the probability of conflict.

[0101] Conflict features corresponding to multiple consecutive mapping relationships are statistically analyzed using a sliding window based on the conflict prediction time window. The sliding method involves moving the window group by group with a fixed step size. If only one mapping relationship is set, the conflict features are fully covered group by group, avoiding the omission of small local conflict features caused by large-step jump statistics. At the same time, there is no redundant calculation. While ensuring the integrity of conflict fluctuation statistics, the computational cost is balanced to adapt to the sliding statistics requirements of multiple consecutive mapping relationships. After each move, the values ​​of all conflict features in the current window are counted. The variance calculation method is used to calculate the dispersion of conflict features in the window. The dispersion is used to characterize the fluctuation of conflict features in the window. The greater the fluctuation, the more unstable the potential for conflict in the continuous mapping process, and the higher the probability of conflict. After the calculation, the dispersion of conflict features in the window is compared with a preset benchmark value. This ratio is the conflict probability. The larger the ratio, the higher the probability of semantic conflict, and vice versa, thus realizing the quantitative characterization of the probability of semantic conflict.

[0102] For example, if the conflict prediction time window is set to include 5 consecutive mapping relationships, with a preset baseline value of 0.5, and the conflict feature values ​​corresponding to the consecutive mapping relationships are 0.6, 0.58, 0.62, 0.59, and 0.61 respectively, and sliding statistics are performed according to the conflict prediction time window, the variance of the conflict features within the window is calculated, and the dispersion is found to be 0.0002. The ratio of this dispersion to the preset baseline value of 0.5 is calculated, and the conflict probability is 0.0004, indicating that the semantic conflict probability in the current consecutive mapping process is low. If the conflict feature values ​​within the window are 0.6, 0.8, 0.4, 0.9, and 0.3 respectively, the calculated dispersion is 0.0625, and the ratio to the preset baseline value of 0.5 is 0.125, indicating that the semantic conflict probability has increased, and the conflict change trend needs to be monitored.

[0103] This allows us to capture the fluctuation trend of conflict features in multiple consecutive mapping relationships, avoiding misjudgments or omissions caused by single mapping conflict features, and improving the comprehensiveness of semantic conflict detection. It also makes conflict determination more objective and operable, and more accurately reflects the actual possibility of semantic conflict. The conflict probability serves as the basis for subsequent determination of whether to generate a compensation plan, providing a basic reference for subsequent compensation plan configuration, and ensuring logical coherence and accurate connection between semantic conflict detection and compensation plan generation.

[0104] Specifically, generating a compensation plan includes:

[0105] When the probability of conflict exceeds the probability threshold, the conflict type is classified according to the dimensional ratio of semantic deviation features and protocol format adaptation features, and the conflict level is labeled by associating the discreteness of the corresponding conflict features.

[0106] The quality score of the data source is determined based on data integrity, temporal stability and field consistency. The quality score is then weighted by a domain weight matrix to obtain the compensation priority for different data sources.

[0107] Configure a compensation plan that includes type identifier, compensation priority, and execution time, based on the conflict type, conflict level, and compensation priority.

[0108] A conflict probability greater than the probability threshold indicates that a semantic conflict requiring intervention has occurred during the mapping process. The causes of semantic conflicts vary; some stem from deviations in the semantic meaning of fields, while others arise from inappropriate protocol format adaptation. Different causes require different compensation methods. Without classifying conflict types, the compensation methods will be mismatched with the conflict causes, making accurate compensation impossible. Specifically, the conflict probability and a preset probability threshold are retrieved first. Here, the preset probability threshold is set to 0.1 to capture early semantic conflicts, prevent the continuous accumulation and deterioration of conflicts, distinguish between normal small fluctuations in data and substantial conflicts, balance the risks of missed and false detections, and adapt to application scenarios involving long-term continuous monitoring of heterogeneous data. The conflict probability is then compared with the probability threshold. Only when the conflict probability is greater than the probability threshold is the conflict type classification and conflict level labeling process initiated. If the conflict probability is less than or equal to the probability threshold, it indicates that the semantic conflict has not reached the level requiring compensation, and this step is not initiated.

[0109] The extracted semantic deviation features and protocol format adaptation features are retrieved, and the dimensionality ratio of these two types of features in the conflicting features is calculated. The calculation method is the ratio of the quantized value of each type of feature to the total quantized value of the conflicting features. The conflict type is classified according to the magnitude of this ratio: if the dimensionality ratio of the semantic deviation feature is greater than that of the protocol format adaptation feature, it is determined to be a semantic deviation conflict, the core cause of which is the deviation in semantic connotation between the standard semantic vector and the extended semantic vector; if the dimensionality ratio of the protocol format adaptation feature is greater than that of the semantic deviation feature, it is determined to be a format adaptation conflict, the core cause of which is the incompatibility of the field protocol format; if the proportions of the two types of features are close, i.e., the difference is within a preset range, it is determined to be a mixed conflict, which involves both semantic deviation and format adaptation issues. Here, the preset range is set to a difference ≤ 0.1 to offset the calculation error generated during feature quantization, accurately define scenarios where the influence of the two types of features is roughly equal, distinguish between single-cause conflicts and mixed conflicts, and ensure the rigor of the conflict type classification.

[0110] The dispersion of conflict characteristics is retrieved and combined with the preset conflict level classification standard to mark the conflict level. The lowest dispersion corresponds to Level 1 conflict, i.e., minor conflict, with a small impact range and stable fluctuations. The medium dispersion corresponds to Level 2 conflict, i.e., moderate conflict, with a moderate impact range and controllable fluctuations. The highest dispersion corresponds to Level 3 conflict, i.e., severe conflict, with a wide impact range and violent fluctuations. The conflict level directly reflects the urgency and impact range of the conflict.

[0111] For example, if the preset probability threshold is 0.1 and the calculated conflict probability is 0.15, which is greater than the probability threshold, this step is initiated. The semantic deviation feature quantization value is retrieved as 0.6, the protocol format adaptation feature quantization value as 0.4, and the total conflict feature quantization value as 1. The calculated semantic deviation feature dimension ratio is 0.6, and the protocol format adaptation feature dimension ratio as 0.4. The semantic deviation feature ratio is higher, indicating a semantic deviation-type conflict. The dispersion of the conflict features is retrieved. If this dispersion is within the preset medium range, it is labeled as a level 2 conflict, indicating that the conflict is mainly semantic deviation-based, with a moderate impact and controllable fluctuations, requiring semantic calibration compensation. If the semantic deviation feature quantization value is 0.45 and the protocol format adaptation feature quantization value is 0.55, and the difference in the proportions of the two types of features is within the preset range, it is determined to be a mixed conflict. Combined with the conflict feature dispersion being in the highest range, it is labeled as a level 3 conflict, indicating that the conflict simultaneously involves semantic deviation and format adaptation issues, with a wide impact and drastic fluctuations, requiring comprehensive compensation.

[0112] This allows for the precise identification of the core causes of conflict, providing a basis for subsequent configuration of targeted compensation strategies and preventing compensation methods from being out of touch with the causes of conflict, thus avoiding ineffective compensation. By combining the degree of dispersion of conflict characteristics to label the conflict level, the severity and urgency of the conflict are quantified, providing a quantitative reference for determining compensation priorities and ensuring that compensation resources are allocated to severe conflicts first, thereby improving compensation efficiency.

[0113] In the process of fusion of heterogeneous data from multiple systems, the quality of different data sources varies. High-quality data sources have highly reliable fields, while low-quality data sources are prone to conflicts and have poor compensation effects. If the quality of data sources is not differentiated and a uniform compensation priority is adopted, it will lead to a waste of compensation resources and will not guarantee the accuracy of the fusion results. Specifically, for each data source, a quantitative evaluation is carried out from three dimensions: data integrity, temporal stability, and field consistency. The scores for each dimension are determined based on preset evaluation criteria. Data integrity is evaluated based on the number and proportion of missing fields. No missing fields result in the highest score, and the higher the missing proportion, the lower the score. Temporal stability is evaluated based on the temporal fluctuations of data transmission and the stability of update frequency. No temporal fluctuations and a stable update frequency result in the highest score, and the greater the fluctuations and the more disordered the updates, the lower the score. Field consistency is evaluated based on the degree of fit between the semantics of the data source fields and the semantics of the system's preset standards. A perfect fit results in the highest score, and the lower the fit, the lower the score.

[0114] The initial quality score for each data source is obtained by weighted summation of the scores across the three dimensions. The weighting coefficients are set based on the degree of impact of the three dimensions on the reliability of the data source. The weighting coefficients for data integrity and field consistency are higher than those for temporal stability, ensuring that the quality score truly reflects the core reliability of the data source. Next, the domain weight matrix is ​​retrieved. The weighting coefficients in this matrix are used to quantify the business priority of data sources in different domains. The weighting coefficients for core domain data sources are higher than those for non-core domain data sources. The initial quality score of each data source is weighted and calculated with the corresponding domain weighting coefficient to obtain the quality score for each data source. Based on the quality score, the compensation priority for different data sources is determined. The higher the quality score, the higher the compensation priority, and the conflicts of that data source are compensated first.

[0115] For example, consider data source A in the industrial control domain and data source B in the government data domain. The evaluation results for data source A are: data integrity score 0.9, time-series stability score 0.8, and field consistency score 0.85. The weighting coefficients for the three dimensions are 0.4, 0.2, and 0.4, respectively, resulting in an initial quality score of 0.9 × 0.4 + 0.8 × 0.2 + 0.85 × 0.4 = 0.86. Retrieving the domain weight matrix, the weight coefficient for the core data source in the industrial control domain is 0.8, and the quality score is 0.86 × 0.8 = 0.688. The evaluation results for data source B are... Based on the integrity score of 0.85, the temporal stability score of 0.9, and the field consistency score of 0.8, the initial quality score is 0.85×0.4+0.9×0.2+0.8×0.4=0.84. The weight coefficient of the core data source in the government data field is 0.7, and the quality score is 0.84×0.7=0.588. The quality score of data source A is higher than that of data source B. Therefore, the compensation priority of data source A is higher than that of data source B. Priority is given to compensating for the conflict of data source A, which is in line with the business priority needs of the industrial control field, while ensuring the reliability of the integration of high-quality data sources.

[0116] This comprehensively and objectively reflects the quality level of each data source, avoiding misjudgments caused by single-dimensional evaluation. By combining the domain weight matrix to weight the quality score, the compensation priority takes into account both the quality of the data source itself and the priority needs of the business domain, achieving a reasonable allocation of compensation resources and avoiding waste. The compensation priority provides a sequential basis for the configuration of subsequent compensation plans, ensuring that data sources in high-quality and high-priority domains are compensated first, improving compensation efficiency and the reliability of the fusion results.

[0117] The conflict type determines the core direction of the compensation strategy, the conflict level determines the intensity and urgency of the compensation, and the compensation priority determines the order of compensation. All three are indispensable; the absence of any parameter will result in a compensation plan lacking focus and operability, failing to achieve precise compensation. By integrating conflict type, conflict level, and compensation priority, corresponding compensation strategy directions are configured for different conflict types. Semantic deviation conflicts are configured with semantic calibration compensation strategies to correct semantic deviations between standard and extended semantic vectors; format adaptation conflicts are configured with format regularization compensation strategies to resolve protocol format incompatibility issues; and hybrid conflicts are configured with a comprehensive compensation strategy combining semantic calibration and format regularization.

[0118] Each conflict type is assigned a type identifier, which uses a combination of characters and numbers to ensure uniqueness and facilitate subsequent identification and matching of the corresponding compensation plan. Simultaneously, the aforementioned compensation priority is directly incorporated into the compensation plan, clarifying the order of compensation for conflicts from different data sources, with higher-priority compensation plans being executed first. Based on the conflict prediction time window, an execution period for the compensation plan is set. The start time of the execution period is the point in time when the conflict probability exceeds the probability threshold, and the end time is the start time of the next conflict prediction time window. This ensures that the compensation plan is executed promptly during the conflict's duration, preventing the conflict from escalating. If the compensation plan is not completed within the execution period, a priority escalation mechanism is automatically triggered to ensure timely implementation of compensation.

[0119] For example, for the aforementioned semantically biased, level-two conflict, industrial control data source A (high priority), the compensation plan is configured with the type identifier set to YC-001, where YC represents semantically biased conflict and 001 is a unique sequence identifier. The compensation priority is set to level one, i.e., the highest priority, and the execution time is set to the remaining duration of the current conflict prediction time window to ensure that compensation is completed within the current conflict prediction time window. The compensation strategy direction is semantic calibration, specifically correcting the semantic bias of the extended semantic vector to make it consistent with the semantic connotation of the standard semantic vector. For the mixed, level-three conflict, government data source B (low priority), the compensation plan is configured with the type identifier set to HY-001, where HY represents mixed conflict and 001 is a unique sequence identifier. The compensation priority is set to level three, and the execution time is set to the remaining duration of the current conflict prediction time window and the first half of the next conflict prediction time window to ensure that comprehensive compensation, i.e., semantic calibration and format normalization, is completed during the conflict period to avoid the conflict from escalating.

[0120] This ensures that the compensation plan is targeted, operable, and reasonable, that the compensation strategy is accurately matched with the causes of conflict, and that the order and intensity of compensation are in line with actual needs; it achieves accurate matching between the compensation plan and subsequent semantic conflicts, avoiding mismatch of compensation plans; it ensures timely execution of compensation, avoids the escalation of conflicts, and improves the compensation effect; the compensation plan is the core basis for the subsequent fusion module to perform compensation operations and match semantic conflicts, the type identifier is used to accurately match with semantic conflicts, the compensation priority is used to determine the order of compensation execution, and the execution time limit is used to limit the compensation execution time, affecting the accuracy and reliability of the fusion results, ensuring that semantic conflicts are effectively resolved, and guaranteeing the smooth progress of heterogeneous data fusion from multiple systems.

[0121] S3. Align the data source fields according to the mapping relationship, determine the confidence weight according to the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, execute the corresponding compensation plan and then perform weighted fusion to obtain the fusion result; otherwise, directly perform weighted fusion and update the predicted conflict probability.

[0122] Furthermore, the fusion results include:

[0123] Align the corresponding fields of standard feature segments and extended feature segments between different data sources according to the mapping relationship, and determine the credibility weight of the fields based on the historical consistency of the data sources.

[0124] If the semantic conflict matches the type identifier in the compensation plan, and the execution time is within the conflict prediction time window, then the corresponding compensation operation will be executed according to the compensation priority.

[0125] The compensated standard feature segments and extended feature segments are weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result.

[0126] The standard feature segments and extended feature segments of heterogeneous data sources from multiple systems have problems such as disordered field arrangement and unclear semantic correspondence. If field alignment is not performed based on the mapping relationship, the corresponding fields of different data sources will not match, and data fusion will not be possible. Specifically, based on the aforementioned generated mapping relationship, the standard feature segment fields of each data source are aligned with the corresponding standard feature segment fields of other data sources, and the extended feature segment fields of each data source are aligned with the corresponding extended feature segment fields of other data sources. During the alignment process, it is ensured that the semantic connotation of the corresponding fields is consistent to avoid fusion errors caused by field mismatch.

[0127] After alignment, historical data records from each data source are retrieved to analyze historical consistency. Analysis dimensions include missing historical field data, semantic fluctuations, and temporal synchronization. Through comprehensive analysis of these dimensions, the historical consistency level of corresponding fields in each data source is determined. Based on the historical consistency level, a credibility weight is assigned to each field; the higher the historical consistency level, the higher the credibility weight, and vice versa. The weight allocation process also considers the domain weight matrix, increasing the field weights of core domain data sources to ensure that the weight allocation aligns with the historical reliability of the data source and the domain's business priorities.

[0128] For example, consider data source A in the industrial control domain and data source B in the government data domain. The mapping relationship is clear: the standard feature segment "Equipment Operating Temperature" of data source A semantically corresponds to the extended feature segment "Environmental Monitoring Temperature" of data source B; the extended feature segment "Equipment Running Time" of data source A semantically corresponds to the standard feature segment "Monitoring Duration" of data source B. Aligning the "Equipment Operating Temperature" field with the "Environmental Monitoring Temperature" field, and aligning the "Equipment Running Time" field with the "Monitoring Duration" field, ensures semantic correspondence between the two types of fields. When analyzing historical consistency, data source A's "Equipment Operating Temperature" field has no missing data, no semantic fluctuations, and stable temporal synchronization over the past 6 months, indicating a high level of historical consistency, and is assigned a confidence weight of 0.8. Data source B's "Environmental Monitoring Temperature" field has a few missing data points and minimal semantic fluctuations over the past 6 months, indicating a moderate level of historical consistency, and is assigned a confidence weight of 0.6. Data source A's "Equipment Running Time" field has a moderate level of historical consistency, and is assigned a weight of 0.7. Data source B's "Monitoring Duration" field has a high level of historical consistency, and is assigned a weight of 0.85. Simultaneously, combining the domain weight matrix, the weight of the industrial control domain fields is appropriately increased by 0.1, while the weight of the government data domain fields remains unchanged, thus determining the confidence weight of each field.

[0129] This enables precise alignment of fields from different data sources, eliminates fusion obstacles caused by disordered field arrangement and semantic mismatch, achieves reliable field differentiation, avoids fusion errors caused by uniform weights, and improves the accuracy of fusion results; it also ensures that weight settings align with business domain needs and guarantees the practicality of fusion results.

[0130] Semantic conflicts are detected during the fusion process. If these conflicts are not compensated, the fusion results will suffer from semantic bias and format incompatibility, reducing fusion quality. Specifically, the current semantic conflict type and the generated compensation plan are retrieved, and the type identifier, compensation execution time limit, and compensation priority in the compensation plan are extracted. The current semantic conflict type is compared with the type identifier in the compensation plan to verify their consistency. If they do not match, it indicates that the compensation plan is not applicable to the current conflict, and no compensation operation is performed. If they match, it is verified whether the compensation execution time limit is within the limited conflict prediction time window. If the time limit exceeds the conflict prediction time window, it indicates that the compensation plan has expired, and no compensation operation is performed. If the identifier matches and the time limit is valid, the corresponding compensation operation is performed according to the compensation priority in the compensation plan.

[0131] The specific type of compensation operation corresponds to the determined conflict type. That is, semantic deviation type conflicts are subject to field semantic calibration to correct the deviation between the extended semantic vector and the standard semantic vector, ensuring field semantic consistency; protocol format adaptation type conflicts are subject to extended field format normalization to adjust the format of the extended feature segment to adapt it to the standard feature segment format; and mixed type conflicts are subject to a comprehensive compensation operation that combines semantic calibration and format normalization. When performing compensation operations, the requirements of the compensation plan are followed, and high-priority compensation plans are executed first to ensure that high-priority conflicts are resolved first and to guarantee the stability of the fusion process.

[0132] For example, if the currently detected semantic conflict is a semantic deviation conflict, the generated compensation plan, with type identifier YC-001 corresponding to a semantic deviation conflict, has a compensation execution timeframe within the current conflict prediction time window, a compensation priority of level one, and a compensation operation of field semantic calibration. If the current conflict type matches the type identifier and the compensation execution timeframe is within the conflict prediction time window, the semantic calibration operation is performed according to the compensation priority. For the environmental monitoring temperature field of data source B, its semantic vector is corrected to be consistent with the semantic vector of the device operating temperature field of data source A, eliminating semantic deviation. If the current semantic conflict is a format adaptation conflict, with compensation plan type identifier GS-001 corresponding to a format adaptation conflict, but the compensation execution timeframe has exceeded the conflict prediction time window, the compensation timeframe is deemed invalid, and the compensation plan is not executed to avoid wasting system resources due to invalid compensation.

[0133] This ensures that compensation operations are targeted and timely, avoiding ineffective compensation and waste of resources; ensures that high-priority conflicts are resolved first, improving compensation efficiency; and achieves precise resolution of semantic conflicts, eliminating the adverse effects of conflicts on the fusion results and providing a high-quality data foundation for subsequent weighted fusion.

[0134] Although the compensated standard feature segments and extended feature segments have eliminated conflicts, the field credibility varies across different data sources. If a simple summation or averaging method is used for fusion, it cannot reflect these differences in field credibility, leading to low-credibility fields negatively impacting the fusion results and reducing their accuracy and reliability. Specifically, the compensated standard and extended feature segments from each data source, along with the credibility weights of each field determined earlier, are retrieved and processed separately to avoid fusion errors caused by semantic and format differences between the two types of feature segments. For standard feature segments, the compensated data for the corresponding field from each data source is weighted by the credibility weight of that field, i.e., the product of the field data and the weight, yielding the weighted data for the corresponding field in the standard feature segment of each data source. Similarly, the weighted data for the corresponding field in the extended feature segment of each data source is obtained.

[0135] Then, the weighted data of the corresponding fields of the standard feature segments of all data sources are summarized to obtain the fused data of the standard feature segments; the weighted data of the corresponding fields of the extended feature segments of all data sources are summarized to obtain the fused data of the extended feature segments; finally, the fused data of the standard feature segments and the fused data of the extended feature segments are concatenated according to the protocol semantic level to obtain the fusion result. The fusion result retains the core data of the standard feature segments and the custom data of the extended feature segments to ensure the integrity and usability of the data.

[0136] For example, after compensation, the equipment operating temperature field data from data source A is a preset standard value, and its credibility weight, enhanced by domain weighting, is 0.9. The environmental monitoring temperature field data from data source B, after semantic calibration, is semantically consistent with the field data from data source A, and its credibility weight is 0.6. When weighted fusion of standard feature segments, only data source A has standard feature segment data for this field, and the weighted calculation yields the fused data for the standard feature segments. When weighted fusion of extended feature segments, only data source B has extended feature segment data for this field, and the weighted calculation yields the fused data for the extended feature segments. The two types of fused data are then concatenated according to the protocol semantic hierarchy to obtain a fusion result containing both the standard equipment operating temperature data and the extended environmental monitoring temperature data. As another example, the compensated data for the equipment runtime field from data source A has a weight of 0.8; the compensated data for the monitoring duration field from data source B has a weight of 0.85. Weighted calculations are performed on the two types of feature segments separately, and the data is then summarized to obtain the fused data for the corresponding fields. Finally, these are concatenated to form the fusion result.

[0137] This reflects the reliability differences of fields from different data sources, making the fusion results more consistent with high-confidence data and improving the accuracy and reliability of the fusion results; it also takes into account the semantic and format differences of the two types of feature segments, avoiding semantic ambiguity and format conflicts in the fusion process, and ensuring the integrity of the fusion results; and it gives the fusion results a unified structure, which facilitates subsequent deviation detection and subsequent data processing of the system.

[0138] Specifically, updating the predicted conflict probability includes:

[0139] If the semantic conflict does not match the type identifier in the compensation plan, or the execution time exceeds the conflict prediction time window, the standard feature segments and extended feature segments between different data sources are directly weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result.

[0140] The actual deviation values ​​of the fields during this weighted fusion process are extracted, and the predicted conflict probability is updated by combining the dispersion of conflict features within the conflict prediction time window.

[0141] If the semantic conflict does not match the type identifier of the compensation plan, it indicates that the type of conflict targeted by the compensation plan is inconsistent with the current actual conflict. Executing the compensation plan will not resolve the current semantic conflict and will instead waste system resources and delay the fusion process. If the compensation execution time exceeds the conflict prediction time window, it indicates that the compensation plan has expired. At this time, executing the compensation operation can no longer specifically resolve the current conflict and may even lead to new semantic conflicts due to timing deviations. Specifically, the first step is to compare the conflict type with the compensation identifier to verify whether the current semantic conflict type is consistent with the type identifier in the compensation plan. The second step is to verify the validity of the time to determine whether the compensation execution time is within the conflict prediction time window. If either of the above two verifications is not met, i.e., the identifier does not match or the time exceeds the time window, it is determined that the compensation plan is not applicable to the current conflict, the compensation operation is skipped, and the weighted fusion process is started directly.

[0142] The specific operation of weighted fusion is consistent with the compensated weighted fusion logic. First, the generated mapping relationship is retrieved to ensure that the corresponding fields of standard feature segments and extended feature segments are accurately aligned between different data sources. Then, the determined confidence weights of each field are retrieved, and the corresponding fields of standard feature segments and extended feature segments are weighted according to their confidence weights. The weighting method is the product of the field data and the corresponding weight. Then, the weighted results of each field are summarized to obtain the fused data of standard feature segments and extended feature segments respectively. Finally, the two types of fused data are concatenated according to the protocol semantic level to obtain the fusion result.

[0143] For example, if the currently detected semantic conflict is a format adaptation conflict, and the type identifier of the compensation plan is YC-001, corresponding to a semantic deviation conflict, which does not match the type identifier of the current format adaptation conflict, then the compensation plan is determined to be unsuitable for the current conflict. The compensation operation is skipped, and weighted fusion is performed directly. The mapping relationship is retrieved, aligning the standard feature segment equipment operating voltage field of data source A in the industrial control field with the extended feature segment monitoring voltage field of data source B in the government data field. The confidence weight of the field is retrieved, i.e., the weight of this field in data source A is 0.85, and the weight of this field in data source B is 0.65. Weighted calculations are performed on the two types of field data respectively, and the fused data of the extended feature segment and the standard feature segment is summarized. The fused data is then concatenated to obtain the fusion result. As another example, if the current semantic conflict is a mixed conflict, and the type identifier of the compensation plan is HY-001, corresponding to a mixed conflict, but the compensation execution time has exceeded the current conflict prediction time window (i.e., the conflict prediction time window is 2 hours from the current time, and the compensation execution time expires 1 hour ago), then the compensation time is determined to be invalid. The compensation operation is skipped, and weighted fusion is performed according to the confidence weight to obtain the fusion result.

[0144] This allows for precise coverage of scenarios where compensation plans are incompatible or expire, skipping invalid compensation operations, reducing system resource waste, and improving the efficiency of the fusion process. It also ensures consistency in fusion results across the two scenarios, guaranteeing the stability of fusion quality. Simultaneously, it directly merges field data that has not undergone invalid compensation, accurately preserving the field deviations caused by the current conflict, and providing basic data for subsequent updates to predict conflict probabilities.

[0145] When the compensation plan is incompatible or expires, the current semantic conflict is not compensated in a targeted manner. After direct fusion, the fields will inevitably have actual deviations. If the predicted conflict probability is not updated based on this deviation, subsequent conflict predictions will still use the old probability data, which will not accurately reflect the actual situation of the current conflict and may easily lead to misjudgment or omission of subsequent conflicts. Specifically, the actual deviation value of each field in this direct weighted fusion process is extracted. The extraction objects are all corresponding fields of the standard feature segment and the extended feature segment. The extraction method is to compare the weighted fusion result of each field with the standard field benchmark value preset by the system and calculate the difference between the two. This difference is the actual deviation value of the field. The sign of the deviation value reflects the direction of the deviation, and the absolute value reflects the degree of deviation. The higher the degree of deviation, the greater the actual impact of the current conflict.

[0146] Based on the protocol definition of the corresponding fields and the normal operating conditions of the data source, standard field baseline values ​​are set. Among them, the standard feature segment is the core field of the protocol, with fixed values ​​and high precision requirements; the extended feature segment is a manufacturer-defined field, and the mainstream values ​​of this type of field that have been operating stably for a long time are selected, allowing small and reasonable fluctuations. It provides a standard reference for the deviation calculation of the whole system and ensures that the deviation judgment results are objective and consistent. For example, the equipment operating temperature field of the standard feature segment of the industrial data source has a standard field baseline value set to 25; the data length field of the standard feature segment of the government data source has a standard field baseline value set to 10; and the environmental monitoring temperature field of the corresponding extended feature segment has a standard field baseline value set to 24.

[0147] Then, the calculated dispersion of conflict features within the current conflict prediction time window is retrieved to reflect the fluctuation and distribution pattern of conflict features within the current conflict prediction time window. The higher the dispersion, the stronger the instability of the conflict and the greater the possibility of subsequent conflict recurrence. The actual deviation value of the field and the dispersion of the conflict features are weighted and summed for calculation. The weighting coefficient is set based on the degree of influence of the two on conflict prediction. The weight of the actual deviation value of the field is higher than the weight of the dispersion, ensuring that the updated predicted conflict probability is more in line with the actual impact of the current conflict.

[0148] Based on the fusion calculation results, the predicted conflict probability calculated initially is updated. If the actual deviation value is large and the dispersion is high, it indicates that the actual impact of the current conflict exceeds the initial prediction, and the predicted conflict probability is adjusted upward. If the actual deviation value is small and the dispersion is low, it indicates that the actual impact of the current conflict is lower than the initial prediction, and the predicted conflict probability is adjusted downward. The updated predicted conflict probability will be stored synchronously as the core basis for subsequent detection of semantic conflicts and determination of whether to generate a compensation plan.

[0149] For example, after this direct weighted fusion, the actual deviation of the fusion result of the standard feature segment equipment operating voltage field of data source A from the system's preset standard field benchmark value is a certain quantified value, and the actual deviation of the extended feature segment monitoring voltage field of data source B from the system's preset standard field benchmark value is another quantified value. The actual deviation values ​​of the two types of fields are extracted and summarized to obtain the quantified value of the actual deviation in this fusion. The dispersion of the conflict features within the current conflict prediction time window is retrieved. This dispersion is at a moderate level, indicating that the conflict fluctuations are relatively stable. The overall actual deviation value and the dispersion are weighted and summed. The fusion result shows that the actual impact of the current conflict is slightly higher than the initial prediction. Therefore, the initially calculated predicted conflict probability is adjusted upwards, and the updated predicted conflict probability will be used for subsequent conflict detection. Conversely, if the actual deviation value of this fusion is small and the dispersion of the conflict features is low, it indicates that the actual impact of the current conflict is lower than the initial prediction. The predicted conflict probability is adjusted downwards to ensure that subsequent conflict predictions are more consistent with the actual situation.

[0150] This enables dynamic updates to the predicted conflict probability, allowing the conflict probability to reflect the actual state of the current conflict in real time, avoiding misjudgments of subsequent conflicts due to fixed probabilities, and improving the accuracy of conflict prediction; it ensures the objectivity and rationality of probability updates, avoiding update deviations caused by a single factor; the updated predicted conflict probability provides accurate basis for conflict detection, making compensation plans more targeted and improving the closed loop of conflict handling.

[0151] S4. Calculate the deviation between the quality index of the fusion result and the preset business constraints, convert the deviation into the adjustment amount for semantic similarity calculation, and feed back the adjustment similarity threshold and conflict prediction time window.

[0152] Specifically, the feedback adjustment of the similarity threshold includes:

[0153] The actual deviation value of the field is used as the quality index of the fusion result. The deviation between the quality index and the preset business constraints is calculated, and the deviation is converted into the adjustment amount for semantic similarity calculation.

[0154] The adjustment amount is corrected based on the dispersion of conflict features within the conflict prediction time window, and the similarity threshold is adjusted based on the corrected adjustment amount.

[0155] The quality of the fusion result is most directly reflected in the actual deviation value of the fields. If an abstract quality evaluation standard is used, the quality indicator cannot be quantified, and thus the similarity threshold cannot be adjusted accurately. Therefore, the actual deviation value of the fields is retrieved and directly used as the quality indicator of the fusion result. The system's preset business constraints are also retrieved. These preset business constraints must conform to the aforementioned defined business scenarios and protocol types, and are set separately for standard feature segments and extended feature segments. That is, standard feature segments correspond to strict business constraints because they are core protocol fields with higher fusion quality requirements, while extended feature segments correspond to relatively lenient business constraints because they are vendor-defined fields and require a certain degree of flexibility. The specific content of the preset business constraints is adapted to the protocol type and field semantic features to ensure the rationality of the constraint standards.

[0156] The deviation between the quality index and the preset business constraints is calculated, which is the difference between the actual deviation value of the field and the upper / lower limit of the deviation allowed by the corresponding business constraint. If the actual deviation value exceeds the upper limit of the business constraint, the difference is positive, indicating that the fusion quality is not up to standard, and the similarity threshold is adjusted to reduce subsequent fusion deviation. If the actual deviation value is lower than the lower limit of the business constraint, the difference is negative, indicating that the matching accuracy is too high and there is redundant calculation, and the similarity threshold is relaxed to improve fusion efficiency. If the actual deviation value is within the business constraint's allowable range, the difference is zero, indicating that the fusion quality meets the standard, and the similarity threshold is adjusted appropriately to improve fusion efficiency. The deviation value is converted into an adjustment amount for semantic similarity calculation. The conversion logic is aligned with the semantic similarity calculation logic, and a linear conversion method is used to convert the deviation into an adjustment amount at a fixed ratio, ensuring that the adjustment amount is positively correlated with the deviation difference, that is, the larger the deviation difference, the larger the adjustment amount. Here, a fixed ratio of 0.5 is set to achieve linear proportional scaling of the deviation difference, which will not cause the threshold to jump drastically and the system to oscillate due to an excessively large ratio, nor will it cause the adjustment to fail due to an excessively small ratio, thus adapting to the dynamic fine-tuning needs of heterogeneous data.

[0157] For example, in the preset business constraints, the allowable deviation range for standard feature segments is a reasonable fluctuation range of the preset benchmark value, while the allowable deviation range for extended feature segments is a relatively loose fluctuation range. The actual deviation values ​​of the fields are retrieved. The actual deviation value of the standard feature segment's device operating temperature exceeds the upper limit of the preset business constraints, while the actual deviation value of the extended feature segment's monitored temperature is within the allowable range of the preset business constraints. The deviation between the quality indicators and the preset business constraints is calculated. A positive deviation difference for the standard feature segments indicates that the fusion quality is substandard; a zero deviation difference for the extended feature segments indicates that the fusion quality meets the standards. Both types of deviation differences are converted into adjustment values. The adjustment value corresponding to the standard feature segments is positive, indicating that the similarity threshold should be adjusted upwards, i.e., tightening the threshold and improving field matching accuracy. The adjustment value corresponding to the extended feature segments is zero, indicating that the similarity threshold does not need adjustment, ensuring the fusion quality of the standard feature segments while retaining the flexibility of the extended feature segments. As another example, if the actual deviation value of the standard feature segments is lower than the lower limit of the allowable range of the business constraints, the deviation difference is negative, and the converted adjustment value is negative, indicating that the similarity threshold can be adjusted downwards appropriately, i.e., relaxing the threshold and improving fusion efficiency.

[0158] This adapts to the differentiated needs of the two types of feature segments, avoiding unreasonable adjustments due to single business constraints; it quantifies the fusion quality deviation into parameters that can be used for threshold adjustment, improving the accuracy and objectivity of threshold adjustment, and providing a clear quantitative basis for subsequent adjustments to the similarity threshold.

[0159] Adjusting the amount solely based on the quality deviation transformation without considering the fluctuations in the current conflict state can easily lead to a mismatch between the threshold adjustment and the current conflict scenario if directly used to adjust the similarity threshold. Therefore, the discreteness of the conflict features within the current conflict prediction time window is retrieved to reflect the fluctuations and distribution patterns of the current semantic conflict. Higher discreteness indicates a more unstable current conflict and a greater likelihood of deviations during subsequent fusion, thus requiring a moderate tightening of the adjustment amount. Conversely, lower discreteness indicates a more stable current conflict, requiring a moderate relaxation of the adjustment amount to ensure the adjustment logic is compatible with the conflict state.

[0160] The adjustment amount is corrected based on the dispersion of conflict features. The correction method is to calculate the adjustment amount by weighting the dispersion. The weighting coefficient is set based on the degree of impact of dispersion on the fusion quality. The higher the dispersion, the larger the weighting coefficient and the larger the correction magnitude, ensuring that the corrected adjustment amount can adapt to the current fluctuation state of conflict. During the correction process, the adjustment amounts corresponding to standard feature segments and extended feature segments are distinguished and corrected separately. That is, the adjustment magnitude of the standard feature segment is appropriately increased because the standard feature segment is a core field and requires higher matching accuracy, while the adjustment magnitude of the extended feature segment is appropriately reduced because the extended feature segment needs to retain flexibility and be consistent with the differentiated settings of preset business constraints.

[0161] After the correction is completed, the similarity threshold is adjusted based on the corrected adjustment amount. If the corrected adjustment amount is positive, it indicates that the similarity threshold needs to be tightened to improve the accuracy of field semantic matching and reduce subsequent fusion deviation. If the corrected adjustment amount is negative, it indicates that the similarity threshold needs to be relaxed to improve fusion efficiency and avoid excessive tightening that leads to matching failure. If the corrected adjustment amount is zero, it indicates that the current similarity threshold has adapted to the fusion quality and conflict status and does not need to be actively adjusted. During the adjustment process, the initial setting logic of the similarity threshold is used to ensure that the adjusted similarity threshold is compatible with the field semantic features and protocol type.

[0162] For example, the adjustment amount of the standard feature segment obtained above is positive, and the adjustment amount of the extended feature segment is zero. The dispersion of the conflict features within the conflict prediction time window is retrieved. The dispersion is at a high level, indicating that the conflict fluctuates violently and the possibility of subsequent fusion deviation is high. The adjustment amount of the standard feature segment is tightened and corrected, and the adjustment range is increased to improve the similarity threshold tightening range, improve the matching accuracy of the standard feature segment field, and reduce the fusion deviation caused by conflict fluctuation. The adjustment amount of the extended feature segment is corrected for fault tolerance. Only boundary verification is performed without changing the value, and the adjustment amount is kept at zero to ensure the flexibility of the extended feature segment.

[0163] If the dispersion of conflict features is at a low level, i.e. the conflict fluctuation is stable, the adjustment amount of the standard feature segment is moderately relaxed and the tightening range is reduced, so as to improve the fusion efficiency while ensuring matching accuracy; the adjustment amount of the extended feature segment remains zero and no correction is required; after the adjustment is completed, the similarity threshold of the standard feature segment is moderately tightened and the similarity threshold of the extended feature segment remains unchanged, which is both suitable for the current conflict fluctuation state and meets the differentiated needs of the two types of feature segments.

[0164] For example, if the adjustment amount is negative and the dispersion of the conflict features is low, the adjustment amount is relaxed to increase the threshold relaxation range and improve the fusion efficiency; if the dispersion of the conflict features is high, the adjustment amount is tightened to reduce the threshold relaxation range and avoid matching deviation due to excessive relaxation.

[0165] This allows the similarity threshold adjustment to adapt to the current conflict fluctuation state, align with the differentiated needs of the two types of feature segments, and improve the targeting of the threshold adjustment; based on the corrected adjustment amount, the similarity threshold is precisely adjusted to ensure that the threshold adjustment meets the fusion quality requirements and conflict fluctuation state, improving the accuracy of subsequent field semantic matching and the reliability of fusion results; the adjusted similarity threshold is used for the generation and screening of subsequent mapping relationships, improving the accuracy of mapping relationships and the quality of subsequent fusion processes; at the same time, it participates in the conflict prediction time window adjustment logic to ensure parameter adaptability and operational stability, providing support for subsequent continuous optimization.

[0166] Specifically, the feedback adjustment time window for conflict prediction includes:

[0167] The cumulative amount of actual deviation values ​​of fields within the current conflict prediction time window is statistically analyzed, and the window deviation adaptation coefficient is calculated in combination with the dispersion of conflict characteristics.

[0168] The window bias adaptation coefficient is weighted and corrected based on the domain weight matrix to distinguish the conflict response differences between the standard feature segment and the extended feature segment, and to generate the scaling ratio of the corresponding conflict prediction time window.

[0169] The duration of the conflict prediction time window is dynamically adjusted according to the scaling ratio.

[0170] The adaptability of the conflict prediction time window directly determines the accuracy of conflict detection. If a fixed-length window or simple linear adjustment is used without considering the actual conflict impact and fluctuation state within the window, the window length will not match the current conflict scenario. Specifically, first, retrieve the time range of the current conflict prediction time window, determine the actual deviation value of the fields corresponding to all mapping relationships included in the window, and sum the actual deviation values ​​of all fields to obtain the cumulative amount of the actual deviation values ​​of the fields within the current window. The larger the cumulative amount, the more significant the overall impact of the conflict on the fusion quality within the window. Adjust the window length to better capture the conflict patterns.

[0171] The dispersion of conflict features within the current conflict prediction time window is retrieved to reflect the fluctuation state of the conflict within the window. The higher the dispersion, the more unstable and volatile the conflict. The window duration is adjusted to adapt to the conflict fluctuations. The lower the dispersion, the more stable the conflict. The window duration can be appropriately optimized to improve detection efficiency. The actual cumulative deviation value and the dispersion of conflict features are weighted and fused to obtain the window deviation fitting coefficient. The weighting coefficient is set based on the influence of the two on the window fit. The weight of the actual cumulative deviation value is higher than the weight of the dispersion to ensure that the fitting coefficient is closer to the actual impact of the conflict within the window and avoids fitting deviation caused by a single factor. During the calculation process, the actual cumulative deviation value and the dispersion of conflict features are normalized to eliminate the difference in their dimensions.

[0172] For example, the current conflict prediction time window contains 5 sets of mapping relationships, covering the standard feature segments of equipment operating temperature and equipment operating voltage fields from data source A in the industrial control field, and the extended feature segments of monitoring temperature and monitoring voltage fields from data source B in the government data field. The actual deviation values ​​of each field are retrieved and summed to obtain the cumulative actual deviation value. The dispersion of conflict features within the current window is found to be at a high level. After normalizing both, a window deviation adaptation coefficient is calculated using a preset weighting coefficient. If this coefficient is greater than 1, it indicates that the current window duration is too short and cannot capture the conflict fluctuation pattern, requiring a longer window duration. If the cumulative actual deviation value within the window is small and the dispersion of conflict features is low, the calculated window deviation adaptation coefficient is less than 1, indicating that the current window duration is too long, introducing invalid conflict data, requiring a shorter window duration. Conversely, if the cumulative actual deviation value is moderate and the dispersion of conflict features is moderate, the calculated window deviation adaptation coefficient is close to 1, indicating that the current window duration is suitable for the conflict state and no adjustment is needed.

[0173] This allows for the quantification of the overall impact of conflicts within a window. By combining the dispersion of conflict characteristics with the degree of impact and fluctuation patterns, the window deviation adaptation coefficient can objectively and accurately characterize the adaptation status of the current window duration, providing a reliable quantitative basis for the generation of subsequent window scaling ratios.

[0174] Data sources from different domains have different business priorities during the fusion process. Conflicts in core domains have a greater impact on fusion quality, and the conflict response characteristics of standard feature segments and extended feature segments differ. If the window deviation adaptation coefficient is not weighted and the differences in conflict response between the two types of feature segments are not distinguished, the generated scaling ratio will not match the business priority and field characteristics, resulting in insufficient window adaptation for core domain conflicts and excessive window adjustment for non-core domain conflicts. Specifically, the domain weight matrix is ​​retrieved first, and the weight coefficient of each data source domain is extracted. The weight coefficient of core domains is higher than that of non-core domains to ensure that the weighted correction matches the business priority.

[0175] The window deviation adaptation coefficients obtained above are weighted and corrected based on the domain weight matrix. The correction method is to multiply the window deviation adaptation coefficients with the corresponding domain weight coefficients. The correction magnitude is larger for the core domain to ensure more accurate conflict window adaptation. At the same time, the conflict response differences between standard feature segments and extended feature segments are distinguished, and the window deviation adaptation coefficients corresponding to the two types of feature segments are corrected separately. The correction coefficient for standard feature segments is higher than that for extended feature segments because the conflict impact of standard feature segments is greater and requires more accurate window adaptation. The correction coefficient for extended feature segments is lower, which takes into account flexibility and adaptability and avoids over-adjustment.

[0176] After correction, the scaling ratio of the conflict prediction time window is generated based on the corrected window bias adaptation coefficient. The corrected window bias adaptation coefficient is used as the scaling ratio. If the corrected adaptation coefficient is greater than 1, the scaling ratio is greater than 1, indicating that the window duration is lengthened, and the lengthening amount is positively correlated with the degree of deviation of the adaptation coefficient from 1. If the corrected adaptation coefficient is less than 1, the scaling ratio is less than 1, indicating that the window duration is shortened, and the shortening amount is positively correlated with the degree of deviation of the adaptation coefficient from 1. If the corrected adaptation coefficient is equal to 1, the scaling ratio is 1, and the window duration does not need to be adjusted. The scaling ratio needs to be verified for reasonableness to ensure that the ratio is within a preset reasonable range, such as 0.5 to 2, to avoid excessive adjustment of the window duration, which would affect the stability of conflict detection.

[0177] For example, if the window deviation adaptation coefficient is 1.3, indicating that the window needs to be lengthened, the domain weight matrix is ​​retrieved. The weight coefficient for the industrial control domain is 0.8, and the weight coefficient for the government data domain is 0.7. The window deviation adaptation coefficient corresponding to the industrial control domain is weighted and corrected, resulting in a coefficient of 1.3 × 0.8 = 1.04. The same correction is applied to the window deviation adaptation coefficient corresponding to the government data domain, resulting in a coefficient of 1.3 × 0.7 = 0.91. Distinguishing between standard feature segments and extended feature segments, the correction coefficient for the standard feature segment is 1.1, and for the extended feature segment it is 0.9. Adjusting the corrected adaptation coefficients, the final adaptation coefficient for the standard feature segment is 1.04 × 1.1 ≈ 1.14, corresponding to a scaling ratio of 1.14 (window lengthened by 14%). The final adaptation coefficient for the extended feature segment is 0.91 × 0.9 ≈ 0.82, corresponding to a scaling ratio of 0.82 (window shortened by 18%).

[0178] By combining the differences in conflict response between the two types of feature segments with domain priority, a scaling ratio of 1.1 is generated for the current conflict prediction time window. This ensures both the window adaptation accuracy of standard feature segments in the industrial control domain and the flexibility of extended feature segments in the government data domain, avoiding over-adjustment. For example, if the corrected adaptation coefficient is 0.8, indicating a need to shorten the window, a scaling ratio of 0.9 is generated after adjusting for domain weights and feature segment differences. This ensures that the window duration is optimized without affecting the integrity of conflict detection.

[0179] This ensures that the deviation adaptation coefficient aligns with business priorities, guaranteeing more accurate conflict window adaptation in core areas and improving the system's business adaptability; it enables differentiated correction and scaling ratio generation, avoiding unreasonable adaptation due to a single ratio and ensuring the stability of conflict detection; and it provides a quantitative basis for subsequent window duration adjustments, ensuring that the adjustment range is highly adapted to the current conflict status, business priorities, and field characteristics.

[0180] Specifically, the aforementioned scaling ratio is retrieved to clarify the direction and magnitude of the window duration adjustment. Combined with the initial duration of the current conflict prediction time window, the adjusted window duration is calculated by multiplying the initial duration by the scaling ratio, ensuring that the adjustment magnitude is accurate and controllable. During the adjustment process, the preset upper and lower limits of the window duration are followed to avoid the adjusted window duration being too short or too long. The upper and lower limits are set based on the system business scenario and protocol type, and are adapted to the protocol type and conflict detection logic to ensure that the adjusted window duration is practically operable.

[0181] After the adjustment is completed, the validity of the adjusted window duration is verified, including whether the adjusted window duration can cover at least one complete mapping relationship, whether it is suitable for the dispersion of the current conflict characteristics, and whether it fits the domain business priority. If the verification is passed, the conflict prediction time window duration is determined. If the verification fails, the scaling ratio is adjusted and the window duration is recalculated until the verification is passed. The adjusted conflict prediction time window will be synchronously fed back to the conflict detection process for the subsequent conflict characteristic statistics and conflict probability calculation of multiple consecutive mapping relationships, ensuring that conflict detection is synchronized with the current window state.

[0182] For example, if the initial duration of the current conflict prediction time window is 10 minutes, and the aforementioned scaling ratio is 1.1 (stretched by 10%), the adjusted window duration is calculated to be 10 × 1.1 = 11 minutes. The preset upper limit of the window duration is 15 minutes and the lower limit is 5 minutes. 11 minutes is within a reasonable range. After validity verification, the adjusted window duration is determined to be 11 minutes. This is then fed back to the conflict detection process for subsequent sliding statistics of conflict features and conflict probability calculation, to more completely capture the current volatile conflict patterns and improve the accuracy of conflict detection. For example, if the initial window duration is 10 minutes and the scaling ratio is 0.9 (reducing by 10%), the adjusted window duration is 9 minutes, which is within a reasonable range. After verification, it is updated synchronously. The shortened window can reduce the introduction of invalid conflict data, improve the conflict detection efficiency, and at the same time take into account the integrity of conflict capture. If the scaling ratio is 1.6, the calculated adjusted window duration is 16 minutes, which exceeds the preset upper limit of 15 minutes, and the verification fails. The scaling ratio is then fine-tuned to 1.5, and the adjusted window duration is 15 minutes. After verification again, it is determined as the final duration.

[0183] This enables precise adaptation of the conflict prediction time window to the current conflict state, business priority, and field characteristics, improving the accuracy and efficiency of conflict detection; avoiding excessive window adjustment, ensuring the stability and operability of conflict detection; and synchronously feeding the adjusted window duration back to the conflict detection process to ensure that conflict prediction remains synchronized with the current window state. This is used for subsequent sliding statistics of conflict features and calculation of conflict probability for multiple consecutive sets of mapping relationships, determining the accuracy of conflict detection, and forming a closed-loop logic of window adjustment, conflict detection, and feedback optimization, thereby improving the system's adaptive handling capability for heterogeneous data conflicts.

[0184] Example 2:

[0185] like Figure 3 The diagram shown illustrates the system architecture of an automated fusion and docking system for heterogeneous data from multiple systems, as provided in this application embodiment. The system includes a sensing module, a mapping module, a fusion module, and a feedback module.

[0186] The perception module is used to send sniffing packets to multiple data source systems, identify the protocol type of different data sources based on the information entropy distribution of data frames, and output the protocol data stream.

[0187] The mapping module is used to extract the semantic features of the fields of the protocol data stream and embed them into a unified vector space. It calculates the semantic similarity across systems based on the system domain type to generate mapping relationships and detects semantic conflicts in the mapping process. When the predicted conflict probability is greater than the probability threshold, a compensation plan is generated based on the data source quality.

[0188] The fusion module is used to align data source fields according to the mapping relationship, determine the confidence weight based on the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, the corresponding compensation plan is executed and then weighted fusion is performed to obtain the fusion result; otherwise, the weighted fusion is performed directly and the predicted conflict probability is updated.

[0189] The feedback module is used to calculate the deviation between the quality index of the fusion result and the preset business constraints, convert the deviation into the adjustment amount for semantic similarity calculation, and feed back the adjustment similarity threshold and conflict prediction time window.

[0190] Since the principle of the method in this application embodiment is similar to that of the system described in this application embodiment, the implementation of the method is the same as that of the system, and the repeated parts will not be described again.

Claims

1. A method for automated fusion and docking of heterogeneous data from multiple systems, characterized in that, include: It sends sniffing packets to multiple data source systems, identifies the protocol type of different data sources based on the information entropy distribution of data frames, and outputs the protocol data stream. Extract the semantic features of the fields in the protocol data stream and embed them into a unified vector space. Calculate the cross-system semantic similarity based on the system domain type to generate a mapping relationship and detect semantic conflicts in the mapping process. When the predicted conflict probability is greater than the probability threshold, generate a compensation plan based on the data source quality. Align the data source fields according to the mapping relationship, determine the confidence weight according to the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, execute the corresponding compensation plan and then perform weighted fusion to obtain the fusion result; otherwise, directly perform weighted fusion and update the predicted conflict probability. The deviation between the quality index of the fusion result and the preset business constraints is calculated, and the deviation is converted into the adjustment amount for semantic similarity calculation. The similarity threshold and conflict prediction time window are then adjusted accordingly.

2. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 1, characterized in that, The output protocol data stream includes: The sniffing packets are sent to multiple data source systems and the returned data frames are obtained. After the data frames are fragmented and reassembled, they are divided into feature segments according to the frame header identifier, data segment length and check bit distribution. The information entropy of different feature segments is calculated to determine the information entropy distribution. The information entropy distribution is matched with a preset protocol entropy feature library, and the protocol type is identified by dynamic threshold clustering. Feature segments whose information entropy deviates from the standard entropy range are marked as extended feature segments. The extended feature segments are organized according to the protocol type and concatenated with the standard feature segments corresponding to the protocol type. The concatenated protocol data is then sorted by field normalization according to the protocol semantic level, and the protocol data stream is output.

3. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 2, characterized in that, The generated mapping relationship includes: Extract the field semantic features of the standard feature segments and extended feature segments in the protocol data stream, and embed them into a unified vector space to obtain the standard semantic vector and extended semantic vector; Based on the system domain type, a cross-system domain weight matrix is ​​constructed. The standard semantic vector and the extended semantic vector are weighted by the domain weight matrix to obtain the semantic similarity between cross-system fields. Semantic similarity is clustered and matched, and valid results are filtered by combining similarity thresholds to generate initial mapping relationships. The initial mapping relationships are then corrected by combining the confidence of protocol type to obtain the final mapping relationship.

4. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 3, characterized in that, Detecting the semantic conflict includes: Semantic deviation features and protocol format adaptation features are extracted from standard semantic vectors and extended semantic vectors. Conflict features are obtained by weighted fusion of semantic deviation features and protocol format adaptation features based on domain weight matrix and protocol type confidence. The conflict characteristics of multiple consecutive mapping relationships are statistically analyzed using a conflict prediction time window. The dispersion of the conflict characteristics within the conflict prediction time window is calculated, and the ratio of the dispersion to a preset benchmark value is used as the conflict probability.

5. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 4, characterized in that, The generated compensation plan includes: When the probability of conflict exceeds the probability threshold, the conflict type is classified according to the dimensional ratio of semantic deviation features and protocol format adaptation features, and the conflict level is labeled by associating the discreteness of the corresponding conflict features. The quality score of the data source is determined based on data integrity, temporal stability and field consistency. The quality score is then weighted by a domain weight matrix to obtain the compensation priority for different data sources. Configure a compensation plan that includes type identifier, compensation priority, and execution time, based on the conflict type, conflict level, and compensation priority.

6. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 5, characterized in that, The obtained fusion results include: Align the corresponding fields of standard feature segments and extended feature segments between different data sources according to the mapping relationship, and determine the credibility weight of the fields based on the historical consistency of the data sources. If the semantic conflict matches the type identifier in the compensation plan, and the execution time is within the conflict prediction time window, then the corresponding compensation operation will be executed according to the compensation priority. The compensated standard feature segments and extended feature segments are weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result.

7. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 6, characterized in that, The updated predicted conflict probability includes: If the semantic conflict does not match the type identifier in the compensation plan, or the execution time exceeds the conflict prediction time window, the standard feature segments and extended feature segments between different data sources are directly weighted and fused according to the confidence weight of the corresponding fields to obtain the fusion result. The actual deviation values ​​of the fields during this weighted fusion process are extracted, and the predicted conflict probability is updated by combining the dispersion of conflict features within the conflict prediction time window.

8. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 7, characterized in that, The feedback adjustment similarity threshold includes: The actual deviation value of the field is used as the quality index of the fusion result. The deviation between the quality index and the preset business constraints is calculated, and the deviation is converted into the adjustment amount for semantic similarity calculation. The adjustment amount is corrected based on the dispersion of conflict features within the conflict prediction time window, and the similarity threshold is adjusted based on the corrected adjustment amount.

9. The method for automated fusion and docking of heterogeneous data from multiple systems as described in claim 8, characterized in that, The feedback adjustment of the conflict prediction time window includes: The cumulative amount of actual deviation values ​​of fields within the current conflict prediction time window is statistically analyzed, and the window deviation adaptation coefficient is calculated in combination with the dispersion of conflict characteristics. The window bias adaptation coefficient is weighted and corrected based on the domain weight matrix to distinguish the conflict response differences between the standard feature segment and the extended feature segment, and to generate the scaling ratio of the corresponding conflict prediction time window. The duration of the conflict prediction time window is dynamically adjusted according to the scaling ratio.

10. A multi-system heterogeneous data automated fusion and docking system, used to implement the multi-system heterogeneous data automated fusion and docking method according to any one of claims 1-9, characterized in that, include: The module consists of a perception module, a mapping module, a fusion module, and a feedback module. The perception module is used to send sniffing packets to multiple data source systems, identify the protocol type of different data sources based on the information entropy distribution of data frames, and output the protocol data stream. The mapping module is used to extract the semantic features of the fields of the protocol data stream and embed them into a unified vector space. It calculates the semantic similarity across systems based on the system domain type to generate mapping relationships and detects semantic conflicts in the mapping process. When the predicted conflict probability is greater than the probability threshold, a compensation plan is generated based on the data source quality. The fusion module is used to align data source fields according to the mapping relationship, determine the confidence weight based on the historical consistency of the data source, and when the semantic conflict matches the type identifier in the compensation plan, the corresponding compensation plan is executed and then weighted fusion is performed to obtain the fusion result; otherwise, the weighted fusion is performed directly and the predicted conflict probability is updated. The feedback module is used to calculate the deviation between the quality index of the fusion result and the preset business constraints, convert the deviation into the adjustment amount for semantic similarity calculation, and feed back the adjustment similarity threshold and conflict prediction time window.

Images