Distributed data integration method and system for multi-source device protocols

Abstract

The present invention discloses a distributed data integration method and system for multi-source device protocols, and relates to the field of data processing technology. The method comprises: connecting a protocol adaptation layer to perform data protocol adaptation and analysis of multi-source protocol data, and outputting conversion data; fusing distributed data in the conversion data to obtain monitoring data; obtaining abnormal warnings by detecting abnormalities in the monitoring data; performing data correction on the abnormal warnings, combining with normal data in the monitoring data, and encrypting and integrating the corrected data to obtain distributed integration results; constructing a distributed architecture with distributed data and distributed integration results, and storing and managing data. The method solves the technical problem of low data integration efficiency caused by incompatibility of multi-source device protocols in the prior art, and achieves seamless connection between different device protocols through protocol adaptation and analysis of the protocol adaptation layer, and fusion of conversion data, thereby achieving the technical effect of improving data integration efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a distributed data integration method and system oriented to a multi-source device protocol. Background Art

[0002] The large amount of data generated by multi-source devices presents the characteristics of diverse protocols, different data formats, and non-uniform transmission methods. In the existing technology, the integration, analysis and processing of data from different devices and systems often face problems such as difficult protocol adaptation and incompatible data formats, which leads to low efficiency and poor accuracy of data fusion, and great technical difficulties in anomaly detection and data correction. Especially in application scenarios, how to process these multi-source data in real time and ensure the accuracy and reliability of monitoring data has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a distributed data integration method and system for multi-source device protocols, which solves the technical problem in the prior art that the incompatibility of multi-source device protocols leads to low data integration efficiency.

[0004] In a first aspect of the present application, a distributed data integration method for a multi-source device protocol is provided, the method comprising: The connection protocol adaptation layer performs data protocol adaptation and analysis of multi-source protocol data, and outputs conversion data; integrates the distributed data in the conversion data to obtain monitoring data; obtains abnormal warnings by detecting abnormalities in the monitoring data; performs data correction on the abnormal warnings, combines the normal data in the monitoring data, and encrypts and integrates the corrected data to obtain distributed integration results; constructs a distributed architecture with the distributed data and the distributed integration results, and stores and manages the data.

[0005] A second aspect of the present application provides a distributed data integration system for a multi-source device protocol, the system comprising: A parsing module is used to connect to the protocol adaptation layer to perform data protocol adaptation and parsing of multi-source protocol data, and output conversion data; a data fusion module is used to fuse the distributed data in the conversion data to obtain monitoring data; an anomaly detection module is used to obtain an abnormal warning by detecting anomalies in the monitoring data; an integration module is used to perform data correction on the abnormal warning, combine the normal data in the monitoring data, and encrypt and integrate the corrected data to obtain a distributed integration result; a storage module is used to construct a distributed architecture with the distributed data and the distributed integration result, and store and manage data.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages: First, the protocol adaptation layer is connected to perform data protocol adaptation and analysis of multi-source protocol data, and the conversion data is output. Then, the distributed data in the conversion data is integrated to obtain monitoring data. Furthermore, anomaly detection of the monitoring data is performed to obtain an abnormal warning. Then, data correction is performed on the abnormal warning, and the corrected data is encrypted and integrated with the normal data in the monitoring data to obtain a distributed integration result. Finally, a distributed architecture is constructed with the distributed data and the distributed integration results, and the data is stored and managed. The technical problem of low data integration efficiency caused by incompatibility of multi-source device protocols in the prior art is solved. Through protocol adaptation and analysis of the protocol adaptation layer, as well as fusion and conversion data, seamless connection between different device protocols is achieved, achieving the technical effect of improving data integration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 A schematic flow chart of a distributed data integration method for a multi-source device protocol provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a distributed data integration system for a multi-source device protocol provided in an embodiment of the present application.

[0009] Explanation of the reference numerals: analysis module 11 , data fusion module 12 , anomaly detection module 13 , integration module 14 , storage module 15 . DETAILED DESCRIPTION

[0010] The present application solves the technical problem in the prior art of low data integration efficiency caused by incompatibility of multi-source device protocols by providing a distributed data integration method and system for multi-source device protocols.

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0012] It should be noted that the terms "including" and "having" are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or are inherent to these processes, methods, products or devices.

[0013] Embodiment 1, as Figure 1 As shown, the present application provides a distributed data integration method for a multi-source device protocol, wherein the method includes: The connection protocol adaptation layer performs data protocol adaptation and analysis of multi-source protocol data and outputs converted data.

[0014] In the process of multi-source device data processing, different devices may use different communication protocols (such as Modbus, MQTT, HTTP, etc.), and there are differences between these protocols, including data format, transmission method, encoding rules, etc. The main task of the protocol adaptation layer is to adapt and parse the data protocols from multiple devices or systems and output the converted data in a unified format. Specifically, the protocol adaptation layer converts the data of different devices into a format that the system can understand and process through a unified protocol parsing mechanism. For example, device A may send data encapsulated in JSON format, while device B uses XML format. The protocol adaptation layer will parse this data through defined rules and methods (such as JSON parser, XML parser) to extract useful information.

[0015] Furthermore, the connection protocol adaptation layer performs data protocol adaptation and parsing of multi-source protocol data and outputs conversion data, including: The multi-source protocol data is received through the protocol adaptation layer; a protocol adapter is dynamically loaded with the multi-source protocol data to obtain an independent adapter, wherein the independent adapter corresponds to the multi-source protocol data one by one; the data format of the multi-source protocol data is parsed according to the independent adapter to obtain a parsed format; the parsed format is format-converted by a protocol conversion engine in the protocol adaptation layer to obtain a standard format; the standard format is encapsulated in a message format in combination with metadata obtained from the multi-source protocol data, and the converted data is output, wherein a mapping relationship exists between the converted data and the multi-source protocol data.

[0016] In the process of connecting the protocol adaptation layer to perform protocol adaptation and parsing of multi-source protocol data and output conversion data, first, the protocol adaptation layer receives multi-source protocol data from different devices or systems, which may use different protocol formats, such as JSON, XML, Modbus, etc. The protocol adaptation layer needs to dynamically load the corresponding protocol adapter according to the data type received, where each protocol has a special adapter, which corresponds to the protocol data one by one and can parse the data format of the protocol. When the protocol adapter is loaded, the adapter begins to parse the multi-source protocol data. The parsing process depends on the specific format of different protocols. For example, the adapter of the JSON protocol will extract key-value pairs, and the adapter of the XML protocol will parse the tags and node content; the parsed data will be converted into an intermediate format, usually a structured data representation, for subsequent processing. Next, the protocol conversion engine in the protocol adaptation layer will convert the format of the parsed data and convert it into a standard format. This standard format is usually a format that the platform can understand and process, such as a unified JSON format or a database table structure. The task of the protocol conversion engine is to map the parsed data to a standard format according to predefined rules to ensure that the entire system can uniformly process data from different protocols. Then, the protocol adaptation layer will combine the metadata in the original data (such as timestamp, device ID, data source, data type, etc.) to encapsulate the data in the standard format. The encapsulated data usually adopts a specific message format, such as Protocol Buffers or Avro, etc., to ensure the efficiency and accuracy of the data flow, storage and subsequent processing in the system. Finally, the converted and encapsulated standard data is used as output to form the conversion data. There is a clear mapping relationship between these conversion data and the original multi-source protocol data, which ensures the consistency and integrity of the data.

[0017] The distributed data in the converted data is integrated to obtain monitoring data.

[0018] The data received and converted into a standard format may contain distributed data from different devices, sensors or systems. These distributed data usually have different timestamps and measurement methods, so they need to be fused to form complete monitoring data.

[0019] The process of data fusion is to integrate these scattered data to form complete monitoring data. This fusion usually includes the time alignment and spatial alignment of the data and the synthesis of data according to specific algorithms (such as weighted average, least squares method, etc.) to ensure the integrity and accuracy of the data.

[0020] Furthermore, the distributed data in the converted data is integrated to obtain monitoring data, including: The conversion data is received by using a message queue, wherein the conversion data includes distributed data; the distributed data is standardized by using a distributed data stream processing framework, and weighted average is performed to obtain the monitoring data.

[0021] The message queue can ensure that the system can stably receive and process each piece of data when facing data sources of different speeds, and pass the data in the correct order to the subsequent processing module; the message queue is used to receive the converted data, which contains distributed data from different devices or systems. Once the data is received through the message queue, the system will use the distributed data stream processing framework to standardize the received distributed data so that the data from different devices, sensors or systems can be converted into a unified format and metric standard, including unit unification, timestamp alignment, missing data filling and other operations to ensure that data from different data sources can be compared and integrated under the same standard. On the basis of standardization, these distributed data are weighted averaged. The weighted average is to assign different weights to each data source according to the reliability, accuracy or other relevant factors of each data source; through weighted average, the high reliability data source can have a greater impact on the final result, while the low reliability or unstable data source has a smaller impact. Through this series of processing, the final monitoring data not only integrates the information from different data sources, but also ensures the improvement and accuracy of data quality.

[0022] Furthermore, the distributed data is obtained, including: Initial distributed data is collected under distributed clocks; the initial distributed data is processed for consistency by a consistency protocol to obtain the distributed data; wherein the consistency processing includes obtaining a distributed copy by asynchronous replication; and copy synchronization is triggered when data on the distributed copy changes.

[0023] Initial distributed data is collected through distributed clocks. In a distributed system, multiple nodes may be distributed in different physical locations, and they need to maintain time consistency. Distributed clocks provide each node with a common time reference to ensure that the data collected by these nodes can be synchronized under a unified time standard. In this way, when the data of each node is collected, they can use the same timestamp to avoid data inconsistency caused by time differences. Through this mechanism, the data collected on all devices or nodes can be regarded as based on the same time point, which is crucial for subsequent data integration and processing. Next, the system will use the consistency protocol to process the initial distributed data consistently; since there may be multiple copies of data in a distributed system, each copy may be out of sync or inconsistent due to network delays, device failures, or data write conflicts. The consistency protocol is used to ensure that the data in all copies is consistent. The consistency protocol formulates rules to ensure that when data is updated between different nodes, the status of all copies can be guaranteed to be consistent. For example, using consistency algorithms such as Paxos or Raft, the system can synchronize and verify data on multiple nodes to ensure the final consistency of all copies.

[0024] Specifically, a core mechanism for consistency processing is asynchronous replication, which is a commonly used distributed data replication method. When data is updated on the master node, the system does not immediately synchronize these updates to all replicas, but gradually propagates the data changes to other replicas through asynchronous replication. This means that the data update of the master node will be completed first, and the data changes will be gradually replicated to other replicas in the background. Asynchronous replication improves the efficiency of the system by reducing the blocking of the master node, but data inconsistencies may occur between replicas for a certain period of time. When the system detects that the data has changed, it will trigger replica synchronization to ensure that the data between distributed replicas is updated in a timely manner; when the data changes, the system will automatically trigger the replica synchronization mechanism to ensure that the data of all replicas is updated consistently; this synchronization process can ensure that when the content of a replica changes, other replicas can also receive the change information in a timely manner and make corresponding data updates.

[0025] Furthermore, obtaining the distributed clock includes: Timestamps are exchanged based on distributed devices to obtain a shared distributed clock; a local clock of the distributed device is combined to calculate a drift difference with the shared distributed clock; an incremental correction is performed on the local clock according to the drift difference to obtain the distributed clock.

[0026] In the process of obtaining distributed clocks, first, the system obtains a shared distributed clock by exchanging timestamps based on distributed devices. In a distributed system, since each device or node may have an independent local clock, these local clocks may have certain deviations or errors. In order to ensure that each node in the system can maintain consistency in time, the system allows different devices to exchange their own time information through timestamp exchange. Through these exchanges, devices can calibrate each other to obtain a shared, unified distributed clock. This shared clock provides a consistent time reference for all devices in the system, so that data collection at different nodes can be synchronized based on the same time reference. Next, the system combines the local clocks of each distributed device to calculate the drift difference between the local clock and the shared distributed clock. Due to hardware differences, temperature fluctuations, network delays, etc., the local clock of each device may have a certain time drift with the shared distributed clock. In order to ensure time synchronization, the system needs to detect and calculate this drift difference. Usually, the system uses a series of algorithms, such as time synchronization protocols (such as NTP or PTP), to calculate the time difference between the local clock and the shared clock. Finally, the system incrementally corrects the local clock based on the calculated drift difference to obtain an accurate distributed clock. The correction process usually involves gradually adjusting the local clock to reduce the gap with the shared clock. Incremental correction is not a one-time adjustment, but rather multiple small adjustments to gradually bring the local clock closer to the shared clock, ensuring the accuracy of time synchronization.

[0027] By detecting the abnormality of the monitoring data, an abnormality warning is obtained.

[0028] After the distributed data fusion is completed and the monitoring data is formed, the system will perform anomaly detection on these monitoring data to identify potential anomalies and generate anomaly warnings. Specifically, the system will perform preliminary screening of the monitoring data based on the set threshold rules or historical data baselines; the threshold rules are usually pre-defined normal ranges. For example, in the temperature monitoring system, the temperature is considered abnormal when it exceeds a certain upper limit or is lower than a certain lower limit; and the historical data baseline determines the fluctuation range of normal data by analyzing the previous data distribution, thereby identifying data that deviates from the normal fluctuation range. Subsequently, the system will apply statistical analysis methods or machine learning algorithms for deeper anomaly detection. Statistical methods such as standard deviation analysis and Z-score can effectively detect extreme values ​​or abnormal fluctuations; while machine learning-based detection methods, such as cluster analysis (K-means), isolation forests, or time series-based models (such as ARIMA, LSTM), can identify anomalies in complex data patterns that are difficult to detect with simple rules. Once an anomaly is detected, the system will generate an exception warning with detailed exception information, such as the type of anomaly, time of occurrence, scope of impact, and related data indicators. These exception warnings can be displayed through real-time notifications, log records, or visual dashboards to help operation and maintenance personnel quickly locate problems.

[0029] Furthermore, by detecting anomalies in the monitoring data, an abnormality warning is obtained, including: The monitoring data includes a data stream obtained based on sliding window monitoring; setting an alarm rule, wherein the alarm rule includes an upper threshold and a lower threshold corresponding to the distributed device; identifying the data stream with the upper threshold and the lower threshold to obtain out-of-limit data; performing anomaly identification on the out-of-limit data based on an isolation forest, and obtaining the anomaly warning by triggering an identification mark.

[0030] In the process of obtaining abnormal warnings by performing anomaly detection on monitoring data, the monitoring data processed by the system is based on the data stream obtained by sliding window monitoring. Sliding window is a technology commonly used in streaming data processing. It divides the continuous data stream into fixed-size time windows or data fragments for real-time analysis. Each sliding window contains data within a certain time range. By continuously sliding forward, the system can continuously monitor the latest data changes and adapt to the characteristics of data streams in dynamic environments. Next, the system will set alarm rules, which are the basis for anomaly detection; the alarm rules define the upper and lower thresholds corresponding to distributed devices to determine whether the data exceeds the normal range; these thresholds can be set based on historical data analysis, expert experience or system operation requirements, with the aim of quickly screening out data fluctuations that do not meet expectations. For example, in a temperature monitoring system, temperatures exceeding 80°C or below 0°C may be defined as abnormal. After the alarm rules are set, the system will use these upper and lower thresholds to identify the data stream in real time to detect whether there is over-limit data; when a data point or a piece of data in the data stream exceeds the preset threshold range, it will be marked as over-limit data.

[0031] In order to further improve the accuracy of anomaly detection, the system will conduct in-depth anomaly identification of over-limit data based on the Isolation Forest algorithm. Isolation Forest is an anomaly detection algorithm based on a tree structure. Its core idea is to measure the difficulty of data points being "isolated" by randomly partitioning data. Specifically, the system will randomly select features and eigenvalues ​​as partitioning conditions and construct multiple random binary trees. Each tree will partition the data layer by layer until each data point is isolated. In this process, the path length required for each data point to be isolated is recorded. Generally, abnormal data can be quickly isolated in fewer partitioning steps due to its large difference from other data, so its path length is short; while normal data requires more partitioning steps to be completely isolated due to its close distribution to other data, and the path length is relatively long. Next, the system will calculate the anomaly score of each data point based on the results of all random trees. The anomaly score reflects the probability of a data point being isolated in the entire forest, and the score range is usually between 0 and 1. The closer the score is to 1, the more likely the data point is to be an outlier; the closer the score is to 0, the closer the data point is to the distribution of normal data; the system can determine which data points need to be marked as abnormal based on preset thresholds; once abnormal data is identified, the system will generate an abnormal warning, indicating the specific information of the abnormal data, including abnormal score, abnormal type, data source and related timestamps.

[0032] After the abnormality is identified, the system will generate an abnormality warning based on the identification results, with detailed abnormality tag information. The abnormality warning usually includes the specific value of the abnormal data, the time of occurrence, the corresponding device ID, the abnormality type (such as over-limit or isolated abnormality), etc. These warnings can be pushed to the operation and maintenance personnel through the real-time notification system, or automatically trigger the predefined emergency response mechanism to quickly respond to potential system risks or failures.

[0033] The abnormal warning is corrected, and the corrected data is combined with the normal data in the monitoring data to perform encryption integration to obtain a distributed integration result.

[0034] In the process of data correction for the abnormal warning, the system will first receive the abnormal warning generated by the abnormal detection algorithm, which contains detailed information of the abnormal data, such as the time when the abnormality occurred, the source of the data, the type of abnormality, and the abnormality score. The abnormal data may be generated due to various reasons such as equipment failure, network delay, data transmission error, or environmental interference; the system will carry out data correction based on the detailed information contained in the abnormal warning, such as the type of abnormality, time of occurrence, source of data, and abnormality score, combined with the normal data in the historical data and monitoring data.

[0035] In the initial stage of data correction, the system will diagnose the abnormal data and determine what type of abnormality it belongs to. Generally, abnormalities can be divided into isolated abnormalities and systematic abnormalities; isolated abnormalities refer to occasional data abnormalities, which are usually not persistent and may be caused by transient noise or short-term failures; while systematic abnormalities are manifested as data that continuously deviates from the normal range over a period of time, which is usually related to hardware failures, system configuration errors, or abnormal data transmission links.

[0036] In order to achieve effective correction, the system will combine the normal data in the monitoring data as the benchmark for data correction. Normal data refers to data that is not marked as abnormal during the anomaly detection process. These data usually have stable patterns and reliable reference value. The system will analyze the position of abnormal data in normal data, consider the continuity of time series, the trend of data fluctuations, and the correlation between multi-source data, and use appropriate algorithms to correct abnormal data; common data correction methods include linear interpolation, polynomial fitting, and prediction models based on machine learning; linear interpolation estimates the reasonable value of abnormal data by connecting normal data points before and after abnormal data; polynomial fitting can capture the nonlinear change trend of data and smooth data with large fluctuations; and prediction models based on machine learning can use the patterns of historical data to intelligently predict and correct abnormal data.

[0037] After completing the data correction, the system will encrypt and integrate the corrected data with the normal data in the monitoring data to ensure the integrity and security of the data. The core purpose of encryption integration is to prevent data from being tampered with or leaked during transmission and storage, while ensuring the consistency and reliability of data in a distributed environment. Encryption technology usually includes symmetric encryption, asymmetric encryption and hash algorithms. Symmetric encryption (such as AES) uses the same key for encryption and decryption, which is suitable for fast encryption of large-scale data; asymmetric encryption (such as RSA) uses public and private key pairs, which is suitable for data transmission scenarios with higher security requirements; hash algorithms are used to generate unique fingerprints of data to ensure that the integrity of the data is not tampered with. In the process of encryption integration, the system does not simply merge the data, but generates the final distributed integration results through data encryption, integrity verification and consistency verification. The distributed integration result refers to the unified data set formed by the system in the distributed architecture after data correction and encryption processing. These data are distributed and stored on multiple nodes to support high availability and fault tolerance, ensuring that the data can still maintain consistency and availability when any node fails.

[0038] A distributed architecture is constructed using the distributed data and the distributed integration results, and data is stored and managed.

[0039] After completing data anomaly detection, data correction, and encryption integration, the system will build a distributed architecture based on the obtained distributed data and distributed integration results, and store and manage the data. Distributed architecture is a system design pattern that can support large-scale data processing, high concurrent access, and fault tolerance.

[0040] The construction of distributed architecture is based on distributed data and distributed integration results; distributed data refers to data from different devices, nodes or systems, which are physically dispersed and have the characteristics of multi-source heterogeneity; and the distributed integration results are high-quality data sets formed after abnormal detection, correction and encryption integration of data, which represent the final results of data processing. In the process of building a distributed architecture, the system will use distributed storage systems such as HDFS (Hadoop Distributed File System), Cassandra, MongoDB, etc. These storage systems can store data fragments on different physical nodes and ensure high availability and fault tolerance of data through data replication mechanisms. For example, HDFS divides data into multiple blocks and stores copies on different nodes to prevent data loss caused by single point failures. Cassandra uses a distributed hashing algorithm to evenly distribute data on each node to achieve fast data reading and writing and high scalability.

[0041] While storing data, the system also needs to achieve efficient data management, including key functions such as data consistency management, data backup and recovery, access control, and data lifecycle management. Data consistency management refers to ensuring that data on different nodes remain consistent. Common consistency models include strong consistency, eventual consistency, and weak consistency. The system will select the appropriate model based on business needs. The data backup and recovery mechanism ensures that data can be quickly restored in the event of a system failure to ensure business continuity. Access control ensures data security and prevents unauthorized access through user authority management, identity authentication, and other methods. Data lifecycle management is responsible for the management of the entire process of data creation, storage, archiving, and deletion, ensuring that data is reasonably processed and stored at different stages.

[0042] Furthermore, constructing a distributed architecture using the distributed data and the distributed integration result includes: The distributed storage architecture is partitioned by the distributed devices corresponding to the distributed data to obtain an initial distributed architecture; the initial distributed architecture is partitioned by taking the distributed integration result as a sharding guide to obtain the distributed architecture.

[0043] Based on the distributed devices corresponding to the distributed data, the distributed storage architecture is partitioned to obtain the initial distributed architecture. Partitioning refers to dividing large-scale data into multiple logical areas, each of which corresponds to a specific distributed device or data source. This device-based partitioning strategy can effectively bind data to its collection source, ensure that the physical location of the data matches the device topology, and thus improve the efficiency of data storage and access. After the partitioning is completed, the system will further shard the initial distributed architecture based on the distributed integration results, and finally form a complete distributed architecture; sharding refers to further refining the data on the basis of partitioning and dividing it into smaller independent data units, and each data shard can be stored and processed independently. Different from partitioning, sharding not only focuses on the physical distribution of data, but also emphasizes the logical division of data, usually based on data content, such as sharding based on user ID, timestamp or geographic location. During the sharding process, the system will determine the rules and strategies for sharding based on the data characteristics in the distributed integration results. For example, if there is a large amount of time series data in the distributed integration results, the system may adopt a sharding strategy based on time range to divide the data into different shards according to time periods; if the data has obvious regional characteristics, it can be sharded according to geographical regions to reduce the delay caused by cross-regional data access. The guiding role of sharding is reflected in its ability to dynamically adjust the granularity and distribution of shards according to the actual usage scenarios and access patterns of the data to optimize data processing performance. After the sharding is completed, the system will establish a mapping relationship between the shards and the storage nodes to ensure that the data of each shard can be efficiently allocated to the appropriate storage nodes. This mapping relationship can be achieved through technologies such as distributed hashing algorithms, range sharding algorithms or consistent hashing to ensure the balance and scalability of data distribution. Finally, after partitioning and sharding processing, the distributed architecture built by the system not only has efficient data storage and management capabilities, but also can provide good performance and reliability when facing large-scale data, high concurrent access and complex computing tasks. This architecture supports fast reading, real-time analysis and dynamic expansion of distributed data, providing a powerful infrastructure for data-driven intelligent decision-making and business optimization.

[0044] In summary, the embodiments of the present application have at least the following technical effects: First, the protocol adaptation layer is connected to perform data protocol adaptation and analysis of multi-source protocol data, and the conversion data is output. Then, the distributed data in the conversion data is integrated to obtain monitoring data. Furthermore, anomaly detection of the monitoring data is performed to obtain an abnormal warning. Then, data correction is performed on the abnormal warning, and the corrected data is encrypted and integrated with the normal data in the monitoring data to obtain a distributed integration result. Finally, a distributed architecture is constructed with the distributed data and the distributed integration results, and the data is stored and managed. The technical problem of low data integration efficiency caused by incompatibility of multi-source device protocols in the prior art is solved. Through protocol adaptation and analysis of the protocol adaptation layer, as well as fusion and conversion data, seamless connection between different device protocols is achieved, achieving the technical effect of improving data integration efficiency.

[0045] Embodiment 2 is based on the same inventive concept as the distributed data integration method for multi-source device protocols in the above embodiment. Figure 2 As shown, the present application provides a distributed data integration system for multi-source device protocols, wherein the system includes: The parsing module 11 is used to connect the protocol adaptation layer to perform data protocol adaptation and parsing of multi-source protocol data, and output conversion data; the data fusion module 12 is used to fuse the distributed data in the conversion data to obtain monitoring data; the anomaly detection module 13 is used to obtain anomaly warnings by detecting anomalies in the monitoring data; the integration module 14 is used to perform data correction on the anomaly warnings, combine the normal data in the monitoring data, and encrypt and integrate the corrected data to obtain a distributed integration result; the storage module 15 is used to construct a distributed architecture with the distributed data and the distributed integration results, and store and manage data.

[0046] Furthermore, the parsing module 11 is used to execute the following method: The multi-source protocol data is received through the protocol adaptation layer; a protocol adapter is dynamically loaded with the multi-source protocol data to obtain an independent adapter, wherein the independent adapter corresponds to the multi-source protocol data one by one; the data format of the multi-source protocol data is parsed according to the independent adapter to obtain a parsed format; the parsed format is format-converted by a protocol conversion engine in the protocol adaptation layer to obtain a standard format; the standard format is encapsulated in a message format in combination with metadata obtained from the multi-source protocol data, and the converted data is output, wherein a mapping relationship exists between the converted data and the multi-source protocol data.

[0047] Furthermore, the data fusion module 12 is used to perform the following method: The conversion data is received by using a message queue, wherein the conversion data includes distributed data; the distributed data is standardized by using a distributed data stream processing framework, and weighted average is performed to obtain the monitoring data.

[0048] Furthermore, the anomaly detection module 13 is used to perform the following method: The monitoring data includes a data stream obtained based on sliding window monitoring; setting an alarm rule, wherein the alarm rule includes an upper threshold and a lower threshold corresponding to the distributed device; identifying the data stream with the upper threshold and the lower threshold to obtain out-of-limit data; performing anomaly identification on the out-of-limit data based on an isolation forest, and obtaining the anomaly warning by triggering an identification mark.

[0049] Furthermore, the parsing module 11 is used to execute the following method: Initial distributed data is collected under distributed clocks; the initial distributed data is processed for consistency by a consistency protocol to obtain the distributed data; wherein the consistency processing includes obtaining a distributed copy by asynchronous replication; and copy synchronization is triggered when data on the distributed copy changes.

[0050] Furthermore, the parsing module 11 is used to execute the following method: Timestamps are exchanged based on distributed devices to obtain a shared distributed clock; a local clock of the distributed device is combined to calculate a drift difference with the shared distributed clock; an incremental correction is performed on the local clock according to the drift difference to obtain the distributed clock.

[0051] Furthermore, the storage module 15 is used to execute the following method: The distributed storage architecture is partitioned by the distributed devices corresponding to the distributed data to obtain an initial distributed architecture; the initial distributed architecture is partitioned by taking the distributed integration result as a sharding guide to obtain the distributed architecture.

[0052] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0053] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0054] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A distributed data integration method for multi-source device protocols, characterized in that: The method comprises: Connect the protocol adaptation layer to perform data protocol adaptation and analysis of multi-source protocol data and output conversion data; fusing the distributed data in the converted data to obtain monitoring data; Obtaining an abnormality warning by detecting an abnormality of the monitoring data; Performing data correction on the abnormal warning, combining with the normal data in the monitoring data, and encrypting and integrating the corrected data to obtain a distributed integration result; A distributed architecture is constructed using the distributed data and the distributed integration results, and data is stored and managed.

2. The distributed data integration method for multi-source device protocols according to claim 1, characterized in that: The connection protocol adaptation layer performs data protocol adaptation and parsing of multi-source protocol data and outputs conversion data, including: Receiving the multi-source protocol data through the protocol adaptation layer; The protocol adapter is dynamically loaded with the multi-source protocol data to obtain an independent adapter, wherein the independent adapter corresponds to the multi-source protocol data one by one; Parsing the data format of the multi-source protocol data by the independent adapter to obtain a parsed format; Performing format conversion on the parsed format by a protocol conversion engine in the protocol adaptation layer to obtain a standard format; The metadata obtained in combination with the multi-source protocol data is encapsulated in a message format to output the converted data, wherein a mapping relationship exists between the converted data and the multi-source protocol data.

3. The distributed data integration method for multi-source device protocols as claimed in claim 1, characterized in that: The distributed data in the converted data is integrated to obtain monitoring data, including: Using a message queue to receive the conversion data, wherein the conversion data includes distributed data; The distributed data is standardized using a distributed data stream processing framework, and weighted averaged to obtain the monitoring data.

4. The distributed data integration method for multi-source device protocols as claimed in claim 1, characterized in that: By detecting anomalies in the monitoring data, an abnormality warning is obtained, including: The monitoring data includes a data stream obtained based on sliding window monitoring; Setting alarm rules, wherein the alarm rules include upper and lower thresholds corresponding to distributed devices; Identify the data flow using the upper threshold and the lower threshold to obtain over-limit data; The out-of-limit data is identified as abnormal based on the isolation forest, and the abnormal warning is obtained by triggering the identification mark.

5. The distributed data integration method for multi-source device protocols as claimed in claim 1, characterized in that: Get distributed data, including: Initial distributed data is acquired under distributed clocks; Performing consistency processing on the initial distributed data using a consistency protocol to obtain the distributed data; Wherein, the consistency processing includes obtaining a distributed copy through asynchronous replication; The replica synchronization is triggered when data changes are made to the distributed replica.

6. The distributed data integration method for multi-source device protocols as claimed in claim 5, characterized in that: Get distributed clocks, including: Exchange timestamps based on distributed devices to obtain shared distributed clocks; In combination with the local clock of the distributed device, calculating the drift difference with the shared distributed clock; The local clock is incrementally corrected according to the drift difference to obtain the distributed clock.

7. The distributed data integration method for multi-source device protocols as claimed in claim 1, characterized in that: Constructing a distributed architecture with the distributed data and the distributed integration result, including: Partitioning the distributed storage architecture with distributed devices corresponding to the distributed data to obtain an initial distributed architecture; The distributed integration result is used as a sharding guide to shard the initial distributed architecture to obtain the distributed architecture.

8. A distributed data integration system for multi-source device protocols, characterized in that: A distributed data integration method for a multi-source device protocol for implementing any one of claims 1 to 7, the system comprising: The parsing module is used to connect to the protocol adaptation layer to perform data protocol adaptation and parsing of multi-source protocol data and output conversion data; A data fusion module, used for fusing the distributed data in the converted data to obtain monitoring data; An anomaly detection module, used to obtain an anomaly warning by detecting anomalies in the monitoring data; An integration module is used to correct the abnormal warning data, combine the normal data in the monitoring data, and perform encryption integration with the corrected data to obtain a distributed integration result; The storage module is used to construct a distributed architecture with the distributed data and the distributed integration results, and to store and manage data.

Images