A multi-element data fusion and unified data processing method

Abstract

Embodiments of the present disclosure provide a multi-element data fusion and unified data processing method. The method comprises: an Internet of Things gateway adaptively collecting sensing device data and performing matching filtering on the collected data; the Internet of Things gateway performing data fusion on the matching filtered data, and performing real-time analysis and calculation and data persistent access; and the Internet of Things gateway transmitting the fused data to an Internet of Things cloud platform through a unified application layer protocol. In this way, unified scheduling and processing of multi-source heterogeneous data can be realized through active power distribution network comprehensive edge agent collaborative control.

Description

Technical Field

[0001] This disclosure relates to the field of the Internet of Things (IoT), and in particular to the field of data processing technology for the fusion and unification of diverse IoT data. Background Technology

[0002] With the rapid development of IoT technology, the demand for data collection from various smart terminals is constantly increasing. Traditional IoT gateway program development mainly focuses on specific business scenarios and data protocols. Because various smart devices have different data transmission protocols and formats, if the device, protocol, or business scenario changes, corresponding gateway program modifications are required. Data collection for each device requires targeted adaptation development, making data collection devices unreusable and increasing the complexity of secondary development; the development cycle and difficulty are both significant. The increase in data collection volume also leads to increased data transmission bandwidth and adds pressure to the cloud. Summary of the Invention

[0003] This disclosure provides a method for the fusion and unified processing of multi-source Internet of Things (IoT) data.

[0004] According to a first aspect of this disclosure, a method for multi-source Internet of Things (IoT) data fusion and unified data processing is provided. The method includes: an IoT gateway adaptively collecting data from sensing devices and performing matching and filtering on the collected data; the IoT gateway fusing the matched and filtered data, performing real-time analysis and computation, and persistently storing the data; and the IoT gateway transmitting the fused data to an IoT cloud platform via a unified application layer protocol.

[0005] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the IoT gateway adaptively collects data from sensing devices by: collecting data from various devices in real time according to the data collection tasks issued by the IoT cloud platform.

[0006] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the IoT gateway performs data fusion on the matched and filtered data, including: the IoT gateway realizes the collection model fusion and the upload model fusion based on the static model issued by the data model service subscribed to by the IoT cloud platform; real-time analysis and computing and persistent data storage and retrieval include: the IoT gateway executes the service algorithm based on the devices and data service topics of interest to the algorithm subscribed to by the IoT cloud platform, and completes data analysis and correlation processing; and performs corresponding storage / retrieval services on the fused data based on the devices and data service topics subscribed to by the IoT cloud platform.

[0007] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the IoT gateway transmits the fused data to the IoT cloud platform through a unified application layer protocol, including: the IoT gateway obtaining the corresponding upload model data according to the device and data upload service topics subscribed to by the IoT cloud platform, and performing protocol conversion for upload.

[0008] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the protocol conversion includes: performing an adaptive protocol conversion of the uploaded model data.

[0009] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the sensing device performs device coding and data coding according to a unified preset device coding and data coding standard; and the model is modeled according to a preset data modeling standard.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the model is downloaded to the IoT gateway after being trained by the IoT cloud platform; the IoT gateway uses real-time data to subsequently train the model.

[0011] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the device encoding is a sensing device address and the data encoding includes a data body and a data item.

[0012] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: performing object modeling of the Internet of Things (IoT) device; the object model includes basic information and device service information of the IoT device.

[0013] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes: establishing an association between the monitored object of the object model and the unique ID of the power grid resource of the holographic power grid model, and fusing the object model of the Internet of Things device with the holographic power grid model.

[0014] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0016] Figure 1 A schematic diagram of an exemplary operating environment in which embodiments of the present disclosure can be implemented is shown;

[0017] Figure 2 A schematic diagram is shown of a data processing method for multi-source Internet of Things data fusion and unification in which embodiments of the present disclosure can be implemented;

[0018] Figure 3 A schematic diagram of the object model structural framework according to an embodiment of the present disclosure is shown;

[0019] Figure 4 A schematic diagram of IoT gateway analysis calculation according to an embodiment of the present disclosure is shown;

[0020] Figure 5 A schematic diagram of a cloud-edge collaboration mechanism for a data model according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0022] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Figure 1 A schematic diagram of an exemplary operating environment 100 in which embodiments of the present disclosure can be implemented is shown;

[0024] In some embodiments, the edge agent terminal (IoT gateway) transmits data from various sensor devices in a unified manner through a unified transmission protocol and uploads it to the IoT cloud platform for processing by the digital twin system platform.

[0025] In some embodiments, the edge agent terminal (IoT gateway) sends control commands issued by the cloud platform to photovoltaic, energy storage, adjustable loads and electricity loads to macro-regulate the entire distribution area and implement regulation, control and integration of multiple ports.

[0026] Figure 2A schematic diagram is shown of a data processing method 200 for multi-source Internet of Things data fusion and unification in which embodiments of the present disclosure are implemented.

[0027] In box 210, the IoT gateway adaptively collects data from sensing devices and performs matching and filtering on the collected data;

[0028] In some embodiments, the IoT gateway device and the sensing device (smart sensor) communicate based on a preset communication definition specification to collect data from the sensing device.

[0029] In some embodiments, taking power Internet of Things (IoT) devices as an example, the sensing devices include charging piles, air source air conditioners, photovoltaic systems, temperature and humidity sensors, water immersion sensors, smoke sensors, video PTZ cameras, intelligent track robots, partial discharge monitoring sensors for medium-voltage switchgear in power distribution, infrared imaging monitoring devices, transformer temperature sensors, etc. These sensing devices form a sensing network.

[0030] The main access methods can be divided into wired access and wireless access.

[0031] In some embodiments, because different sensing networks use different protocol encapsulation forms to encapsulate the collected sensing information, this leads to soft isolation of data between different sensing networks. On the one hand, physically interconnected sensing networks cannot communicate with each other; on the other hand, sensing networks cannot communicate with the core switching network, thus preventing remote access to sensing data. Therefore, IoT smart gateways need to have the ability to interoperate with heterogeneous networks, the ability to monitor, control, and manage, and the ability to accommodate new node access. Simultaneously, all smart gateway nodes need to use standardized communication methods. In summary, considering IoT application scenarios, to standardize the address allocation of sensor devices accessing the IoT and unify communication between smart gateway devices and downlink devices, this disclosure adopts communication definition rules applicable to the communication between smart sensors and IoT gateway devices.

[0032] In some embodiments, a unified IoT device data coding standard is used to ensure the uniqueness of IoT device codes and the standardization of data codes, providing support for asset management and data modeling of the digital twin platform. IoT device data coding mainly consists of two parts: device coding and data coding. The coding objects include IoT devices, including IoT gateway devices and sensing devices. The coding system is as follows: Device coding: Primarily used to encode IoT devices, identifying their unique identity within the digital twin platform. Data coding: Primarily used to standardize the data capabilities of IoT devices, including data themes and data item definitions. Data themes are mainly used for data classification, identifying different data types of IoT devices. For example, the data themes of an electricity meter can be divided into operational data, meter reading data, and alarm data. Data items are the smallest granular data descriptions of IoT devices, used to describe a specific piece of data from the IoT device, including data name, data description, data type, data length, etc., similar to a database data dictionary.

[0033] In some embodiments, the device code can be a sensing device address, i.e., a sensor device address. Sensor device addresses are assigned based on the sensor manufacturer, type, and function. Smart gateways typically provide multiple RS485 serial ports and Ethernet ports, allowing for flexible adaptation. Sensors used in the Internet of Things (IoT) are uniformly connected to the smart gateway device via RS485 serial ports or Ethernet ports. The RS485 communication sensor device address range should be between 1 and 128, and the address allocation principle varies depending on the manufacturer, as shown in Table 1.

[0034] Table 1. Power Distribution IoT Sensor Address Allocation Table

[0035]

[0036]

[0037] Note: The default address refers to the initial address set for the device, which will be modified according to the actual access situation during actual use.

[0038] In some embodiments, the object model of the aforementioned IoT device is integrated with a holographic power grid model, such as SG-CIM (State Grid Corporation of China Public Information Model), to achieve data fusion between IoT devices and primary power grid equipment, and between the object model and the holographic power grid model, providing fundamental support for the use of IoT device data in power grid business applications.

[0039] In some embodiments, the object model, i.e., the data model defining the characteristic information and service capabilities of IoT devices, includes basic device information, device services (attributes, events, commands), and other information. The device coding and data capability attributes of the object model follow the coding rules of the "Power Internet of Things Device Data Coding Specification," as shown in the attached figure. Figure 3 As shown.

[0040] In some embodiments, an object-oriented approach is used to model the object model of IoT devices. Through object modeling, the digital twin platform can quickly and comprehensively identify IoT devices and their service capabilities, laying the foundation for rapid IoT device access.

[0041] In some embodiments, based on the production system ledger, a holographic power grid model is constructed using the SG-CIM (State Grid Corporation of China Public Information Model) modeling specification, primarily focusing on primary power equipment and covering power generation, transmission, distribution, and consumption. By establishing a correlation between the monitoring objects of the physical model and the unique IDs of the power grid resources in the holographic power grid model, data fusion is achieved between IoT devices and primary power grid equipment, and between the physical model and the holographic power grid model.

[0042] Based on the power grid's production system ledger and equipment ledger, the equipment categories and functional location structures of primary equipment in the main and distribution networks are identified. Then, the correspondence between the equipment categories in the production ledger and the power grid SGCIM model is established (e.g., transformers in the production ledger are mapped to Transformer objects in the power grid SGCIM model). Simultaneously, the necessary data information for holographic modeling (such as functional location numbers and superior functional location numbers) is extracted from the equipment ledger. The identified functional structure and data information of the primary equipment in the main and distribution networks are then used to generate a unified holographic power grid model according to the SG-CIM modeling specifications and the mapping relationship between the equipment categories in the production ledger and the power grid SG-CIM model. During modeling, the "rdf:ID" (i.e., the unique ID of the power grid resource) in the holographic power grid model should be consistent with the functional location code in the production ledger. In the basic equipment information of the IoT device model, monitoring object and monitoring location information are expanded to describe the primary power grid equipment monitored by the IoT devices and their location information. By establishing a correlation between the monitored objects of the object model and the "rdf:ID" (i.e., the unique ID of the power grid resource) of the holographic power grid model, data fusion between IoT devices and primary power grid equipment, and between the object model and the holographic power grid model, is achieved, providing basic support for the use of IoT device data in power grid business applications.

[0043] In some embodiments, the IoT gateway device performs matching and filtering on the collected data for subsequent edge computing; wherein, the matching and filtering includes:

[0044] String matching and parsing: Due to the self-configuration nature of the sensing device, matching and parsing are required. The system determines whether the collected data matches; if so, it performs matching mode parsing; otherwise, it performs non-matching mode parsing; finally, the actual data is obtained. This parsing involves string matching, and to facilitate data input and reading, byte matching is used to parse the required bytes. It mainly consists of matching mode and non-matching mode. Matching mode involves matching the entire data frame, such as 0203 0405 06; non-matching mode is mainly used for longer data frames, where byte manipulation can be used to extract the data.

[0045] Edge data filtering: Edge data filtering primarily employs two methods: incremental thresholding and maximum / minimum value filtering. The difference between the currently acquired data and previous data is calculated, and the percentage of this difference relative to the current data collection value is determined. If the difference exceeds the incremental threshold, the data is retained; otherwise, it is discarded. For maximum / minimum values, a comparison method is used, comparing the parsed and transformed results with preset maximum and minimum values ​​to determine whether the data should be retained. The retained data is then uploaded to the IoT cloud platform (power distribution network platform).

[0046] In some embodiments, based on the southbound device data model, data acquisition, analysis and computation, and data access are all initiated and subscribed to by the IoT cloud platform (power distribution network platform) from the IoT gateway (edge ​​data center) in a service-oriented manner. Specifically, the acquisition service collects data from various devices in real time and publishes the data acquisition tasks to the edge data center in real time. The edge computing service subscribes to the device and data service topics of interest to the algorithm from the edge data center, enabling the edge data center to execute the service algorithm and complete data analysis and correlation processing. The persistent data storage / retrieval service subscribes to the configured device and data service topics from the edge data center, enabling the edge data center to execute corresponding storage / retrieval services. The data upload service subscribes to the configured device and data upload service topics from the edge data center, enabling the edge data center to obtain the corresponding upload model data, perform protocol conversion, and upload it.

[0047] In some embodiments, the IoT gateway collects data from various devices in real time according to the data collection tasks issued by the IoT cloud platform (power distribution network platform).

[0048] In some embodiments, the adaptive acquisition refers to the adaptability of the acquisition interval, that is, the data is preprocessed at the edge and targeted data filtering is performed to adaptively adjust the acquisition interval, thereby reducing the amount of data without causing the loss of critical data.

[0049] In box 220, the IoT gateway performs data fusion on the matched and filtered data, and performs real-time analysis and calculation as well as persistent data storage and retrieval.

[0050] In some embodiments, such as Figure 4 As shown, the IoT gateway performs data fusion on the matched and filtered data, and performs real-time analysis and calculation as well as persistent data storage.

[0051] In some embodiments, the IoT gateway uses the static model distributed by the data model service subscribed to by the IoT cloud platform (power distribution network platform) to realize the fusion of collected models and the fusion of uploaded models.

[0052] In some embodiments, the IoT gateway executes service algorithms based on the device and data service topics of interest to the algorithms subscribed to by the IoT cloud platform (power distribution network platform), and completes data analysis and correlation processing; including retrieving computational data and historical data based on the original model data for real-time analysis and computation to obtain computational result data.

[0053] In some embodiments, such as a schematic diagram of a data model cloud-edge collaboration mechanism Figure 5 As shown, the model used for computation by the IoT gateway is based on cloud-edge collaboration technology, trained by the IoT cloud platform and downloaded to the IoT gateway. While performing computational tasks, the IoT gateway also utilizes real-time collected data to further train the model, prioritizing the delivery of high-level early warning data to the IoT cloud platform. This alleviates the pressure on the IoT cloud platform, reduces the amount of data transmitted over the network, and ensures rapid delivery of early warning data. Through collaboration with the cloud data center, an iterative update mechanism for the existing model (active power distribution network application scenario model) is established on the IoT gateway (edge ​​side), achieving efficient linkage of cloud and edge data streams and consistent data services.

[0054] In some embodiments, the IoT gateway performs corresponding storage / retrieval services on the merged data based on the device and data service topics subscribed to by the IoT cloud platform (power distribution network platform); including storing historical data and retrieving historical data.

[0055] In box 230, the IoT gateway transmits the merged data to the IoT cloud platform via a unified application layer protocol;

[0056] In some embodiments, the IoT gateway subscribes to the configured device and data upload service topics on the IoT cloud platform (power distribution network platform), obtains the corresponding upload model data, performs protocol conversion, and uploads it.

[0057] In some embodiments, the IoT gateway (smart gateway for power distribution IoT devices) uses the MQTT protocol to uniformly access the IoT cloud platform (power distribution network platform), following preset device coding, data coding, and data modeling specifications. The preset data modeling specifications include: the subject and load should adopt specific formats during modeling. Modeling should focus on sensors (such as IoT gateways, smoke sensors, temperature sensors, etc.), and data items with similar uses and collection frequencies should be modeled using the same subject for data reporting. Device coding and data coding, as described above, will not be repeated here.

[0058] According to embodiments of this disclosure, based on cloud-edge collaboration, data processing between the edge agent device and the intelligent sensing cloud is realized by considering the hardware resources, data format, communication interface, security protocol, and transmission throughput of the IoT gateway (edge ​​agent device) and the IoT cloud platform (intelligent sensing cloud). This reduces the amount of data transmitted over the network, improves the computing efficiency of cloud computing, collaboratively fulfills user business needs, and achieves rapid response through data collaborative caching, realizing "sensory autonomy" within the region and enhancing the real-time performance of collaborative work.

[0059] According to the embodiments of this disclosure, the following technical effects are achieved:

[0060] To address the issues of inconsistent specifications for sensing layer devices from different manufacturers and models, difficulties in access, and time-consuming protocol parsing, a unified device data model is established to achieve a unified device model, unified coding standards, and unified communication protocols, thus meeting the needs of power distribution network application scenarios.

[0061] The IoT gateway supports automatic switching between multiple communication protocols in the power industry and intelligent adaptation to various interface types of power IoT terminals, enabling quick and flexible access for a variety of IoT devices.

[0062] For the access and forwarding of data from active power distribution network terminal equipment, the IoT cloud platform was connected to subsystems such as energy storage and photovoltaics, and the data acquisition, processing and control strategies of the active power distribution network were distributed to the IoT gateway.

[0063] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0064] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0065] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method for the fusion and unification of multi-source Internet of Things (IoT) data, comprising: The IoT gateway adaptively collects data from sensing devices and performs matching and filtering on the collected data; The IoT gateway performs data fusion on the matched and filtered data, and performs real-time analysis and calculation as well as persistent data storage and retrieval. The IoT gateway transmits the merged data to the IoT cloud platform through a unified application layer protocol; The IoT gateway performs data fusion on the matched and filtered data, including: The IoT gateway uses the static model distributed by the data model service subscribed to by the IoT cloud platform to realize the fusion of collected models and the fusion of uploaded models. Real-time analytics and persistent data access include: The IoT gateway executes service algorithms based on the devices and data service topics that the algorithms subscribed to by the IoT cloud platform are interested in, and completes data analysis and correlation processing; based on the devices and data service topics subscribed to by the IoT cloud platform, it performs corresponding storage / retrieval services on the fused data; The IoT gateway transmits the merged data to the IoT cloud platform via a unified application layer protocol, including: The IoT gateway obtains the corresponding upload model data based on the device and data upload service topics subscribed to by the IoT cloud platform, performs protocol conversion, and uploads the data; the protocol conversion includes: performing adaptive protocol conversion of the upload model data.

2. The method according to claim 1, wherein, The IoT gateway adaptively collects data from sensing devices by: collecting data from various devices in real time according to the data collection tasks issued by the IoT cloud platform.

3. The method according to claim 1, wherein, The method further includes: IoT devices and data are coded according to a unified preset device coding and data coding standard; the model is modeled according to a preset data modeling standard.

4. The method according to claim 3, wherein, The model is trained by the IoT cloud platform and then downloaded to the IoT gateway; the IoT gateway uses real-time data to further train the model.

5. The method according to claim 3, wherein, The device code is the address of the sensing device, and the data code includes the data body and the data items.

6. The method according to claim 3, wherein, The method further includes: Perform object modeling for IoT devices; the object model includes basic information and device service information of the IoT devices.

7. The method according to claim 6, wherein, The method further includes: By establishing a correlation between the monitored objects of the object model and the unique ID of the power grid resources in the holographic power grid model, the object model of the IoT device is fused with the holographic power grid model.

Images