Data processing method of multi-source heterogeneous acquisition system

Abstract

The invention relates to a data processing method of a multi-source heterogeneous acquisition system, which can effectively guarantee the data consistency and transmission security of multi-source heterogeneous equipment. Comprising the following steps: receiving a communication request sent by a multi-source heterogeneous device in an acquisition system, searching a local device library for matching according to device information in device communication, and establishing communication with the multi-source heterogeneous device through a unified communication protocol; determining a corresponding IP address and a port number according to the physical position of the equipment, and distributing a receiving node according to the data sending frequency and the data volume of the multi-source heterogeneous equipment; and receiving data sent by the multi-source heterogeneous equipment through the corresponding node, judging whether the equipment virtual physical address for sending the data is matched with the equipment virtual physical address stored locally, if so, receiving the data and storing the data in a local database, and otherwise, returning a data sending failure signal.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method for a multi-source heterogeneous acquisition system. Background Art

[0002] Intelligence and automation have become the development trend of industrial production, and fully automatic intelligent production is its future development direction. The process of a certain production line is complex, with the significant characteristics of multiple processes, multiple equipment, multiple interfaces, multiple protocols, and big data, which is a typical multi-source heterogeneous environment. In the data collection process, there are problems such as diverse process data types, large data volume, inconsistent communication protocols, difficult synchronization perception, and complex data storage management. In view of this complex environment, it is necessary to study the processing method of multi-source heterogeneous data, realize the real-time collection of multi-source heterogeneous data, and ensure the consistency and transmission security of multi-source heterogeneous data in the multi-source heterogeneous data collection system. Summary of the invention

[0003] In view of the problems in the prior art, the present application provides a method for processing multi-source heterogeneous data collected in a multi-source heterogeneous data acquisition system, which can effectively ensure the consistency and transmission security of multi-source heterogeneous device data. The present application provides a method for processing multi-source heterogeneous data collected in a multi-source heterogeneous data acquisition system, comprising:

[0004] Receive communication requests sent by multi-source heterogeneous devices in the acquisition system, search the local device library for matching according to the device information in the device communication, and establish communication with the multi-source heterogeneous devices through a unified communication protocol; determine the corresponding IP address and port number according to the physical location of the device, and allocate receiving nodes according to the data sending frequency and data volume of the multi-source heterogeneous devices; receive data sent by multi-source heterogeneous devices through corresponding nodes, and determine whether the device virtual physical address of the data sent matches the device virtual physical address stored locally. If so, receive the data and store the data in the local database, otherwise, return a data sending failure signal.

[0005] Beneficial Effects

[0006] 1. This application can effectively ensure the consistency of multi-source heterogeneous data in a multi-source heterogeneous data acquisition system.

[0007] 2. This application solves the problems of device identity authentication and data transmission security, significantly improving the security of the IoT system. Through this technical solution, the IoT management platform can effectively manage and control access devices to ensure the smooth operation of the system and the secure transmission of data.

[0008] 3. This application allocates receiving nodes according to the data transmission frequency and data volume of multi-source heterogeneous devices. The bandwidth and maximum network data transmission and reception volume corresponding to each receiving node are different, so as to receive data information more reasonably and efficiently.

[0009] 4. This application will dynamically adjust the network identifier and connection port according to the current network communication performance and network bandwidth. Network communication performance includes indicators such as network latency, bandwidth utilization, and data transmission rate. The system monitors these indicators in real time and dynamically optimizes the network configuration to improve the overall network performance and device communication efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 , flow chart of the method of the present invention. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. The acquisition, storage, use, and processing of data in the technical solutions of this application comply with the relevant provisions of national laws and regulations.

[0012] Considering the existing technology, due to the heterogeneity of multi-source heterogeneous devices, the data produced by different devices may be difficult to integrate due to inconsistent formats or inconsistent semantics, which requires the establishment of a unified data exchange standard between devices and platforms. For example, OPC UA and MQTT have become widely accepted solutions in the industry, which help to eliminate the data gap by providing a unified data interface specification. However, due to the existence of historical legacy equipment and the coexistence of old and new equipment, the full promotion of these standardized solutions still faces challenges.

[0013] The present application provides a method for processing multi-source heterogeneous data collected in a multi-source heterogeneous data collection system, receiving a communication request sent by a multi-source heterogeneous device in the collection system, searching a local device library for matching according to device information in device communication, and establishing communication with the multi-source heterogeneous device through a unified communication protocol; determining a corresponding IP address and port number according to the physical location of the device, and allocating a receiving node according to the data sending frequency and data volume of the multi-source heterogeneous device; receiving data sent by the multi-source heterogeneous device through a corresponding node, and determining whether a device virtual physical address for sending the data matches a device virtual physical address stored locally; if so, receiving the data and storing the data in a local database; otherwise, returning a data sending failure signal.

[0014] In order to effectively ensure data consistency and transmission security, the present application provides an embodiment of a data processing method for a multi-source heterogeneous data acquisition system, which specifically includes the following contents:

[0015] 1. Receive communication requests sent by multi-source heterogeneous devices in the acquisition system, search the local device library for matching according to the device information in the device communication, and establish communication with multi-source heterogeneous devices through a unified communication protocol.

[0016] The system first receives startup communication requests from multi-source heterogeneous devices in the collection system. Whenever a device starts, it automatically sends a communication request to the management platform. The communication request contains a device identifier and a device signature so that the management platform can identify and verify the identity of the device. The device identifier is usually unique and can be the device's MAC address, serial number, or other unique identifier.

[0017] After receiving a device communication request, the system will search the local pre-stored device library according to the device identifier in the request to find a matching device. The device library is a database that stores all legitimate devices, and each device has a unique identifier. The system uses key infrastructure technology to verify the device in the device communication request to ensure the legitimacy of the device identity. After finding the corresponding device, the specific information and transmission method of the device are identified, and the identified information is integrated into a unified communication protocol. This solution solves the problems of device identity authentication and data transmission security, and significantly improves the security and reliability of the IoT system. Through this technical solution, the IoT management platform can effectively manage and control access devices to ensure the smooth operation of the system and the secure transmission of data.

[0018] 2. Determine the corresponding IP address and port number according to the physical location of the device, and allocate the receiving node according to the data transmission frequency and data volume of the multi-source heterogeneous devices:

[0019] The system associates and binds the device virtual physical address in the device communication request with the device identification and stores it in the local database. The device virtual physical address is usually a virtual identifier used to identify the location and identity of the device in the network. By binding the device virtual physical address with the device identification, the system can quickly locate and manage the device, ensuring the uniqueness and traceability of the device in the network. This step involves database operations, including insertion, update, and query, to ensure the accuracy and consistency of device information.

[0020] Based on the physical location of the device in the device connection request, the system determines the corresponding network identifier and connection port. The physical location of the device is usually expressed in the connection request in the form of geographic coordinates or network topology location. Based on this information, the system assigns the appropriate network identifier and connection port to ensure that the device can correctly access the network and communicate.

[0021] Receiving nodes are allocated according to the data transmission frequency and data volume of multi-source heterogeneous devices. The bandwidth and maximum network data transmission and reception volume corresponding to each receiving node are different, so as to receive data information more reasonably and efficiently.

[0022] The system will dynamically adjust the network identifier and connection port based on the current network communication performance and network bandwidth. Network communication performance includes indicators such as network latency, bandwidth utilization, and data transmission rate. The system monitors these indicators in real time and dynamically optimizes the network configuration to improve overall network performance and device communication efficiency.

[0023] 3. Receive data sent by multi-source heterogeneous devices through corresponding nodes, and determine whether the device virtual physical address of the data is matched with the device virtual physical address stored locally. If so, receive the data and store it in the local database. Otherwise, return a data sending failure signal:

[0024] The system checks whether the device virtual physical address in the request matches the locally stored device virtual physical address. The device virtual physical address is the unique identifier of the device in the network, and the system has bound it to the device ID and stored it during the initial connection request processing. If the virtual physical address in the upload request does not match the locally stored one, it means that the request may come from an unauthorized device or the data has been tampered with, and the system will return a data upload failure signal.

[0025] If the device virtual physical address matches, the system will continue to process the upload request. The system extracts the uploaded data in the request and stores it in the local time series database. A time series database is a database specifically used to store and query time series data. It is very suitable for storing device data in IoT systems. The time series database can efficiently store a large amount of timestamp data and provide fast query and analysis functions, supporting real-time data processing and historical data analysis.

Claims

1. A data processing method for a multi-source heterogeneous acquisition system, characterized by: Including receiving communication requests sent by multi-source heterogeneous devices in the acquisition system; Search the local device library for matching based on the device information in device communication, and establish communication with multi-source heterogeneous devices through a unified communication protocol; Determine the corresponding IP address and port number according to the physical location of the device, and allocate the receiving node by using the data transmission frequency and data volume of the multi-source heterogeneous devices; The data sent by multi-source heterogeneous devices are received through corresponding nodes, and it is determined whether the device virtual physical address of the data is matched with the device virtual physical address stored locally. If so, the data is received and stored in the local database; otherwise, a data sending failure signal is returned.

2. A data processing method for a multi-source heterogeneous acquisition system according to claim 1, characterized in that: The communication request includes a device identifier and a device signature.

3. A data processing method for a multi-source heterogeneous acquisition system according to claim 1, characterized in that: The device library is a database storing all legal devices, and the unique identifier of each device is determined.

4. A data processing method for a multi-source heterogeneous acquisition system according to claim 3, characterized in that: Utilize key infrastructure technology to verify the device in the device communication request to ensure the legitimacy of the device identity.

5. The data processing method of any one of claims 1 to 4 for a multi-source heterogeneous acquisition system is characterized by: The local database is a database specifically used for storing and querying time series data.

Images