A method for receiving and parsing heterogeneous data in multiple communication modes

Abstract

The present invention discloses a method for receiving and parsing heterogeneous data under multiple communication modes. The method is divided into a communication layer, a parsing layer, a configuration layer and a storage layer, which form a set of standardized data processing processes and output data formats. The communication layer realizes device access with MQTT, HTTP and CoAP as communication protocols by integrating the existing communication framework; the parsing layer realizes data parsing by integrating the characteristics of three data formats: JSON, string and byte stream, and converts it into a standard JSON data structure, thereby realizing unified output of data format, which is convenient for later business systems to use; the configuration layer implements the organization of communication parameters and parsing parameters in accordance with the principle that convention is greater than configuration and configuration is greater than code, and parameter configuration is realized by inputting through a visual interface; the storage layer is responsible for storing the above configuration information, and adopts Redis and MySQL as storage media. The method effectively ensures that the technical basis and configuration basis are provided when the data for TCP and UDP protocols are expanded in the later stage.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method for receiving and parsing heterogeneous data under multiple communication modes. Background Art

[0002] With the rapid development of global computer technology and information and communications technology, IoT-related data collection and transmission technologies have also flourished. Consequently, the degree of informatization in corresponding data collection systems has also increased, leading to a rapid increase in the amount of data shared by society and an increasing diversity of data formats. Therefore, as the middle layer connecting devices and service platforms, the role of IoT gateways is becoming increasingly important. They have gradually become the link between sensor networks and traditional communication networks. The requirements for them are also becoming increasingly high, requiring them to connect to more communication methods and protocols.

[0003] Early remote data transmission mainly used TCP and UDP in communication, and the data format was mainly byte stream, such as Mobus TCP, Iec104, etc. Later, with the development of operating systems, hardware specifications, and communication technologies, MQTT, HTTP, and CoAP gradually emerged as communication methods for data transmission:

[0004] The MQTT protocol is a message queue transmission protocol and a lightweight communication protocol based on the publish / subscribe model. Built on the TCP / IP protocol, its greatest advantage is that it can provide real-time and reliable messaging services to remote devices with minimal code and limited bandwidth. As a low-overhead, low-bandwidth instant messaging protocol, MQTT has a wide range of applications in the Internet of Things, small devices, mobile applications, and other areas. It can transmit data in different formats through different topics. The data sender and receiver are both TCP clients, and data is transmitted through the broker in the middle. In terms of technical implementation, this method uses the open source network communication framework Netty to implement an MQTT communication framework.

[0005] HTTP, one of the most common communication protocols, is built on top of TCP / IP. It supports a client / server model, operates in a simple and reliable manner, has well-defined status codes and methods, and is stateless. Using the POST method to transmit data is a common method in the Internet of Things. Implementing URLs in the RESTful style is currently a popular approach. Technically, using Spring Boot as the underlying technology to support the web framework allows for rapid construction of service interfaces.

[0006] CoAP adopts the same request-response working mode as the HTTP protocol. It is a special Web protocol that can be used on unreliable devices or networks. These devices are generally microcontrollers with only a single byte. This protocol is mainly used for point-to-point applications, such as low-power devices and smart devices, and also supports multicast. CoAP is similar to HTTP. CoAP can be regarded as a scaled-down version of HTTP for small storage and unreliable environments. The Californium framework is a CoAP technology framework implemented based on Java. This project implements various request-response definitions of the CoAP protocol and supports different reliability transmission modes such as CON / NON. Californium is based on a hierarchical design and is highly scalable.

[0007] In terms of the data content transmitted, the data formats transmitted by devices with different characteristics are also diverse. The content involved includes JSON, formatted strings, and byte streams: JSON is a lightweight data interchange format. In essence, it is a string and is composed of one or more sets of key-value pair data to form an object, such as "{\"name\":\"Zhang Si\",\"values\":1}". A formatted string is a set of strings that can be parsed through head and tail identifiers or data separation characters, such as the data format in the Ntrip protocol "$GPGGA,134658.00,5106.9792,N,11402.3003,W,2,09,1.0,1048.47,M,-16.27,M,08,0000*60\r\n". Byte stream data means that the transmitted data is a stream with bytes as the basic unit of transmission, including a continuous stream of boundary data; a byte stream is composed of bytes and is mainly used for processing binary format data. For example, ModbusTCP belongs to the byte stream format data "000100000006010100020004".

[0008] Currently, there are various mainstream Internet of Things products. Most of them support one communication method and one data format. However, in actual work, there is a need to support the access of different communication methods and different data formats of multiple devices in certain specific scenarios. However, developing and deploying different device processing platforms increases the R & D costs and operation and maintenance costs. Therefore, it is necessary to integrate the technical characteristics of the above multiple communication methods and multiple data parsing methods to implement a method for accessing and parsing protocols of multiple types of Internet of Things devices under a single platform to support actual business applications. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method for receiving and parsing heterogeneous data under multiple communication modes. The method starts from the actual application scenarios of processing the above-mentioned common communication modes and common data formats of various devices in the Internet of Things, and divides the implementation method into four parts: communication layer, parsing layer, configuration layer and storage layer, forming a set of standardized data processing processes and output data formats, ensuring that a technical foundation and configuration foundation are provided for the subsequent expansion of data for TCP and UDP protocols.

[0010] The purpose of the present invention is achieved through the following technical solutions:

[0011] A method for receiving and parsing heterogeneous data in multiple communication modes, comprising the following steps:

[0012] Step 1: Enter the communication and resolution configurations related to IoT device access through the interface, including device information, communication information, overall resolution information, segment resolution information, and other related configurations, and update them to Redis and MySQL;

[0013] Step 2: The communication layer obtains the communication configuration through Redis, starts the MQTT, CoAP, and HTTP communication services, and waits for the device to access.

[0014] Step 3: When the device is connected via HTTP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and original data are sent to the parsing layer;

[0015] Step 3.1: When a device is connected via CoAP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and original data are sent to the parsing layer;

[0016] Step 3.2: When the device is connected via MQTT, the device information is queried in Redis based on the Topic the device is connected to and the username (i.e., device ID) sent to determine whether the device is valid. If the device is valid, the device's "configuration items" and raw data are sent to the parsing layer.

[0017] Step 4: After the data is transferred to the parsing layer, the configuration layer searches for specific parsing information from Redis based on the "configuration item", including JSON, byte stream, and string content. This completes the acquisition of the basic information required for data parsing.

[0018] Step 5: Create a task through the Guava thread pool: create the corresponding data parsing entity class object according to the "configuration item"; then pass the information obtained in step 4 into the entity class;

[0019] Step 6: Start the task in the Guava thread pool to start the data task. First, perform data verification to determine whether the imported original data is valid and parseable data. If it is valid data, perform data parsing.

[0020] Step 7: After the execution is completed, the JSON data with a standard unified data structure will be output to the storage layer. Currently, it can be output to Kafka and RocketMQ middleware.

[0021] Preferably, the communication layer integrates existing communication frameworks to enable device access using MQTT, HTTP, and CoAP as communication protocols;

[0022] The parsing layer integrates the characteristics of three data formats: JSON, string, and byte stream to parse the data and convert it into a standard JSON data structure, thereby achieving unified output of the data format and facilitating subsequent use in business systems.

[0023] The configuration layer follows the principle of convention over configuration and configuration over code to organize communication parameters and parsing parameters, and implements parameter configuration through input in a visual interface.

[0024] The storage layer is responsible for storing the above configuration information, using Redis and MySQL as storage media.

[0025] Preferably, the parsing layer implements HTTP communication and basic program framework through the SpringBoot framework, and uses the California framework to implement CoAP communication; these two frameworks do not require technical modifications to implement the communication architecture; the MQTT communication protocol is implemented through the Netty framework, and some functions of the Broker in the communication architecture are adjusted, removing the TopicTree and MQTT subscriber-side Subscribe-related data subscription functions, retaining the MQTT communication processing function, and when the MQTT data sender Publish data, the data in PayLoad is directly processed to ensure that the data reception and processing process remains consistent with HTTP, CoAP, and other Request / Response.

[0026] Preferably, the configuration layer divides the parsing parameters of the transmission format into two parts for the data formats supported by the three different communication protocols. The first part is the overall parsing configuration, and the second part is the parsing configuration for each data format:

[0027] HTTP and CoAP access rules: Use POST to upload. Parameters are introduced in the URL as device parsing data information, consisting of the following two parts: the server IP and port are implemented in the configuration file, and the URL path is described ( / api / v1 / {configuration item} / {device code} / telemetry, where "configuration item" is configured as described in the table above and "device code" is the device's unique identifier).

[0028] MQTT access rules: The Broker's IP and port are implemented through the configuration file. The Topic in the table above is configured through the interface as the data parsing (data format and data processing details search) criterion; the device identifier is used as the user name to determine access permissions.

[0029] Preferably, the storage layer includes storing in redis for real-time calling to increase the speed of device processing access data parsing and storing in MySQL database for archiving. When the configuration information needs to be modified, it is necessary to obtain information from the MySQL database and display it on the interface. After modification through the interface, the redis and MySQL database are updated.

[0030] The advantages and technical effects of the present invention are:

[0031] 1. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, and proposes a layered processing concept to ensure high decoupling between modules.

[0032] 2. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, which adopts a distributed architecture. During deployment, a load balancing node can be added to the front end of the service and multiple service nodes can be dynamically deployed at the back end, thereby improving the system's availability and data processing capabilities.

[0033] 3. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, and a configurable data parsing method, which can dynamically expand data parsing in three formats: JSON, string, and byte stream, without having to write code.

[0034] 4. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, unifies the output data format, ensures the consistency of data output by each device, and facilitates secondary processing of the data by other applications.

[0035] 5. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, and an independent and abstract communication technology implementation method, which can achieve rapid expansion of device access modes and directly connect to data processing modules to achieve rapid business expansion.

[0036] 6. The present invention provides a method for receiving and parsing heterogeneous data under multiple communication modes, which provides a set of standardized data processing processes and output data formats, ensuring the technical and configuration basis for subsequent expansion of data for TCP and UDP protocols. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of data flow of the present invention;

[0038] Figure 2 Schematic diagram of parameter configuration and abstract relationship in the present invention;

[0039] Figure 3 Schematic diagram of parameter classification corresponding to the three data formats in the present invention. DETAILED DESCRIPTION

[0040] In order to further understand the content, features and effects of the present invention, the following embodiments are given as examples and described in detail with reference to the accompanying drawings. It should be noted that the embodiments are illustrative and not restrictive, and the scope of protection of the present invention cannot be limited thereby.

[0041] like Figure 1 As shown, the present invention is a method for receiving and parsing heterogeneous data in multiple communication modes, and the specific steps are as follows:

[0042] (1) Enter the communication configuration and analysis configuration related to the IoT device access through the interface, including: device information, communication information, overall analysis information, segmented analysis information and other related configurations, and update them to Redis and MySQL.

[0043] (2) The communication layer obtains the communication configuration through Redis, starts the MQTT, CoAP, and HTTP communication services, and waits for the device to access.

[0044] (3) When a device is connected via HTTP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and original data are sent to the parsing layer.

[0045] (4) When a device is connected via CoAP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and original data are sent to the parsing layer.

[0046] (5) When the device is connected via MQTT, the corresponding device information is queried in Redis based on the Topic the device is connected to and the user name (i.e., device ID) sent to determine whether the device is valid. If the device is valid, the device's "configuration items" and original data are sent to the parsing layer.

[0047] (6) When the data is transferred to the parsing layer, the specific parsing (JSON, byte stream, string) configuration is searched from Redis according to the "configuration item". At this time, the basic information required for data parsing is obtained.

[0048] (7) Create a task through the Guava thread pool: Create the corresponding data parsing entity class object based on the "configuration item". Then pass the information obtained in (6) to the entity class.

[0049] (8) Start the task in the Guava thread pool to start the data task. First, perform data verification to determine whether the imported original data is valid and parsable data. If it is valid data, perform data parsing.

[0050] (9) After execution is completed, the JSON data with a standard unified data structure will be output, which can currently be output to Kafka and RocketMQ middleware.

[0051] The present invention is based on the actual application scenarios of processing the above-mentioned common communication methods and common data formats of various devices in the Internet of Things, and divides the implementation method into four parts: communication layer, parsing layer, configuration layer and storage layer.

[0052] The communication layer integrates existing communication frameworks to enable device access using MQTT, HTTP, and CoAP as communication protocols;

[0053] The parsing layer integrates the characteristics of three data formats: JSON, string, and byte stream to parse the data and convert it into a standard JSON data structure, thereby achieving unified output of the data format and facilitating subsequent use in business systems.

[0054] The configuration layer follows the principle of "convention over configuration, configuration over code" to organize communication parameters and parsing parameters, and implements parameter configuration by entering them through a visual interface.

[0055] The storage layer is responsible for storing the above configuration information, using Redis and MySQL as storage media.

[0056] In view of the characteristics of the communication layer, the present invention abstracts the following according to the different business data formats supported by different communication modes: Figure 2 The relationship shown serves as the basis for parameter configuration.

[0057] Integrate conventional data parsing methods: including big-endian and small-endian processing of data; conventional data types include string, short integer, bit, integer, short floating point, long integer, long floating point, time, ASCII, latitude and longitude, and other types of data parsing.

[0058] The parsing layer uses the SpringBoot framework to implement HTTP communication and the underlying programming framework, and the California framework to implement CoAP communication. These two frameworks can implement the communication architecture without requiring any technical modifications. The MQTT communication protocol is implemented using the Netty framework, and some Broker functionality within the communication architecture has been adjusted, removing the TopicTree and MQTT subscriber-side Subscribe functions. While retaining the MQTT communication processing functionality, when the MQTT data sender receives Publish data, the data in the PayLoad is directly processed, ensuring that the data reception and processing process remains consistent with the HTTP and CoAP request / response systems.

[0059] The configuration layer divides the parsing parameters of the transmission format into two parts according to the data formats supported by the three different communication protocols. The first part is the overall parsing configuration, and the second part is the parsing configuration of the data format. The parameters corresponding to these three data formats are shown in Figure 3 .

[0060] HTTP and CoAP access rules: Use POST to upload. Parameters are introduced in the URL as device parsing data information, consisting of the following two parts: the server IP and port implemented in the configuration file, and the URL path description ( / api / v1 / {configuration item} / {device code} / telemetry, where "configuration item" is configured as described in the table above and "device code" is the device's unique identifier).

[0061] MQTT access rules: The Broker's IP and port are configured in the configuration file. The topics in the table above are configured through the interface as the criteria for data parsing (data format and data processing details). The device identifier is used as the username to determine access permissions.

[0062] In the parsing layer, this paper abstracts the parsing interface to implement four major interfaces: parsing initialization, format determination, parsing rule import, and data parsing. Then, corresponding parsing entity classes, JSON, string, and byte stream, are derived to implement the specific interfaces. In each specific implementation method, each method parses each data item according to the language characteristics.

[0063] Parsing thread tasks are assigned through the Guava thread pool, an open source framework of Google. In the thread tasks, the specific data parsing entity objects will parse the data one by one according to the "data parsing configuration", and convert the parsed data into the key of the output JSON data according to the "output field name", so as to achieve the output of a unified JSON structure.

[0064] The storage layer includes data storage in redis for real-time calls to increase the speed of device processing and access data parsing, and storage in the MySQL database for archiving. When the configuration information needs to be modified, it is necessary to obtain the information from the MySQL database and display it on the interface. After modification through the interface, the redis and MySQL database are updated.

[0065] Finally, it is preferred that the present invention adopts mature products and mature technical means in the prior art for all the parts not described in the present invention.

[0066] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for receiving and parsing heterogeneous data in multiple communication modes, characterized in that: The following steps are involved: Step 1: Enter the communication and resolution configurations related to IoT device access through the interface, including device information, communication information, overall resolution information, segment resolution information, and other related configurations, and update them to Redis and MySQL; Step 2: The communication layer obtains the communication configuration through Redis, starts the MQTT, CoAP, and HTTP communication services, and waits for the device to access. Step 3: When the device is connected via HTTP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and raw data are sent to the parsing layer; Step 3.1: When a device is connected via CoAP, the device ID information obtained from the URL in the data sent by the device is used to query the corresponding device information in Redis to determine whether the device is valid. If the device is valid, the device's "configuration items" and raw data are sent to the parsing layer. Step 3.2: When the device is connected via MQTT, the device information is queried in Redis based on the Topic the device is connected to and the username sent to determine whether the device is valid. If the device is valid, the device's "configuration items" and raw data are sent to the parsing layer. Step 4: After the data is transferred to the parsing layer, the configuration layer searches for specific parsing information from Redis based on the "configuration item", including JSON, byte stream, and string content. This completes the acquisition of the basic information required for data parsing. Step 5: Create a task through the Guava thread pool: create the corresponding data parsing entity class object according to the "configuration item"; then pass the information obtained in step 4 into the entity class; Step 6: Start the task in the Guava thread pool to start the data task. First, perform data verification to determine whether the imported original data is valid and parseable data. If it is valid data, perform data parsing. Step 7: After the execution is completed, the JSON data with a standard unified data structure will be output to the storage layer and then output to Kafka and RocketMQ middleware; The communication layer integrates existing communication frameworks to enable device access using MQTT, HTTP, and CoAP as communication protocols; The parsing layer integrates the characteristics of three data formats: JSON, string, and byte stream to parse the data and convert it into a standard JSON data structure, thereby achieving unified output of the data format and facilitating the use of subsequent business systems. The configuration layer follows the principle of convention over configuration and configuration over code to organize communication parameters and parsing parameters, and implements parameter configuration through input in a visual interface. The storage layer is responsible for storing the above configuration information, using Redis and MySQL as storage media; The parsing layer implements HTTP communication and the basic program framework through the SpringBoot framework, and uses the California framework to implement CoAP communication. The MQTT communication protocol is implemented through the Netty framework, and some functions of the Broker in the communication architecture are adjusted, removing the TopicTree and MQTT subscriber-side Subscribe-related data subscription functions, while retaining the MQTT communication processing function. When the MQTT data sender Publish data comes in, the data in PayLoad is directly processed to ensure that the data reception and processing process is consistent with the Request / Response of HTTP and CoAP.

2. The method for receiving and parsing heterogeneous data in multiple communication modes according to claim 1, characterized in that: The configuration layer divides the transmission format parsing parameters into two parts for the data formats supported by three different communication protocols. The first part is the overall parsing configuration, and the second part is the parsing configuration for each data format: HTTP and CoAP access rules: Use POST to upload data. Parameters are introduced in the URL as device parsing data information, consisting of the following two parts: the server IP and port are implemented in the configuration file, and the URL path is described ( / api / v1 / {configuration item} / {device code} / telemetry), where "configuration item" is configured as described in the table above and "device code" is the device's unique identifier. MQTT access rules: The Broker's IP and port are implemented through the configuration file. The Topic in the table above is configured through the interface as the data parsing criteria, including data format and data processing details. The device identifier is used as the user name to determine access permissions.

3. The method for receiving and parsing heterogeneous data in multiple communication modes according to claim 1, characterized in that: The storage layer includes storing in redis for real-time calling to increase the speed of device processing access data parsing and storing in MySQL database for archiving. When the configuration information needs to be modified, it is necessary to obtain information from the MySQL database and display it on the interface. After modification through the interface, the redis and MySQL database are updated.

Images