Data processing method and device in data acquisition gateway based on Mqtt protocol and JSON message
By adopting data processing methods based on Mqtt protocol and JSON packets in the digital acquisition gateway, the problems of multi-north-direction protocol docking difficulty and data mapping complexity are solved, efficient data acquisition and flexible transmission are achieved, and the reliability of the system is improved. and operational convenience.
Patent Information
- Application Number
- CN202411202697.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
The existing digital procurement gateways need to develop multiple northbound protocols when connecting with the backend of different companies, which increases the difficulty of using equipment and system complexity, and the data mapping and processing operations are not flexible enough.
Using the data processing method based on the Mqtt protocol and JSON packets, by configuring the data acquisition objects of the southbound devices and generating cloud objects, creating publishing objects and subscription objects, and using JSON format subscription variable analysis to map data to the southbound devices, realizing Efficient acquisition and flexible transmission of data.
It simplifies data mapping and processing operations, reduces system complexity, improves the working efficiency of digital procurement gateways, provides a clear system architecture and simple configuration process, and improves the reliability, versatility and operational convenience of the system.
Smart Images

Figure CN119071287A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of Internet of Things data acquisition, and in particular to a data processing method and device in a data acquisition gateway based on the MQTT protocol and JSON messages. Background Art
[0002] The data acquisition gateway is a device that collects data from industrial field equipment and uploads it to the remote background. The field equipment it collects is also called southbound equipment, and the communication protocol with the background is called northbound protocol. Mqtt is widely used in gateways as a lightweight IoT communication protocol, usually in string format. In order to connect to the backgrounds of different companies, it is necessary to develop corresponding northbound protocols. Even for different applications of the same company, different northbound protocols need to be developed. A gateway manufacturer will provide a large number of northbound protocol modules that connect to different company platforms, which increases the difficulty of using gateway devices. Summary of the invention
[0003] Based on this, it is necessary to provide a data processing method and device based on Mqtt protocol and JSON message in a data acquisition gateway to address the above technical problems.
[0004] A data processing method based on MQTT protocol and JSON message in a data acquisition gateway, the method comprising:
[0005] Configure the data collection object of the southbound device; the data collection object is the collection variable corresponding to the data point;
[0006] According to the background that needs to be connected, configure the northbound MQTT protocol to drive the generation of cloud objects, and create publishing objects and subscription objects under the cloud objects; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol;
[0007] By parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data point of the southbound device, so that the data collection protocol of the southbound device is called to write the collection variables of the data point into the southbound device, and reported to the background through the publishing object.
[0008] In one of the embodiments, the data collection object is configured as devices, sub-devices, collection tasks, and collection variables arranged in a hierarchical manner.
[0009] In one of the embodiments, the configuration of the publication object includes: configuring the reporting type, publication subject, and publication package content of the publication object.
[0010] In one of the embodiments, the data types in the release package content include string classes and replacement functions.
[0011] In one of the embodiments, it also includes: according to the data subject in the received subscription message, traversing the subscription object to obtain the subscription variable in JSON format corresponding to the subscription object, and mapping the data to be collected to the data point of the southbound device according to the Payload data in the subscription variable in JSON format and the data point of the southbound device associated with the subscription variable.
[0012] In one of the embodiments, the subscription variable in JSON format is associated with the data point of the southbound device by parsing the keyword hierarchy of the subscription variable in JSON format.
[0013] In one of the embodiments, parsing the keyword hierarchy of the subscription variable in JSON format includes parsing a single hierarchy or parsing multiple hierarchies.
[0014] A data processing device based on MQTT protocol and JSON message in a data acquisition gateway, the device comprising:
[0015] A southbound device configuration module, used to configure a data collection object of a southbound device; the data collection object is a collection variable corresponding to a data point;
[0016] A northbound protocol configuration module is used to configure the northbound MQTT protocol to drive the generation of cloud objects according to the background that needs to be connected, and create a publishing object and a subscription object under the cloud object; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol;
[0017] The data parsing module is used to map the data to be collected to the data point of the southbound device by parsing the subscription variables in JSON format in the subscription object, thereby calling the data collection protocol of the southbound device to write the collection variables of the data point into the southbound device, and report to the background through the publishing object.
[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0019] Configure the data collection object of the southbound device; the data collection object is the collection variable corresponding to the data point;
[0020] According to the background that needs to be connected, configure the northbound MQTT protocol to drive the generation of cloud objects, and create publishing objects and subscription objects under the cloud objects; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol;
[0021] By parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data point of the southbound device, so that the data collection protocol of the southbound device is called to write the collection variables of the data point into the southbound device, and reported to the background through the publishing object.
[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0023] Configure the data collection object of the southbound device; the data collection object is the collection variable corresponding to the data point;
[0024] According to the background that needs to be connected, configure the northbound MQTT protocol to drive the generation of cloud objects, and create publishing objects and subscription objects under the cloud objects; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol;
[0025] By parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data point of the southbound device, so that the data collection protocol of the southbound device is called to write the collection variables of the data point into the southbound device, and reported to the background through the publishing object.
[0026] The above-mentioned data processing method and device based on the Mqtt protocol and JSON message in the data acquisition gateway, by configuring the data acquisition object of the southbound device, accurately maps the data points of the underlying device to the acquisition variables, and realizes efficient data acquisition. The method also combines the northbound Mqtt protocol to generate cloud objects according to different background requirements, and realizes flexible data transmission and processing by creating publishing objects and subscription objects. During the data processing process, the subscription variable parsing in JSON format can intuitively map the data to be collected to the data points of the southbound device. This process simplifies the mapping and processing operations of the data, reduces the complexity of the system, and ensures the reliability and consistency of data transmission. The present invention not only improves the working efficiency of the digital acquisition gateway, but also provides a clear system architecture and a simple configuration process, so that users can flexibly adapt to a variety of application scenarios, significantly improving the reliability, versatility and ease of operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flowchart of a data processing method based on the Mqtt protocol and JSON message in a data acquisition gateway in one embodiment;
[0028] Figure 2 A schematic diagram of a JSON format configuration mapping method in an embodiment;
[0029] Figure 3 It is a structural block diagram of a data processing device in a data acquisition gateway based on the Mqtt protocol and JSON messages in an embodiment;
[0030] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0032] At present, data acquisition gateways from different companies all provide configuration tools, which are configured according to the actual needs of the application site. The configuration process is mainly divided into two parts: the southbound field data acquisition configuration part and the northbound protocol configuration part. The southbound configuration mainly selects the device type and the data points connected to the device (input sensor data, temperature, humidity, etc.). The northbound protocol first selects the Mqtt protocol object of the corresponding company, and then configures the upload topic as needed.
[0033] In one embodiment, Figure 1 As shown, a data processing method based on the MQTT protocol and JSON message in a data acquisition gateway is provided, comprising the following steps:
[0034] Step 102: configure the data collection object of the southbound device.
[0035] Different from the data collection process in traditional technology, this step abstracts the data points of the data collection object into collection variables.
[0036] Step 104, according to the background to be connected, configure the northbound MQTT protocol to drive the generation of cloud objects, and create publishing objects and subscription objects under the cloud objects.
[0037] The publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define that the collected data is processed through the northbound MQTT protocol.
[0038] Step 106, by parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data point of the southbound device, thereby calling the data collection protocol of the southbound device to write the collection variables of the data point into the southbound device, and reporting to the background by publishing the object.
[0039] The above technical solution abstracts the data collected by the southbound device into variables in the data acquisition gateway. Based on the configurable processing method of the Mqtt protocol, the data packet sent in the form of a string can be arbitrarily configured. At the same time, the collected data is inserted according to the specified format of the configuration and reported to the background in different ways. The reported data format is not limited to JSON. The JSON format data received by the Mqtt protocol is first classified according to the topic, and then the required data is extracted from the data packet according to the configuration information and written to the southbound device. This method can meet the vast majority of communication applications based on the Mqtt protocol and in JSON format. At present, this method has been applied in the China Power Internet Gateway.
[0040] The above-mentioned data processing method based on the Mqtt protocol and JSON message in the data acquisition gateway, by configuring the data acquisition object of the southbound device, accurately maps the data points of the underlying device to the acquisition variables, and realizes efficient data acquisition. This method also combines the northbound Mqtt protocol to generate cloud objects according to different background requirements, and realizes flexible data transmission and processing by creating publishing objects and subscription objects. During the data processing process, the subscription variable parsing in JSON format can intuitively map the data to be collected to the data points of the southbound device. This process simplifies the mapping and processing operations of the data, reduces the complexity of the system, and ensures the reliability and consistency of data transmission. The present invention not only improves the working efficiency of the digital acquisition gateway, but also provides a clear system architecture and a simple configuration process, so that users can flexibly adapt to a variety of application scenarios, significantly improving the reliability, versatility and ease of operation of the system.
[0041] In one embodiment, the data collection object is configured as devices, sub-devices, collection tasks, and collection variables arranged in a hierarchical manner.
[0042] In another embodiment, the configuration of the publishing object includes: configuring the reporting type of the publishing object, the publishing subject, and the publishing package content. Specifically, the data in the publishing package content is mainly divided into two categories: general string class and replacement function. The replacement function is in the form of $ followed by a string and brackets. The replacement function can be replaced with time, timestamp, variable value and other information. For example, "$tagValue(dev1.SubDev_1.task1.WinSpeed,%.2lf)}" is a replacement variable value, and two decimal places are retained. The final expression of the function is "xxx.xx", and the number of digits before the decimal point is determined according to the actual value, and the number of digits after the decimal point is fixed to 2; in addition, $localtime(%F) represents the insertion time, and the format is a timestamp.
[0043] In one embodiment, according to the data subject in the received subscription message, the subscription object is traversed to obtain the subscription variable in JSON format corresponding to the subscription object, and according to the Payload data in the subscription variable in JSON format and the data point of the southbound device associated with the subscription variable, the data to be collected is mapped to the data point of the southbound device. In this embodiment, the content of the release package must be in JSON format, so that information search and positioning can be achieved.
[0044] In one embodiment, the subscription variable in JSON format is associated with the data point of the southbound device by parsing the keyword hierarchy of the subscription variable in JSON format.
[0045] Specifically, the required keywords can be directly located through the keyword hierarchy, as shown in the following JSON package:
[0046] {"device":"123","data":{"params":{"InParams":{"PlatformPatrol":1}}}}. The data packet format is relatively simple. According to the JSON parsing rules, the object can be found directly through the keyword "data", and then through the sub-object "params" of the object. In this way, "PlatformPatrol" can be found directly one level at a time. After the data is received, it needs to be set to the data point wkmode in the southbound device mem. Therefore, the positioning configuration method of "PlatformPatrol" in the JSON is as follows Figure 2 shown.
[0047] In another embodiment, parsing the keyword level of the subscription variable in JSON format includes parsing a single level or parsing multiple levels. When multi-level positioning judgment is required, such as the above JSON format, when the data packets sent from the same topic have the same format, the device needs to be judged first when associating variables, then two positionings are required, first locating whether the device value matches, so the configuration list requires two items.
[0048] Finally, after the configuration is completed, a configuration file will be generated and stored in the specified location of the data acquisition gateway device. Restarting the gateway program can complete the sending and receiving analysis of the data.
[0049] In summary, the method of the present invention is applied to the data acquisition gateway device, and the method is implemented by configuring the software. When the data acquisition gateway device is running, the user configuration information is first loaded, and the corresponding southbound protocol and northbound protocol are loaded according to the configuration information, and then the data acquisition object is further established. The southbound protocol can perform both data acquisition and data writing operations; after the southbound protocol is successfully established, the data is read regularly according to the configuration cycle, and when the data written to the north is received, the data acquisition gateway program calls the protocol writing interface to write the parsed data into the device. The northbound protocol is a module developed based on Mqtt, which establishes a publishing topic and a subscription topic according to the user configuration. After the topic is established, the Payload data is dynamically generated and reported to the platform according to the configuration information of the publishing topic when data needs to be reported; when the subscription message is received, it is first matched according to the topic. When the corresponding topic is found, the subscription variables under the topic are polled in turn, and the specified information is hierarchically found in the received Payload according to the configuration of the subscription variable, and then the data is parsed according to the southbound variable type associated with the subscription variable, and finally the southbound protocol interface is called to write the data to the device.
[0050] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0051] In one embodiment, Figure 3As shown, a data processing device based on the MQTT protocol and the JSON message in the data acquisition gateway is provided, including: a southbound device configuration module 302, a northbound protocol configuration module 304 and a data parsing module 306, wherein:
[0052] The southbound device configuration module 302 is used to configure the data collection object of the southbound device; the data collection object is the collection variable corresponding to the data point;
[0053] The northbound protocol configuration module 304 is used to configure the northbound MQTT protocol to drive the generation of cloud objects according to the background that needs to be connected, and create a publishing object and a subscription object under the cloud object; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol;
[0054] The data parsing module 306 is used to map the data to be collected to the data point of the southbound device by parsing the subscription variables in JSON format in the subscription object, thereby calling the data collection protocol of the southbound device to write the collection variables of the data point into the southbound device, and report it to the background through the publishing object.
[0055] In one of the embodiments, the data collection object is configured as devices, sub-devices, collection tasks, and collection variables arranged in a hierarchical manner.
[0056] In one of the embodiments, the configuration of the publication object includes: configuring the reporting type, publication subject, and publication package content of the publication object.
[0057] In one of the embodiments, the data types in the release package content include string classes and replacement functions.
[0058] In one of the embodiments, the data parsing module 306 is also used to traverse the subscription object to obtain the subscription variable in JSON format corresponding to the subscription object according to the data subject in the received subscription message, and map the data to be collected to the data point of the southbound device according to the Payload data in the subscription variable in JSON format and the data point of the southbound device associated with the subscription variable.
[0059] In one of the embodiments, the subscription variable in JSON format is associated with the data point of the southbound device by parsing the keyword hierarchy of the subscription variable in JSON format.
[0060] In one of the embodiments, parsing the keyword hierarchy of the subscription variable in JSON format includes parsing a single hierarchy or parsing multiple hierarchies.
[0061] For the specific definition of the data processing device based on the Mqtt protocol and JSON message in the data acquisition gateway, please refer to the definition of the data processing method based on the Mqtt protocol and JSON message in the data acquisition gateway above, which will not be repeated here. Each module in the above-mentioned data processing device based on the Mqtt protocol and JSON message in the data acquisition gateway can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0062] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as user accounts and communication methods. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a general scheduling method for unmanned vehicles is implemented.
[0063] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0064] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0065] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0066] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0067] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0068] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A data processing method based on MQTT protocol and JSON message in data acquisition gateway, It is characterized in that The method comprises: Configure the data collection object of the southbound device; the data collection object is the collection variable corresponding to the data point; According to the background that needs to be connected, configure the northbound MQTT protocol to drive the generation of cloud objects, and create publishing objects and subscription objects under the cloud objects; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol; By parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data point of the southbound device, so that the data collection protocol of the southbound device is called to write the collection variables of the data point into the southbound device, and reported to the background through the publishing object.
2. The method according to claim 1, It is characterized in that The data collection object is configured as devices, sub-devices, collection tasks, and collection variables arranged in a hierarchical manner.
3. The method according to claim 1, It is characterized in that The configuration of the published object includes: Configure the reporting type, publishing topic, and publishing package content of the publishing object.
4. The method according to claim 3, It is characterized in that The data types in the release package content include a string class and a replacement function.
5. The method according to any one of claims 1 to 4, It is characterized in that By parsing the subscription variables in JSON format in the subscription object, the data to be collected is mapped to the data points of the southbound device, including: According to the data subject in the received subscription message, traverse the subscription object to obtain the subscription variable in JSON format corresponding to the subscription object; according to the Payload data in the subscription variable in JSON format and the data point of the southbound device associated with the subscription variable, map the data to be collected to the data point of the southbound device.
6. The method according to claim 5, It is characterized in that The manner of associating the subscription variable in JSON format with the data point of the southbound device is obtained by parsing the keyword hierarchy of the subscription variable in JSON format.
7. The method according to claim 6, It is characterized in that Parsing the keyword level of the subscription variable in JSON format includes parsing a single level or parsing multiple levels.
8. A data processing device based on Mqtt protocol and JSON message in data acquisition gateway, It is characterized in that The device comprises: A southbound device configuration module, used to configure a data collection object of a southbound device; the data collection object is a collection variable corresponding to a data point; A northbound protocol configuration module is used to configure the northbound MQTT protocol to drive the generation of cloud objects according to the background that needs to be connected, and create a publishing object and a subscription object under the cloud object; the publishing object is used to define that the collected data is reported to the background through the northbound MQTT protocol, and the subscription object is used to define the processing of the collected data through the northbound MQTT protocol; The data parsing module is used to map the data to be collected to the data point of the southbound device by parsing the subscription variables in JSON format in the subscription object, thereby calling the data collection protocol of the southbound device to write the collection variables of the data point into the southbound device, and report to the background through the publishing object.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Internet of Things data transmission method, system and device based on MQTT and storage medium
CN120416362A
Data management method of actual controller and related device
CN120704224A
Data transmission method, system and device and storage medium
CN120729965A
A data transmission method, system, device and storage medium
CN120729965B
