Internet of Things data transmission method, system and device based on MQTT and storage medium

Abstract

The invention relates to the technical field of data transmission, and particularly provides an Internet of Things data transmission method, system and device based on MQTT, and a storage medium, and the method comprises the steps: employing a dynamic plug-in to analyze Internet of Things data employing a plurality of communication protocols, and obtaining initial data in a unified format; extracting feature data from the initial data, and determining an MQTT theme according to the type of the feature data; and packaging the feature data into an MQTT message according to the MQTT theme, and issuing the MQTT message to a subscriber. According to the invention, the network pressure of data transmission of the Internet of Things is reduced, and the data transmission efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and particularly relates to an Internet of Things data transmission method, system, device, and storage medium based on MQTT. Background Art

[0002] With the rapid development of Internet of Things technology, the number of terminal devices has grown exponentially. In the existing Internet of Things data transmission process, it relies on a centralized message broker to directly distribute Internet of Things data. Its static routing mechanism has the problems of needing to pre-maintain the mapping relationship between device IDs and data consumers (the maintenance cost increases linearly with the number of devices), and the data classification depends on an artificial rule engine (the rule update delay reaches the hour level), which leads to the system bottleneck of low data distribution efficiency. In addition, in the traditional solution, the original Internet of Things data is directly distributed to data consumers without distinguishing the data value density, resulting in the mixed transmission of high-value and redundant data, wasting network resources and reducing the data transmission efficiency. Summary of the Invention

[0003] In view of the above deficiencies of the prior art, the present invention provides an Internet of Things data transmission method, system, device, and storage medium based on MQTT to solve the above technical problems.

[0004] In a first aspect, the present invention provides an Internet of Things data transmission method based on MQTT, including: Using a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format; Extracting feature data from the initial data and determining the MQTT topic according to the type of the feature data; Encapsulating the feature data into an MQTT message according to the MQTT topic and publishing the MQTT message to subscribers.

[0005] In an optional embodiment, using a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format includes: Pre-configuring plug-ins corresponding to communication protocols; Determining the communication protocol type by parsing the data packet header or content features of the Internet of Things data; Invoking the corresponding plug-in according to the communication protocol type and using the corresponding plug-in to parse the Internet of Things data to obtain the corresponding initial data.

[0006] In an optional embodiment, pre-configuring plug-ins corresponding to communication protocols includes: Configuring corresponding plug-ins according to the communication protocol adopted by the Internet of Things data and storing the plug-ins in a preset plug-in storage path; Write the plugin storage path and the information of the plugin into a configuration file; Set to read the configuration file each time it starts to load the plugin.

[0007] In an alternative embodiment, extracting feature data from the initial data and determining an MQTT topic according to the type of the feature data includes: Generating a device fingerprint vector of the initial data by using a pre-trained meta-network, where the device fingerprint vector characterizes the device information and behavior model of the source device of the initial data; According to the matching degree between the device fingerprint vector and the feature extraction path, allocating a corresponding target feature extraction path for the initial data, where the feature extraction path is used to extract feature data from the initial data; According to the correspondence between the type of preset feature data and the MQTT topic, allocating an MQTT topic for the feature data.

[0008] In an alternative embodiment, according to the matching degree between the device fingerprint vector and the feature extraction path, allocating a corresponding target feature extraction path for the initial data includes: Using a pre-constructed dynamic feature extraction backbone to allocate a corresponding target feature extraction path for the initial data according to the matching degree between the device fingerprint vector and the feature extraction path; The dynamic feature extraction backbone includes multiple feature extraction paths, and multiple feature extraction paths all include an adjustable attention mechanism, and the attention mechanism is used to allocate computing resources according to the importance of the current task; The dynamic feature extraction backbone determines a corresponding target feature extraction path according to the correspondence between the device fingerprint vector and the feature extraction path, and adjusts the attention mechanism of the target feature extraction path to allocate a first computing resource for processing the initial data to the target feature extraction path.

[0009] In an alternative embodiment, multiple feature extraction paths include: A lightweight path for extracting feature values, where the feature values include mean, variance, and extreme values; A composite feature path for extracting time series features, frequency domain features, and semantic features; A diagnostic path for extracting anomaly detection features, where the anomaly detection features include wavelet packet decomposition features and isolated forest enhancement features.

[0010] In a second aspect, the present invention provides an MQTT-based Internet of Things data transmission system, including: A parsing module for parsing Internet of Things data using multiple communication protocols by using a dynamic plugin to obtain initial data in a unified format; A processing module, configured to extract feature data from the initial data and determine an MQTT topic according to the type of the feature data; A publishing module, configured to encapsulate the feature data into an MQTT message according to the MQTT topic and publish the MQTT message to subscribers.

[0011] In an optional embodiment, the parsing module includes: A plugin configuration unit, configured to pre-configure plugins corresponding to communication protocols; A protocol determination unit, configured to determine the type of communication protocol by parsing the packet header or content features of the Internet of Things data; A plugin invocation unit, configured to invoke a corresponding plugin according to the type of communication protocol and use the corresponding plugin to parse the Internet of Things data to obtain corresponding initial data.

[0012] In a third aspect, a device is provided, including: A memory, configured to store an Internet of Things data transmission program based on MQTT; A processor, configured to implement the steps of the Internet of Things data transmission method based on MQTT provided in the first aspect when executing the Internet of Things data transmission program based on MQTT.

[0013] In a fourth aspect, a computer-readable storage medium is provided, on which an Internet of Things data transmission program based on MQTT is stored. When the Internet of Things data transmission program based on MQTT is executed by a processor, the steps of the Internet of Things data transmission method based on MQTT provided in the first aspect are implemented.

[0014] The beneficial effects of the present invention are as follows. The Internet of Things data transmission method, system, device and storage medium based on MQTT provided by the present invention use dynamic plugins to unify the formats of different Internet of Things data, and then extract feature data from the Internet of Things data in the unified format. Compared with the original Internet of Things data, the amount of feature data is greatly reduced and there is no redundant data. Therefore, only transmitting the feature data greatly reduces the data transmission volume. In addition, MQTT topics are assigned to the feature data and published to data consumers through the MQTT mechanism, without maintaining the mapping relationship between device IDs and data consumers and not relying on classification rules, thus improving the data transmission efficiency.

[0015] In addition, the design principle of the present invention is reliable, the structure is simple, and it has a very wide application prospect. Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0017] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention.

[0018] Figure 2 It is a schematic block diagram of the system according to an embodiment of the present invention.

[0019] Figure 3 It is a schematic structural diagram of a device provided by an embodiment of the present invention. Detailed implementation manners

[0020] In order to enable those skilled in the art of the present technology to better understand the technical solutions in the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention. The terms used in the description of the present invention in this specification are only for the purpose of describing specific embodiments, and are not intended to limit the present invention.

[0022] The MQTT-based Internet of Things data transmission method provided by the embodiments of the present invention is executed by a computer device. Correspondingly, the MQTT-based Internet of Things data transmission system runs in the computer device.

[0023] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention. Among them, Figure 1 The execution subject can be an MQTT-based Internet of Things data transmission system. According to different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0024] As Figure 1 shown, the method includes: S1. Use a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format; S2. Extract feature data from the initial data and determine the MQTT topic according to the type of the feature data; S3. Encapsulate the feature data into an MQTT message according to the MQTT topic, and publish the MQTT message to the subscriber.

[0025] In an embodiment of the present invention, based on step S1, a possible embodiment will be given below to non - restrictively elaborate on its specific implementation.

[0026] S101. Pre - configure a plugin corresponding to the communication protocol.

[0027] For different communication protocols, such as the widely used MQTT, the lightweight CoAP, and the LoRaWAN suitable for long - distance low - power communication, etc., independent plugins need to be developed. Taking the MQTT plugin as an example, its core function is to accurately extract key field information from the received data strictly according to the MQTT protocol specification. For example, the device ID, which is the unique identifier for different devices, is obtained through a specific protocol position or format; sensor values, such as the temperature data collected by a temperature sensor, need to be parsed according to the relevant data structures and encoding methods specified in the MQTT protocol; the timestamp is used to record the moment when the data is generated, and it also needs to be accurately extracted from the data packet according to the MQTT protocol standard to ensure the timeliness and time series of the data. For the CoAP plugin, it is a similar principle, extracting the corresponding data fields according to the CoAP protocol specification.

[0028] Plugin management mechanism: Classify and store plugins by type. For example, the preset / plugins / mqtt / directory is specifically used to store MQTT - related plugins. The internal structure of each plugin is also relatively complex. It contains a binary parsing library, which is a key component for efficiently parsing binary - format data. It converts the received binary data into understandable information according to the binary encoding rules of a specific protocol. At the same time, it is also equipped with a configuration file, which plays a crucial role in the normal operation of the plugin. Configuration file: The configuration file details many key information of the plugin. The plugin path clarifies the specific location of the plugin in the storage system, facilitating the system to quickly locate and load it when needed. The supported protocol version information is very important because different versions of the protocol may have differences in functions, data formats, etc. Recording this information can ensure the compatibility of the plugin with the actual protocol version used. The data field mapping rules are even more essential. For example, in some raw data, the temperature field may be named temp_raw, while in the whole system, the standard field temperature is uniformly used to represent temperature. By recording this mapping relationship in the configuration file, when the plugin processes data, it can accurately convert the original non - standard field name into the standard field, facilitating subsequent unified data processing and analysis.

[0029] Dynamic loading: When the system starts up, it will automatically initiate a scanning process to traverse the preset storage paths of configuration files and read the configuration files one by one. During the reading process, the system will perform integrity verification on the plugins, checking whether key components of the plugins are missing, whether the binary parsing libraries are damaged, whether the configuration files are complete and correct, etc. Only the plugins that pass the integrity verification will be successfully loaded into memory, preparing for subsequent processing of IoT data. In addition, the system also supports the hot update function, which means that when a new plugin version is released while the system is running continuously, the old plugin version can be directly replaced without restarting the entire system. The specific implementation method is that when the system detects a plugin update, it will first pause the part of the business process related to the plugin, load the new plugin into memory according to the established loading process, and then update the relevant business call logic to ensure that the new plugin can seamlessly replace the old plugin and continue to provide services to the system, greatly ensuring the maintainability of the system during continuous operation.

[0030] Specifically, it includes: Plugin standard design: Develop independent plugins for each communication protocol (such as MQTT, CoAP, LoRaWAN). Each plugin contains core functions: extracting key fields (such as device ID, sensor value, timestamp) according to the protocol specifications.

[0031] Plugin management mechanism: Storage structure: All plugins are classified and stored in a preset directory (such as / plugins / mqtt / ) according to the protocol type. Each plugin contains a binary parsing library and a configuration file; Configuration file: Records the plugin path, supported protocol version, and data field mapping rules (such as mapping the original temperature field temp_raw to the standard field temperature); Dynamic loading: The system automatically scans the configuration files at startup, loads them into memory after verifying the plugin integrity, and supports hot updates (replacing the plugin version without restarting).

[0032] This kind of dynamic plugin has the following advantages: For new protocols, only the corresponding plugins need to be developed, and the deployment time is shortened from the weekly level to the minute level; The plugin isolation design avoids mutual interference between protocol parsing modules; The dynamic loading mechanism ensures the maintainability of the system during continuous operation.

[0033] S102. Determine the communication protocol type by parsing the packet header or content features of the IoT data.

[0034] After completing the plug-in configuration, enter the S102 phase, that is, determine the communication protocol type by parsing the packet header or content features of the Internet of Things data. This parsing process is a step-by-step and multi-level analysis process.

[0035] Basic feature matching: Port number identification: Different communication protocols usually use specific default ports for data transmission. For example, the MQTT protocol defaults to using port 1883. When the system receives a packet, it will first check the port number used by the packet. If the port number is 1883, then it is initially judged that the packet may be transmitted based on the MQTT protocol. However, this is only a preliminary and quick screening basis because the port number may be modified manually or there may be port multiplexing. Therefore, other features need to be further analyzed. The CoAP protocol defaults to port 5683. Similarly, if the port number of the received packet is 5683, the CoAP protocol will also be considered.

[0036] Protocol identification field detection: Each communication protocol has a unique protocol identification field in the packet. Taking MQTT as an example, if the first byte of its fixed header is 0x10, it means this is a CONNECT message. By detecting such specific identification fields, it is possible to more accurately judge whether the packet follows the MQTT protocol. The positions and contents of the identification fields of different protocols are defined by their specifications, and the system will detect the fields at specific positions in the packet according to these specifications.

[0037] Content pattern analysis: Parse the packet header structure: The packet header structures of different protocols have their own characteristics. For example, the URI path format of the CoAP protocol is usually in the form of / sensors / temperature. When the system parses the packet, it will analyze the URI path in the packet header. If it conforms to the format specified by the CoAP protocol, then it can be further confirmed that the packet is related to the CoAP protocol. By parsing and comparing each field in the packet header structure, more clues about the communication protocol can be obtained.

[0038] Statistical byte entropy value to judge the encryption type: In some cases, data may be transmitted encrypted. By statistically calculating the entropy value of the packet bytes, the encryption type can be initially judged. For example, the byte entropy value of data encrypted by AES is significantly higher than that of plaintext data. The system will calculate the byte entropy value of the received packet and compare the calculation result with the known entropy value range of the encryption type. If the entropy value is near the entropy value range of AES-encrypted data, then it can be speculated that the packet may use the AES encryption method. This also has a certain auxiliary effect on determining the communication protocol type because different protocols may have preferences or regulations in data encryption processing.

[0039] Dynamic decision-making process: Quickly filter by port number: The system first uses the port number, a simple and quickly accessible piece of information, to perform a preliminary screening to narrow down the possible communication protocols. If the port number matches the default port of a common protocol, that protocol will be considered a possible candidate.

[0040] Further analysis of protocol signature strings for unrecognizable packets: For packets whose port numbers cannot clearly identify the protocol type, the system further analyzes the protocol signature strings within the packet. For example, the MQTT protocol's KEEPALIVE mechanism contains specific strings or data structures within the packet to represent relevant information about the mechanism. The system searches for these known protocol signature strings within the packet and determines the protocol type by matching the signature strings.

[0041] Optimizing Identification Results by Incorporating Historical Traffic Patterns: The system also considers historical traffic patterns to optimize protocol identification results. For example, if a device has been periodically sending packets conforming to the LoRaWAN protocol format, then upon receiving a packet from that device again, even if the port number and the characteristics of the current packet cannot clearly determine the protocol type, historical traffic patterns can be used to infer that the packet is still transmitted using the LoRaWAN protocol. By combining multiple analysis methods and referencing historical data, the communication protocol type used by IoT data can be more accurately determined.

[0042] S103. Call the corresponding plug-in according to the communication protocol type, and use the corresponding plug-in to parse the IoT data to obtain corresponding initial data.

[0043] Select the corresponding parsing plug-in based on the protocol identification result (for example, if it is identified as MQTT, load mqtt_parser), and pass the original data packet and device metadata (such as device ID and network topology information) to the plug-in.

[0044] The plug-in's data parsing process includes: Field extraction: Parse the protocol header to obtain control information (such as MQTT QoS level and reserved flag); extract sensor values from the payload data (such as temperature 23.5°C and humidity 65% RH).

[0045] Standardization conversion: Convert device-specific formats into standard fields (e.g., converting temp_raw to temperature); add contextual information (e.g., automatically adding altitude = 150m based on the device location).

[0046] Quality verification: Filter invalid data (e.g., -999°C returned by the temperature sensor is significantly abnormal); repair missing values (e.g., complete the missing humidity data through time series prediction).

[0047] In an embodiment of the present invention, based on step S2, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation.

[0048] S201. Use a pre-trained meta-network to generate a device fingerprint vector for the initial data, where the device fingerprint vector characterizes the device information and behavior model of the source device of the initial data.

[0049] Extract device metadata (such as device ID, protocol type, timestamp, data upload period) and sensor values (such as temperature, humidity) from the initial data; Unify data with different sampling frequencies into a 1-second time window (e.g., downsample LoRaWAN device data to 1Hz).

[0050] The meta-network architecture includes: Input layer: Receive the standardized device status vector (dimension 128, including device type, historical error codes, network latency, etc.); Feature fusion module: Hardware feature encoder: Use the Embedding layer to convert the device model (such as "STM32F407VG") into a 64-dimensional vector; Behavior pattern encoder: The LSTM network processes time-series behavior data (such as the fluctuation pattern of data reporting intervals); Output layer: Generate a 128-dimensional device fingerprint vector (the first 64 dimensions characterize physical attributes, and the last 64 dimensions characterize behavior features).

[0051] S202. According to the matching degree between the device fingerprint vector and the feature extraction path, assign a corresponding target feature extraction path to the initial data, where the feature extraction path is used to extract feature data from the initial data.

[0052] Use a pre-constructed dynamic feature extraction backbone to assign a corresponding target feature extraction path to the initial data according to the matching degree between the device fingerprint vector and the feature extraction path.

[0053] Among them, the matching degree calculation method uses similarity calculation, and the cosine similarity is used to calculate the matching degree between the device fingerprint and the predefined path template (the calculation amount <5ms): Decision threshold: matching degree > 0.9 → directly select the optimal path; 0.7 ≤ matching degree ≤ 0.9 → trigger the hybrid path (lightweight + diagnostic); matching degree < 0.7 → start the adaptive exploration mode.

[0054] The dynamic feature extraction backbone includes multiple feature extraction paths, and each of the multiple feature extraction paths includes an adjustable attention mechanism, which is used to allocate computing resources according to the importance of the current task; The dynamic feature extraction backbone determines the corresponding target feature extraction path according to the correspondence between the device fingerprint vector and the feature extraction path, and adjusts the attention mechanism of the target feature extraction path to allocate the first computing resources for processing the initial data to the target feature extraction path.

[0055] Among them, the resource allocation formula:

[0056] Among them, is the proportion of computing resources allocated to the i-th feature extraction path, is the matching degree between the current path and the device fingerprint vector, is the adjustment factor, and n is the total number of optional feature extraction paths, is the matching degree between the j-th feature extraction path and the device fingerprint vector.

[0057] Through the matching degree-driven dynamic weight allocation, the resource allocation formula achieves the following goals: paths with high matching degrees obtain more resources to improve processing efficiency; avoid inefficient paths from occupying too many computing resources; dynamically balance accuracy and performance according to real-time requirements.

[0058] When it is detected that the CPU load > 80%, the weight coefficient of the high-computation path is automatically reduced.

[0059] The multiple feature extraction paths include: a lightweight path for extracting feature values, where the feature values include mean, variance, and extreme values; a composite feature path for extracting time series features, frequency domain features, and semantic features; and a diagnostic path for extracting anomaly detection features, where the anomaly detection features include wavelet packet decomposition features and isolated forest enhancement features.

[0060] S203. According to the correspondence between the preset type of feature data and the MQTT topic, allocate MQTT topics to the feature data.

[0061] Construct a feature-topic mapping table and use a Bloom filter to achieve O(1) time complexity for topic query. Implement batch publishing (merge once every 10 seconds) for high-frequency feature data (such as 50 samples per second from a vibration sensor).

[0062] First, the correspondence between the types of feature data and MQTT topics needs to be predefined. This is usually determined during the system design phase. For example, in an industrial Internet of Things scenario, it may be stipulated that the temperature feature data of a device corresponds to the "factory / device / temperature" topic, and the pressure feature data of the device corresponds to the "factory / device / pressure" topic, etc. This correspondence will be stored in the form of a certain data structure, which may be a simple list of key-value pairs, where the key is the feature data type and the value is the corresponding MQTT topic. During the actual operation process, when the system receives new feature data, it will look up in the preset correspondence data structure according to the type information carried by the data. Taking the Python language as an example, if a dictionary is used to store this correspondence: type_topic_mapping = { "temperature": "factory / device / temperature", "pressure": "factory / device / pressure"}, new_data_type = "temperature" # Assume the type of newly received data is temperature; mqtt_topic = type_topic_mapping[new_data_type]。

[0063] In this way, the corresponding MQTT topic can be quickly and accurately assigned to the feature data.

[0064] Construct a feature-topic mapping table and use a Bloom filter to achieve fast topic query.

[0065] Constructing a feature-topic mapping table is to more efficiently manage and query the correspondence between feature data and MQTT topics. The mapping table can be a complex data structure that not only stores the correspondence between feature data types and MQTT topics, but may also contain some auxiliary information, such as the publishing frequency and data format of the feature data. To achieve topic queries with O(1) time complexity, a Bloom filter is introduced. A Bloom filter is a probabilistic data structure with high space efficiency. Its principle is to map an element to multiple positions in a bit array through multiple hash functions and mark them as 1. When querying, the same hash functions are used to calculate the element to be queried, and check whether the corresponding positions in the bit array are all 1. If so, it is very likely that the element exists (there is a certain false positive rate, but the false positive rate can be reduced by reasonably setting the number of hash functions and the size of the bit array). When constructing a system that combines a feature-topic mapping table and a Bloom filter, first initialize a Bloom filter object and set the appropriate number of hash functions and the size of the bit array. For example, in Python, use the pybloomfiltermmap library to create a Bloom filter: from pybloomfiltermmap import BloomFilter, # Create a Bloom filter that can hold 1000 elements with a false positive rate of 0.01, bloom_filter = BloomFilter(capacity = 1000, error_rate = 0.01).

[0066] Then, traverse each feature data type in the feature-topic mapping table and add it to the Bloom filter.

[0067] When querying the MQTT topic corresponding to a certain feature data type, first quickly judge whether the feature data type may exist in the mapping table through the Bloom filter. If the Bloom filter judges that it exists, then perform an accurate search in the mapping table.

[0068] In this way, the efficiency of topic queries is greatly improved, especially when dealing with a large amount of feature data, which can significantly reduce the query time. Implement batch publishing for high-frequency feature data: For high-frequency feature data, such as 50 samples per second from a vibration sensor, if each sample is published immediately, a large number of MQTT messages will be generated, increasing the network load and system overhead. Therefore, a batch publishing strategy needs to be implemented. The system will set a timer with a period of 10 seconds. Within each period, the system will cache the vibration sensor data received. A list can be used to store this data. When the vibration sensor data is received each time, it will be added to the cache list. When the 10-second timer expires, the system will process the data in the cache list by merging. The merging method can be based on actual requirements. For example, the data can be calculated for the mean value, summed up, or the list can be serialized as a whole directly.

[0069] Through this batch publishing strategy, the sending frequency of MQTT messages is reduced, effectively reducing the network load and the consumption of system resources, while also meeting the need for summarizing and publishing high-frequency feature data within a certain time period.

[0070] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to non-restrictively elaborate on its specific implementation scheme.

[0071] The data publishing process of MQTT (Message Queuing Telemetry Transport) involves multiple links. The following is a detailed natural language description: Data preprocessing: Before publishing the data through MQTT, data encryption is performed, and a specific encryption algorithm is used to ensure the security of the data.

[0072] Connect to the MQTT Broker: The MQTT Broker is an intermediate server for message transmission, responsible for receiving and distributing messages. The publishing end needs to establish a connection with the MQTT Broker. When connecting, the address and port number of the Broker need to be specified. If the Broker has set access permissions, a username and password also need to be provided. When the connection is successful or fails, corresponding processing mechanisms will be triggered. For example, a success prompt will be received when the connection is successful, and the failure reason and error code will be displayed when the connection fails.

[0073] MQTT message encapsulation: After the connection is established, the preprocessed data needs to be encapsulated into an MQTT message. An MQTT message mainly consists of two parts, the topic and the payload. The topic is a string used to identify the message category, usually using a hierarchical structure, with each level separated by a slash. For example, "home / room1 / temperature" represents the temperature-related message in room 1 of the home; the payload is the actual effective data content to be transmitted, that is, the preprocessed data.

[0074] Set QoS and retained messages: According to the requirements of the service for message transmission reliability and status retention, set the Quality of Service (QoS) and retained flag of the message. There are three levels of QoS. QoS0 means at most once delivery, and the message may be lost; QoS1 means at least once delivery, and the message may be repeated; QoS2 means exactly once delivery, which can ensure that the message is delivered only once. If the retained message flag is set, the MQTT Broker will retain the latest message with this flag. When a new subscriber subscribes to this topic, the Broker will immediately send this retained message to the new subscriber.

[0075] Publish a message to a specified topic: After completing the above settings, publish the encapsulated MQTT message to the specified topic. The publish operation can be performed asynchronously to improve the publish performance, so that the publisher does not need to wait for the message to be completely sent successfully before continuing to execute other tasks.

[0076] Process the publish result: After the message is published, the publisher will receive feedback on the publish result. Determine whether the publish is successful based on the returned error code. If it is a successful error code (such as MQTT_ERR_SUCCESS), it means the message is published successfully; if it is other error codes, it represents a publish failure, and corresponding error handling needs to be performed according to the specific error code, such as network connection problems, message queue full, etc.

[0077] MQTT Broker message distribution: After the MQTT Broker receives the message sent by the publisher, it will distribute the message to all clients subscribed to this topic according to the message topic and the topic matching rules (supporting wildcards "+" to match a single level and "#" to match multiple levels).

[0078] Subscriber processing: The client (such as a cloud server, monitoring system, other devices, etc.) that subscribes to the relevant topic will trigger the corresponding processing logic when receiving the message. For example, the cloud server may store and analyze the received data; the monitoring system will monitor the device status in real time according to the received data; other devices may perform operations such as communication and control between devices according to the message content.

[0079] In some embodiments, the MQTT-based Internet of Things data transmission system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the MQTT-based Internet of Things data transmission system can be stored in the memory of the computer device and executed by at least one processor to execute (see details in Figure 1 description) the functions of MQTT-based Internet of Things data transmission.

[0080] In this embodiment, the MQTT-based Internet of Things data transmission system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 shown. The module referred to in the present invention means a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0081] The parsing module is used to parse Internet of Things data using multiple communication protocols with dynamic plugins to obtain initial data in a unified format; The processing module is used to extract feature data from the initial data and determine the MQTT topic according to the type of the feature data; The publishing module is used to encapsulate the feature data into an MQTT message according to the MQTT topic and publish the MQTT message to subscribers.

[0082] Optionally, as an embodiment of the present invention, the parsing module includes: The plugin configuration unit is used to pre-configure plugins corresponding to communication protocols; The protocol determination unit is used to determine the communication protocol type by parsing the data packet header or content features of the Internet of Things data; The plugin invocation unit is used to invoke the corresponding plugin according to the communication protocol type and parse the Internet of Things data with the corresponding plugin to obtain the corresponding initial data.

[0083] Figure 3 The MQTT-based Internet of Things data transmission method provided by the embodiments of this application can be applied to devices. Those skilled in the art can understand that the device structure involved in the embodiments of the present invention does not constitute a limitation on the device. The device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In the embodiments of the present invention, the device includes but is not limited to laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described herein and / or claimed.

[0084] Among them, the device 300 may include: a processor 310, a memory 320, and a communication unit 330. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation on the present invention. It can be a bus structure, a star structure, and may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0085] Among them, the memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 can execute some or all of the steps in the above method embodiments.

[0086] The processor 310 is the control center of the storage device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 320, and calling the data stored in the memory, it executes various functions of the electronic device and / or processes data. The processor may be composed of an integrated circuit (IC). For example, it may be composed of a single packaged IC, or may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may only include a central processing unit (CPU). In the embodiment of the present invention, the CPU may be a single arithmetic core or may include multiple arithmetic cores.

[0087] The communication unit 330 is used to establish a communication channel so that the storage device can communicate with other devices. It receives user data sent by other devices or sends user data to other devices.

[0088] The present invention also provides a computer storage medium. Among them, the computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in the embodiments provided by the present invention. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0089] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc, etc., various media that can store program codes, including several instructions to enable a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0090] For the same or similar parts among the various embodiments in this specification, reference can be made to each other. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the descriptions in the method embodiments.

[0091] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of systems or modules can be in electrical, mechanical or other forms.

[0092] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place, or they can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, in each embodiment of the present invention, the various functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0094] Although the present invention has been described in detail by reference to the accompanying drawings and in conjunction with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions should all be within the scope of the present invention. / Any person skilled in the art within the technical scope disclosed by the present invention can easily conceive of changes or substitutions, which should all be covered by the protection scope of the present invention.

Claims

1. An Internet of Things data transmission method based on MQTT, characterized in that, Including: Using a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format; Extracting feature data from the initial data and determining an MQTT topic according to the type of the feature data; Encapsulating the feature data into an MQTT message according to the MQTT topic and publishing the MQTT message to subscribers.

2. The method according to claim 1, wherein Using a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format, including: Pre-configuring a plug-in corresponding to the communication protocol; Determining the communication protocol type by parsing the data packet header or content feature of the Internet of Things data; Invoking the corresponding plug-in according to the communication protocol type and using the corresponding plug-in to parse the Internet of Things data to obtain the corresponding initial data.

3. The method according to claim 2, characterized in that, Pre-configuring a plug-in corresponding to the communication protocol, including: Configuring a corresponding plug-in according to the communication protocol adopted by the Internet of Things data and storing the plug-in to a preset plug-in storage path; Writing the plug-in storage path and the information of the plug-in into a configuration file; Setting to read the configuration file each time starting up to load the plug-in.

4. The method according to claim 1, wherein Extracting feature data from the initial data and determining an MQTT topic according to the type of the feature data, including: Using a pre-trained meta-network to generate a device fingerprint vector of the initial data, where the device fingerprint vector characterizes the device information and behavior model of the source device of the initial data; Allocating a corresponding target feature extraction path for the initial data according to the matching degree between the device fingerprint vector and the feature extraction path, where the feature extraction path is used to extract feature data from the initial data; Allocating an MQTT topic for the feature data according to the corresponding relationship between the preset type of the feature data and the MQTT topic.

5. The method according to claim 4, characterized in that, Allocating a corresponding target feature extraction path for the initial data according to the matching degree between the device fingerprint vector and the feature extraction path, including: Using a pre-constructed dynamic feature extraction backbone to allocate a corresponding target feature extraction path for the initial data according to the matching degree between the device fingerprint vector and the feature extraction path; The dynamic feature extraction backbone includes multiple feature extraction paths, and multiple feature extraction paths all include an adjustable attention mechanism, and the attention mechanism is used to allocate computing resources according to the current task importance; The dynamic feature extraction backbone determines the corresponding target feature extraction path according to the corresponding relationship between the device fingerprint vector and the feature extraction path, and adjusts the attention mechanism of the target feature extraction path to allocate a first computing resource for processing the initial data to the target feature extraction path.

6. The method according to claim 5, wherein The multiple feature extraction paths include: A lightweight path for extracting feature values, where the feature values include mean, variance, and extreme values; A composite feature path for extracting time series features, frequency domain features, and semantic features; A diagnosis path for extracting anomaly detection features, where the anomaly detection features include wavelet packet decomposition features and isolated forest enhancement features.

7. An Internet of Things data transmission system based on MQTT, characterized in that, Including: A parsing module for using a dynamic plug-in to parse Internet of Things data using multiple communication protocols to obtain initial data in a unified format; A processing module, configured to extract feature data from the initial data and determine an MQTT topic according to the type of the feature data; A publishing module, configured to encapsulate the feature data into an MQTT message according to the MQTT topic and publish the MQTT message to subscribers.

8. The system according to claim 7, characterized in that, The parsing module includes: A plugin configuration unit, configured to pre-configure a plugin corresponding to a communication protocol; A protocol determination unit, configured to determine the type of the communication protocol by parsing the packet header or content feature of the Internet of Things data; A plugin invocation unit, configured to invoke a corresponding plugin according to the type of the communication protocol and parse the Internet of Things data by using the corresponding plugin to obtain corresponding initial data.

9. A device, characterized in that, It includes: A memory, configured to store an Internet of Things data transmission program based on MQTT; A processor, configured to implement the steps of the Internet of Things data transmission method based on MQTT according to any one of claims 1-6 when executing the Internet of Things data transmission program based on MQTT.

10. A computer-readable storage medium storing a computer program, characterized in that, An Internet of Things data transmission program based on MQTT is stored on the readable storage medium, and when the Internet of Things data transmission program based on MQTT is executed by a processor, the steps of the Internet of Things data transmission method based on MQTT according to any one of claims 1-6 are implemented.

Images