Multi-protocol fusion edge message communication system

Abstract

The invention discloses a multi-protocol fusion edge message communication system, and particularly relates to the field of message communication. The system comprises a protocol access layer, a message abstraction layer, a message scheduling layer and a storage service layer. The protocol access layer comprises an MQTT gateway module and an HTTP gateway module, the MQTT gateway module processes all connections and messages related to the MQTT protocol, and the HTTP gateway module provides message publishing and subscription services based on the HTTP protocol and supports three working modes of RESTful API interface, server push event SSE real-time subscription and polling subscription. The message abstraction layer defines a standardized message structure and can completely represent message semantics of the MQTT protocol and the HTTP protocol, the message scheduling layer unifies message agency, adopts a singleton mode design and unifies and manages client connection, subscription relation and message distribution of all protocols, seamless fusion of the MQTT protocol and the HTTP protocol is achieved, and the message scheduling layer has the advantages of being simple in structure, convenient to use and high in efficiency. Uniform message formats and semantic representations are provided, and multiple message subscription modes are supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of message communication, and in particular to a multi-protocol fusion edge message communication system. BACKGROUND

[0002] The message communication systems existing in the market at present are mainly divided into the following categories: Single-protocol MQTT message broker system The existing MQTT message broker systems such as Eclipse Mosquitto, EMQ X, HiveMQ, etc. adopt a standard MQTT broker architecture, including a client connection management module, a topic subscription management module, a message routing and forwarding module, and a session state management module. The working principle is as follows: a client accesses a broker server through a TCP connection, sends a CONNECT packet to establish a session, and then subscribes to the topics of interest through a SUBSCRIBE packet. When a client publishes a message to the corresponding topic, the broker server forwards the message to all clients subscribing to the topic.

[0003] Single-protocol HTTP message system The message system based on the HTTP protocol usually adopts RESTful API or WebSocket technology to implement, including a message publishing interface module, a subscription management module, a long connection maintenance module, and a message pushing module. The working principle is as follows: a client publishes a message to a specified API endpoint through an HTTP POST request, and the server stores the message after receiving it; a subscription client receives real-time message pushing through WebSocket long connection or Server-Sent Events (SSE) technology.

[0004] Drawbacks of the prior art Protocol isolation is serious: MQTT devices cannot directly receive messages published by HTTP clients, and HTTP clients cannot subscribe to messages published by MQTT devices, requiring the deployment of an independent protocol conversion gateway, which increases system complexity and maintenance costs.

[0005] Message format is not unified: MQTT messages adopt a binary payload format, with QoS (Quality of Service level), retain (retained message) attributes, etc.; HTTP messages usually adopt JSON format, lack QoS guarantee mechanism, and the message semantics of the two protocols are inconsistent.

[0006] Subscription mode is limited: HTTP clients can only obtain messages through polling or rely on WebSocket long connection, and cannot flexibly subscribe to multiple topics using wildcards like MQTT, and also lack offline message mechanism.

[0007] State management separation: The online state and subscription relationship of the MQTT client are managed by the MQTT broker, and the session state of the HTTP client needs to be managed separately, and all client states cannot be uniformly monitored and managed.

[0008] Limited scalability: Adding support for new protocols requires redesigning the entire message processing flow, and the system architecture lacks good scalability. SUMMARY

[0009] The purpose of the present application is to address the above-mentioned deficiencies, and to provide a multi-protocol integrated edge message communication system that seamlessly integrates MQTT and HTTP protocols, provides unified message format and semantic representation, supports multiple message subscription modes, and uniformly manages the connection state and subscription relationship of all protocol clients.

[0010] The present application specifically adopts the following technical solutions: The multi-protocol integrated edge message communication system includes a protocol access layer, a message abstraction layer, a message scheduling layer, and a storage service layer. The protocol access layer includes an MQTT gateway module and an HTTP gateway module, the MQTT gateway module handles all MQTT protocol related connections and message processing, including a connection management submodule, a protocol parsing submodule, and a message conversion submodule. The HTTP gateway module is the entrance of the entire message service system, which is based on the HTTP protocol and provides message publishing and subscription services for clients, and the HTTP gateway module includes three working modes: RESTful API interface, server push event SSE real-time subscription, and polling subscription. Mode one: RESTful API interface The RESTful API interface is an interface specification designed based on the HTTP protocol, which uses HTTP methods to implement resource operations and provides standard HTTP POST and GET interfaces for message publishing and querying, respectively. When the client needs to publish a message, it sends a POST request to the specified topic to achieve this. GET requests are mainly used to query historical messages or check topic status. Mode two: SSE real-time push SSE is a kind of real-time data push from server to client based on HTTP. Clients establish an SSE connection by sending a special HTTP request, and once the connection is established, the server keeps the connection open, and when new messages arrive for the topic the client is subscribed to, the server immediately pushes the messages to the client through the open connection, achieving real-time communication similar to MQTT. Mode Three Polling Subscription Polling subscription is a message acquisition mechanism suitable for HTTP clients that do not support long connections In this mode, the client periodically sends HTTP requests to the server to query new messages for the subscribed topic, and the server returns all new messages since the last query by the client after receiving the request. To improve the efficiency and accuracy of message acquisition, this submodule also supports paging query and message marking mechanism. Paging query refers to dividing a large number of new messages into multiple pages, and the client can acquire messages page by page to avoid network congestion or excessive client processing pressure caused by acquiring too many messages at once. The message marking mechanism marks messages that have been acquired by the client, so that the server can determine which messages have been acquired based on the marks when the client queries next time, thus returning only new messages that have not been marked to avoid repeated acquisition of messages and save network bandwidth and client processing resources. The message abstraction layer defines a standardized message structure that can fully represent the message semantics of both MQTT and HTTP protocols, specifically: Message unique identifier: a globally unique ID generated by the Snowflake algorithm to ensure traceability of messages (Snowflake-generated unique IDs have a time sequence that increases: since the ID contains a timestamp component, and time is constantly advancing, the generated ID will increase over time. This increasing feature makes the ID more efficient when stored in a database, making query and sorting operations more convenient and effectively improving system performance. The Snowflake algorithm generates IDs at a very fast speed, meeting the performance requirements for ID generation in high-concurrency scenarios. For example, in high-concurrency and large-volume generation scenarios, the Snowflake algorithm can quickly generate unique IDs without becoming a bottleneck for the system. Topic name: the target topic of the message, following the MQTT topic naming conventions, supporting hierarchical structure and wildcards; Message payload: the actual message content, supporting binary data and text data; Quality of service level: QoS concept inherited from MQTT protocol, defining the reliability level of message transmission; Reserved message flag: indicating whether the message should be retained on the server for subsequent subscribers to acquire; Message source: Identify the message source protocol MQTT or HTTP and client information; Timestamp: Creation time and reception time of the message; Custom attributes: Extensible key-value pairs, support for protocol-specific attribute extensions; Multi-format conversion engine: The multi-format conversion engine is responsible for converting between the unified message model and different protocol formats; The MQTT format converter converts the unified message model into the standard MQTT PUBLISH message format, including setting the fixed header message type, QoS level, and reserved flag, setting the variable header topic name, and the message payload. The conversion process maintains the integrity of all MQTT protocol attributes; The HTTP format converter converts the unified message model into the HTTP response format, and for SSE push, formats the message into the event stream format according to the SSE specification; The message scheduling layer unified message broker is designed using the singleton pattern, and uniformly manages all protocol client connections, subscription relationships, and message distribution, including the following three submodules; (1) The subscription relationship management submodule maintains a global topic subscription mapping table, recording all subscriber information under each topic, including subscriber type MQTT or HTTP client, subscription mode, and subscription time; supports MQTT wildcard subscription rules such as single-level wildcard + and multi-level wildcard #; (2) The message routing and distribution submodule is responsible for distributing received messages to all relevant subscribers. When a message arrives, the submodule queries the subscription mapping table to find all matching subscribers, and then calls the corresponding message push mechanism according to the subscriber type. For MQTT subscribers, send MQTT PUBLISH messages directly; for HTTP subscribers, use different push strategies according to their subscription mode SSE or round. (3) The client state unified management submodule maintains the connection state, session information, and statistical data of all clients; including client online / offline state, last active time, message transmission statistics, error count information, supports unified monitoring and management across protocols.

[0011] Preferably, the connection management submodule maintains all MQTT client TCP connections, handles client connection establishment, heartbeat detection, and disconnection reconnection, verifies client identity information when the client sends a CONNECT message, assigns a client identifier, and establishes a session state; The protocol analysis submodule is responsible for analyzing the MQTT message format, including the message type, QoS level, and reserved flag field of the fixed header, and the topic name and message identifier information of the variable header, and parsing the binary message into a structured message object; The message conversion submodule converts the parsed MQTT message into a unified message format within the system, preserving the unique properties of the MQTT protocol such as QoS level, reserved message flag, and repeated message flag, while adding message source identification, timestamp, and other system attributes.

[0012] The present application has the following beneficial effects: Seamless protocol interworking: The temperature sensor data published by the MQTT device can be directly displayed in real time by the web page of the HTTP client, and the control instructions sent by the HTTP client can also directly reach the MQTT device, realizing truly protocol-transparent communication.

[0013] Unified message semantics: Through the unified message model, the HTTP client can also enjoy the QoS guarantee mechanism of the MQTT protocol, ensuring reliable transmission of important messages even in unstable network conditions.

[0014] Flexible subscription mode: The HTTP client can use MQTT-style topic wildcards for subscription, such as subscribing to "sensors / + / temperature" to receive temperature data from all sensors, greatly improving the flexibility of subscription.

[0015] Reduced message transmission delay: Since the message is directly converted at the protocol access layer, multiple forwarding and protocol conversion in traditional schemes are avoided, and the end-to-end message transmission delay is reduced by more than 30%.

[0016] Improved system concurrency: The unified message broker can more effectively manage and schedule system resources, supporting more than 100,000 concurrent client connections, which is 60% higher than traditional separate architecture.

[0017] Optimized resource utilization: Unified subscription relationship management avoids repeated data structures and processing logic, improving memory usage efficiency by 25% and reducing CPU occupancy by 20%.

[0018] Simplified architecture: Reducing independent protocol conversion components, the system deployment is simplified from the original 5 independent services to a single service, greatly reducing the operation and maintenance complexity.

[0019] Unified monitoring: All protocol client states, message traffic, and error information are displayed in the unified management interface, facilitating system monitoring and troubleshooting.

[0020] Extensibility enhancement: New protocol support only needs to add corresponding gateway module in the protocol access layer, without modifying the core message processing logic, and the system extensibility is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Fig. 1 is a system overall architecture diagram, showing a four-layer separation architecture and the composition of each layer module; Fig. 2 is a message publishing process flowchart, showing the complete process from client publishing to subscriber receiving; Fig. 3 is a client subscription process flowchart, showing the subscription establishment process of different protocol clients. DETAILED DESCRIPTION

[0022] The specific embodiments of the present application will be further described below in conjunction with the drawings and specific embodiments: In conjunction with Figs. 1-3 , a multi-protocol integrated edge message communication system, including a protocol access layer, a message abstraction layer, a message scheduling layer, and a storage service layer.

[0023] The protocol access layer includes an MQTT gateway module and an HTTP gateway module. The MQTT gateway module handles all MQTT protocol-related connections and message processing, including a connection management submodule, a protocol parsing submodule, and a message conversion submodule.

[0024] The connection management submodule maintains all MQTT client TCP connections, handles client connection establishment, heartbeat detection, and disconnection reconnection, verifies client identity information when the client sends a CONNECT packet, assigns a client identifier, and establishes a session state.

[0025] The protocol parsing submodule is responsible for parsing MQTT packet formats, including message types, QoS levels, and reserved flag fields in the fixed header, and topic names and packet identifier information in the variable header, and parsing binary packets into structured message objects.

[0026] The message conversion submodule converts the parsed MQTT message into a unified message format within the system, retains MQTT protocol-specific properties such as QoS level, reserved message flag, and duplicate message flag, and adds message source identifier, timestamp, and other system properties.

[0027] The HTTP gateway module is the entrance to the entire message service system, which is based on the HTTP protocol and provides message publishing and subscription services for clients. In order to meet the needs of clients in different scenarios, the HTTP gateway module includes RESTful API interfaces, server push event SSE real-time subscription, and polling subscription in three working modes: Mode one RESTful API interface RESTful API interface is an interface specification designed based on HTTP protocol, which uses HTTP methods to implement operations on resources, and provides standard HTTP POST and GET interfaces for message publishing and querying, respectively; When the client needs to publish a message, it sends a POST request to the specified topic to achieve this; GET request is mainly used to query historical messages or check the status of the topic; Mode two SSE real-time push SSE is a kind of HTTP-based server pushing real-time data to the client; The client establishes an SSE connection by sending a special HTTP request, and once the connection is established, the server will keep the connection open, rather than closing the connection after responding like a normal HTTP request. When a new message arrives at the topic subscribed by the client, the server will immediately push the message to the client through the open connection, achieving a real-time communication effect similar to MQTT; MQTT (Message Queuing Telemetry Transport) is a lightweight publish / subscribe message transmission protocol, widely used in Internet of Things and other scenarios with high real-time requirements. The SSE real-time push submodule can achieve similar real-time communication effects, meaning it can play a role in many scenarios that require real-time message acquisition; such as real-time monitoring systems, instant notification services, etc. Mode three polling subscription Polling subscription is a message acquisition mechanism suitable for HTTP clients that do not support long connections In this mode, the client sends HTTP requests to the server at regular intervals to query new messages for the subscribed topic. The server returns all new messages since the last query by the client. To improve the efficiency and accuracy of message acquisition, the submodule also supports paging query and message marking mechanisms; Paging query means dividing a large number of new messages into multiple pages, and the client can acquire messages page by page to avoid network congestion or excessive client processing pressure caused by acquiring too many messages at once. The message marking mechanism marks messages that have been acquired by the client, so that the server can determine which messages have been acquired when the client queries next time, and only return new messages that have not been marked, avoiding repeated acquisition of messages and saving network bandwidth and client processing resources; These three working modes are designed for different client characteristics and application scenarios. The RESTful API interface is suitable for regular message publishing and historical message querying; SSE real-time push is suitable for clients that need to obtain messages in real time and support long connections; and polling subscription provides a reliable way to obtain messages for clients that do not support long connections. The three complement each other and together constitute the complete message service system of the HTTP gateway module.

[0028] The message abstraction layer defines a standardized message structure that can fully represent the message semantics of both MQTT and HTTP protocols, specifically: Unique Message Identifier: A globally unique ID generated using the Snowflake algorithm ensures message traceability. (The unique IDs generated by the Snowflake algorithm are time-sequentially increasing: because the ID includes a timestamp, and time progresses forward, the generated IDs increment over time. This increasing characteristic makes indexing more efficient and querying and sorting operations more convenient when storing IDs in the database, effectively improving system performance. The Snowflake algorithm generates IDs very quickly, meeting the performance requirements of ID generation in high-concurrency scenarios. For example, in scenarios with high concurrency and large-scale ID generation, the Snowflake algorithm can quickly generate unique IDs without becoming a system bottleneck.) Topic Name: The target topic of the message, following the MQTT topic naming conventions, supporting hierarchical structures and wildcards; Message payload: The actual message content, supporting both binary and text data; Quality of Service (QoS) level: A QoS concept inherited from the MQTT protocol, defining the reliability level of message transmission; Retain message flag: Indicates whether the message should be retained on the server for subsequent subscribers to retrieve; Message Source: Identifies the message source protocol (MQTT or HTTP) and client information; Timestamp: The time the message was created and the time it was received; Custom attributes: Extensible key-value pairs, supporting protocol-specific attribute extensions; Multi-format conversion engine: The multi-format conversion engine is responsible for converting between a unified message model and different protocol formats; The MQTT format converter transforms the unified message model into the standard MQTT PUBLISH message format, including setting the message type, QoS level, and reservation flags with fixed headers, setting the subject name with variable headers, and the message payload, while maintaining the integrity of all MQTT protocol attributes during the conversion process. The HTTP format converter converts the unified message model into an HTTP response format, and formats the message into an event stream format conforming to the SSE specification for SSE push; The message scheduling layer unifies the message broker and is designed in a singleton mode, which uniformly manages all protocol client connections, subscription relationships, and message distribution, including the following three submodules. (1) The subscription relationship management submodule maintains a global topic subscription mapping table, which records all subscriber information under each topic, including the subscriber type MQTT or HTTP client, subscription mode, and subscription time. It supports MQTT wildcard subscription rules, such as single-level wildcard + and multi-level wildcard #. (2) The message routing and distribution submodule is responsible for distributing received messages to all relevant subscribers. When a message arrives, this module queries the subscription mapping table to find all matching subscribers and then calls the corresponding message push mechanism according to the subscriber type. For MQTT subscribers, it directly sends MQTT PUBLISH packets. For HTTP subscribers, it uses different push strategies based on their subscription modes SSE or round. (3) The client state unified management submodule maintains the connection state, session information, and statistical data of all clients, including online / offline state, last active time, message transmission statistics, and error count information, supporting unified monitoring and management across protocols.

[0029] The client state unified management submodule maintains the connection state, session information, and statistical data of all clients, including online / offline state, last active time, message transmission statistics, and error count information, supporting unified monitoring and management across protocols.

[0030] This scheme sets up a four-layer separation architecture: protocol access layer, message abstraction layer, message scheduling layer, and storage service layer, realizing a protocol-independent message processing architecture.

[0031] Unified message model: A standardized data structure that can fully represent the semantics of different protocol messages, is the core foundation for protocol integration.

[0032] Multi-format conversion engine: Supports the mutual conversion of various message formats such as MQTT binary format, HTTP JSON format, and SSE event stream format.

[0033] Global subscription management: Cross-protocol unified subscription relationship management, supporting the application of MQTT wildcard rules in HTTP protocol.

[0034] Hybrid push mechanism: Simultaneously supports real-time push for MQTT, SSE push for HTTP, and polling for three message delivery modes.

[0035] Protocol conversion method: Technical method for uniformly converting MQTT PUBLISH message and HTTP POST request into internal message format, including field mapping rules and data type conversion algorithm.

[0036] Message model design: Data structure design of unified message model, including core attribute definition, extension mechanism, and version compatibility processing.

[0037] Subscription matching algorithm: Subscription matching algorithm based on topic wildcard, especially the application of MQTT wildcard rules to HTTP protocol.

[0038] Mixed push strategy: Algorithm for selecting different push mechanisms according to client type and subscription mode, including SSE connection management and polling cache strategy.

[0039] State synchronization mechanism: Unified management method for cross-protocol client state, including connection state mapping and session information synchronization.

[0040] Message format standardization: Complete message format conversion rules must be defined to ensure that message semantics are not lost between different protocols.

[0041] Subscription relationship consistency: Global subscription mapping table must be synchronized in real time to avoid message loss caused by inconsistent subscription relationships.

[0042] Connection state management: Connection life cycle of different protocol clients needs to be managed uniformly to handle abnormal disconnection and reconnection.

[0043] Performance optimization strategy: In high-concurrency scenarios, message routing and distribution algorithms need to be optimized to avoid performance bottlenecks.

[0044] Error handling mechanism: Establish a complete cross-protocol error handling and recovery mechanism to ensure system stability and reliability.

[0045] Of course, the above description is not a limitation of the invention, and the invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the scope of the invention should also be within the protection scope of the invention.

Claims

1. A multi-protocol converged edge message communication system, characterized by, The message service system includes a protocol access layer, a message abstraction layer, a message scheduling layer, and a storage service layer; The protocol access layer includes an MQTT gateway module and an HTTP gateway module. The MQTT gateway module processes all MQTT protocol-related connections and message processing, including a connection management submodule, a protocol analysis submodule, and a message conversion submodule; The HTTP gateway module is the entrance of the entire message service system. It is based on the HTTP protocol and provides message publishing and subscription services for clients. The HTTP gateway module includes three working modes: RESTful API interface, server push event SSE real-time subscription, and polling subscription: Mode one: RESTful API interface RESTful API interface is an interface specification designed based on the HTTP protocol. It uses HTTP methods to implement operations on resources and provides standard HTTP POST and GET interfaces for message publishing and querying, respectively. When a client needs to publish a message, it sends a POST request to the specified topic. GET requests are mainly used to query historical messages or check the status of a topic. Mode two: SSE real-time push SSE is a server push real-time data based on HTTP. The client establishes an SSE connection by sending a special HTTP request. Once the connection is established, the server keeps the connection open. When a new message arrives for a topic subscribed by the client, the server immediately pushes the message to the client through the open connection, achieving a real-time communication effect similar to MQTT. The SSE real-time push submodule plays a role in many scenarios that require real-time message acquisition. Mode three: Polling subscription Polling subscription is a message acquisition mechanism suitable for HTTP clients that do not support long connections. The client sends an HTTP request to the server at regular intervals to query new messages for the subscribed topic. The server returns all new messages since the last query by the client. To improve the efficiency and accuracy of message acquisition, the submodule supports page query and message marking mechanisms. Page query divides a large number of new messages into multiple pages, allowing the client to acquire messages page by page, avoiding network congestion or excessive client processing pressure caused by acquiring too many messages at once. The message marking mechanism marks messages that have been acquired by the client. When the client queries again, the server determines which messages have been acquired based on the marks and returns only new messages that have not been marked, avoiding duplicate message acquisition and saving network bandwidth and client processing resources. The message abstraction layer defines a standardized message structure that can fully represent the message semantics of both MQTT and HTTP protocols. Specifically: Message unique identifier: A globally unique ID generated using the snowflake algorithm to ensure message traceability. Topic name: The target topic of the message, following the MQTT topic naming specification and supporting hierarchical structure and wildcards. Message payload: the actual message content, supporting binary data and text data; Quality of service level: the QoS concept inherited from the MQTT protocol, defining the reliability level of message transmission; Retained message flag: indicating whether the message should be retained on the server for subsequent subscribers to obtain; Message source: identifying the message source protocol MQTT or HTTP and client information; Timestamp: the creation time and reception time of the message; Custom attributes: extensible key-value pairs, supporting protocol-specific attribute extensions; Multi-format conversion engine: The multi-format conversion engine is responsible for converting between the unified message model and different protocol formats. The MQTT format converter converts the unified message model into the standard MQTT PUBLISH message format, including setting the fixed header message type, QoS level, and retained flag, setting the variable header topic name, and the message payload, and maintaining the integrity of all MQTT protocol attributes during the conversion process. The HTTP format converter converts the unified message model into the HTTP response format, and for SSE push, formats the message into the event stream format that conforms to the SSE specification. The message scheduling layer unified message broker, designed in a singleton pattern, uniformly manages all protocol client connections, subscription relationships, and message distribution, including the following three submodules: (1) The subscription relationship management submodule maintains a global topic subscription mapping table, recording all subscriber information under each topic, including the subscriber type MQTT or HTTP client, subscription mode, and subscription time. Supporting MQTT wildcard subscription rules such as single-level wildcard + and multi-level wildcard #. (2) The message routing and distribution submodule is responsible for distributing received messages to all relevant subscribers. When a message arrives, this module queries the subscription mapping table to find all matching subscribers, and then calls the corresponding message push mechanism based on the subscriber type. For MQTT subscribers, send MQTT PUBLISH messages directly; for HTTP subscribers, use different push strategies based on their subscription mode SSE or round. (3) The client state unified management submodule maintains the connection state, session information, and statistical data of all clients, including online / offline state, last active time, message transmission statistics, and error count information, supporting unified monitoring and management across protocols.

2. The multi-protocol integrated edge message communication system of claim 1, wherein The connection management submodule maintains all MQTT client TCP connections, handles client connection establishment, heartbeat detection, and disconnection reconnection, verifies client identity information when the client sends a CONNECT message, assigns a client identifier, and establishes session state. The protocol analysis submodule is responsible for parsing MQTT message formats, including the fixed header message type, QoS level, and retained flag fields, as well as the variable header topic name and message identifier information, and parses binary messages into structured message objects. The message conversion sub-module converts the parsed MQTT message into a unified message format inside the system, retains the unique properties of the MQTT protocol such as the QoS level, the reserved message flag, the repeated message flag, and adds the message source identification and the timestamp system attribute.

Images