Method for accessing non-standard Zigbee protocol equipment and dynamically analyzing data based on zigbee gateway

Abstract

The invention belongs to the technical field of wireless communication, and particularly relates to a method for accessing non-standard Zigbee protocol equipment and dynamically analyzing data based on a zigbee gateway, which comprises the following steps of: S1, initiating a network access request by the non-standard Zigbee protocol equipment, and executing a standardized network access interaction process by a Zigbee communication sub-module; s2, according to a protocol rule loaded by an analysis engine, a protocol analysis module receives data allocated by a Zigbee communication sub-module, executes CRC verification and serial number de-duplication, and then performs data point reverse analysis on original data in a Hex format; s3, the data preprocessing module executes field extraction, data normalization and cache queue storage processing on the analyzed data; and S4, the data packaging module performs JSON formatting on the preprocessed data in the cache queue to generate a data packet containing a unique identifier of the equipment, a UTC timestamp and standardized service data, and the data packet is uploaded to a cloud end through a cloud communication sub-module of the network communication module.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to a method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway. Background Technology

[0002] ZigBee, as a low-power, short-range, and low-data-rate wireless communication technology, has been widely used in the Internet of Things (IoT) field. However, existing ZigBee gateway technologies have the following drawbacks when connecting non-standard ZigBee protocol devices: 1. Closed Ecosystem: Most existing Zigbee gateways only support standard Zigbee protocol devices within specific ecosystems, resulting in poor compatibility with non-standard devices. Different manufacturers and models of Zigbee devices may use different communication protocols and data formats, causing gateways to be unable to directly recognize and process the data from these devices.

[0003] 2. Manual Parsing: Existing solutions for connecting ecosystem devices typically require manually writing or modifying parsing code and flashing firmware to adapt to the device's communication protocol and data format when connecting new devices. This is not only time-consuming and labor-intensive, but also prone to introducing errors, increasing development costs and maintenance difficulty.

[0004] 3. Lack of flexibility: When the communication protocol or data format of a device changes, or when a new device protocol is added, the gateway needs to be recompiled and redeployed. This not only affects the system's flexibility and scalability, but may also cause devices to fail to update in a timely manner, impacting the user experience. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides a method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway, thereby solving the problems mentioned in the background technology.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The method includes the following steps: When a non-standard Zigbee protocol device initiates a network access request, the gateway's Zigbee communication submodule executes a standardized network access interaction process. According to the pre-loaded protocol rules, the data reported by the network access devices is verified, deduplicated, and dynamically parsed to obtain standardized business data; The parsed data is processed by field extraction and normalization, and then stored in a cache queue; Finally, the data in the queue is packaged into a standardized data packet and uploaded to the cloud.

[0007] Furthermore, the standardized network access interaction process described in step S1 is as follows: The gateway's coordinator broadcasts a permission command to join the network; Once the device to be connected to the network enters the network access state, it scans the designated channels in the order of priority channel first and then secondary channel, and sends beacon requests in sequence. After receiving a beacon request, the coordinator on the same channel sends a beacon response message. The devices waiting to join the network filter out the compatible networks and send an association request message. The association request message carries the device type, address and device capability information. The coordinator determines whether to allow network access based on the current resource status. If allowed, it returns an associated response message, which includes the target network's personal area network identifier and the short address assigned to the device. After a device successfully joins the network, it broadcasts a device announcement message to declare that it has joined the network. The device announcement message includes the device's long address, short address, and device capability information. The coordinator sends an attribute read request message to read the attributes of the device's underlying cluster. The device replies with the manufacturer name and model identifier through the attribute response message. The gateway identifies the device product type based on the manufacturer name and model identifier.

[0008] Furthermore, the data frame structure processed by the Zigbee communication submodule includes a preamble, frame control, vendor code, cluster identifier, data payload, and cyclic redundancy check.

[0009] Furthermore, the preamble and start-of-frame delimiter detection is used to verify the legality of the physical layer frame and locate the start position of the data. Specifically, it scans the binary stream and matches a 4-byte standard preamble and a 1-byte start-of-frame delimiter. If the detection passes, it returns the start position of the frame after skipping the preamble and start-of-frame delimiter. If no preamble and start-of-frame delimiter that meet the requirements are detected, it is determined to be an invalid data frame and is directly discarded.

[0010] Furthermore, the vendor code is extracted from the manufacturer identifier field of the Zigbee application layer protocol frame, and the extraction position is a preset fixed offset position of the application layer identifier in the application layer protocol frame; during extraction, the two bytes of data at this position are combined into a 16-bit vendor code by left-shifting the high bit by 8 bits and concatenating it with the low bit.

[0011] Furthermore, the parsing engine of the protocol parsing module supports parsing protocol files in two standardized formats: JSON and XML. The protocol file contains two core fields: device configuration file and security configuration. The device configuration file defines the manufacturer identifier, device type, protocol version, custom cluster, data points, and reverse control instructions. The data point definition includes name, data type, byte order, scaling factor, offset, unit, starting offset bytes, and verification rules. The security configuration field defines the AES-128-CCM encryption algorithm and the RSA-2048 digital signature algorithm.

[0012] Furthermore, the reverse control command includes a cluster identifier, a command identifier, and parameter configuration. The parameter configuration includes parameter name, encoding format, scaling factor, and offset. The parsing engine can generate a control message that the device can recognize based on the reverse control command. The control message is sent to the target device through the Zigbee communication submodule to achieve reverse control.

[0013] Furthermore, the cyclic redundancy check and sequence number deduplication mechanism specifically includes: The integrity of data frames is checked using a cyclic redundancy check algorithm. If the check fails, the data frame is discarded. A serial number cache is established to record the device media access control address and a 16-bit custom serial number. Custom sequence numbers are generated incrementally by the sender and used cyclically. The data frame corresponding to the first occurrence of a custom sequence number enters the parsing queue, while data packets corresponding to custom sequence numbers that do not appear for the first time are determined to be duplicate data packets and discarded. The serial number cache library is configured with a maximum storage capacity and a timestamp expiration cleanup mechanism to automatically delete custom serial number records that exceed a preset duration.

[0014] Furthermore, the specific process of reverse parsing the data points is as follows: The parsing rule is searched for in the preset hash table based on the combination of cluster identifier and attribute identifier; Extract the raw data fields from the data frame payload according to the starting offset bytes and data length specified by the parsing rules; Perform byte order conversion on the original data field according to the byte order defined by the parsing rules; The scaling factor and offset in the application parsing rules are used to calculate the standard business data using the formula "business data = original data × scaling factor + offset". The validity of standard business data is verified by parsing the validation rules in the rules; the validated business readable data is then output.

[0015] Furthermore, the cloud communication submodule adopts a dual-channel transmission architecture of message queue telemetry transmission protocol + Hypertext Transfer Protocol version 2: The message queue telemetry transmission protocol is responsible for real-time transmission of device data and issuance of reverse control commands, and ensures message reachability through the quality of service level. Hypertext Transfer Protocol version 2 is responsible for batch data synchronization and remote protocol update protocol file transfer.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Adopting the core logic of "underlying compliance with the IEEE 802.15.4 standard and dynamic adaptation at the application layer", non-standard Zigbee protocol devices can complete the access process through a standardized network access interaction process without hardware modification or protocol reconstruction. This completely solves the pain point of existing gateways "only supporting specific ecosystem standard devices" and achieves seamless compatibility with non-standard devices from multiple manufacturers and models.

[0017] 2. The parsing engine supports parsing JSON and XML dual-format protocol files. Combined with the precise matching mechanism of "cluster identifier + attribute identifier", it can adapt to various non-standard protocol features such as custom cluster identifier, private data frame format, and differentiated unit definition. It covers most non-standard Zigbee devices in scenarios such as smart home and industrial IoT, and the adaptation range is improved by more than 80% compared with existing technologies.

[0018] 3. The standardized network access process employs a hierarchical scanning strategy of "priority channel + secondary channel" and interactive logic based on Zigbee protocol stack primitives to ensure the stability and compatibility of device access. Even if the underlying parameters of devices from different manufacturers differ, network access and identity recognition can still be completed through a unified process. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the method for accessing and dynamically parsing data from non-standard Zigbee protocol devices based on a Zigbee gateway, as described in this invention. Detailed Implementation To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] To address the shortcomings of the aforementioned technologies, the following solution is proposed: Develop a gateway solution capable of supporting the access of non-standard ZigBee protocol devices, enabling automatic identification and data parsing of ZigBee devices from different manufacturers and models. Through technologies such as a built-in dynamic protocol parsing engine and protocol template library, the device access process is automated and intelligent. This reduces development costs and maintenance complexity, while improving system flexibility and scalability.

[0021] The specific method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway is characterized by the following steps: S1. A non-standard Zigbee protocol device initiates a network access request, and the Zigbee communication submodule executes a standardized network access interaction process; the Zigbee communication submodule sequentially performs preamble / SFD detection, vendor code extraction and matching, and non-standard protocol feature identification on the data frames sent by the device; Specifically, the non-standard Zigbee device described in this invention refers to a Zigbee device that conforms to the physical layer and MAC layer protocols specified in the IEEE 802.15.4 standard, but whose application layer structure, Cluster ID allocation, data payload format, and communication command encoding do not conform to the specifications released by the Zigbee Alliance (such as Zigbee Home Automation, Zigbee LightLink, etc.), and has custom protocol features.

[0022] The network access request is a continuous process of "network discovery → association request", which relies on the primitive interaction between the Zigbee network layer (NWK) and the MAC layer: Step 1: Network Discovery (Acquiring Target Network Information) The device triggers the MAC layer to execute MLME_SCAN.request (MAC layer management entity - scan request primitive) by calling NLME_NETWORK_DISCOVERY.request. Channels 11-26 are scanned in the order of "priority channels (11, 15, 20, 25, 26) → secondary channels (12, 13, 14, 16-19, 21-24)". During the scan, the device sends "Beacon Request" frames to each channel, receives "Beacon Frames" returned by the coordinator / router in the same channel, and extracts key information (such as network PANID, coordinator address, channel quality, and whether association is allowed) from the beacon frames using the MLME_BEACONNOTIFY.indication primitive. Networks with matching protocol IDs are then stored in the "neighbor table".

[0023] Step 2: After the target network and parent node selection scan is completed, the MAC layer reports the results to the network layer through the MLME_SCAN.confirm primitive. The network layer then returns the list of available networks to the device application layer through the NLME_NETWORK_DISCOVERY.confirm primitive. The device selects the target network from the list that is "unique PANID, has the least channel interference, and allows association" (prioritizing channels without other networks) and filters the potential parent nodes from the neighbor table that are "link cost ≤ 3 and device type match (FFD / RFD)" (prioritizing the node with the smallest coordinator depth).

[0024] Step 3: Initiating an Association Request (Core Network Entry Action) The device calls NLME_JOIN.request (Network Layer Management Entity - Join Request Primitive), triggering the MAC layer to generate MLME-ASSOCIATE.request (MAC Layer Management Entity - Association Request Primitive), and sends an "Association Request Command Frame" to the selected parent node (usually the gateway coordinator). This command frame carries core device information: device capabilities (such as whether it supports routing, power type, defined by the CapabilityInformation parameter), 64-bit extended address (long address), and protocol version identifier. It is the core carrier of the "Network Entry Request"—after receiving it, the parent node parses the frame using the MLME-ASSOCIATE.indication primitive to complete the initial verification of the device identity.

[0025] The specific steps of the standardized network access interaction process include: Step 1: The coordinator opens the network access window (execution entity: gateway coordinator). The gateway coordinator broadcasts a "allow network access command" by calling Zigbee network layer primitives, explicitly opening a network access window of a specified duration (default 30 seconds, configurable). This command sets the MAC layer "allow association" attribute to enabled, sending a "network access allowed" signal to surrounding non-networked devices, providing the prerequisite for devices to join the network.

[0026] Step 2: Device Scanning and Information Collection (Executor: Non-standard Zigbee Protocol Device) After detecting the "Allow Network Entry Command," the device enters the network entry state and scans channels 11-26 in the order of "Priority Channels (11, 15, 20, 25, 26) → Secondary Channels (12, 13, 14, 16-19, 21-24)." During the scan, the device sends a "Beacon Request" to each channel, receives "Beacon Frames" returned by the coordinator within the same channel, extracts key information from the frames (network PANID, coordinator address, channel quality, protocol version), and filters out target networks that are "allowed to be associated with minimal channel interference," storing them in the local neighbor table.

[0027] Step 3: The device sends an association request (executor: non-standard Zigbee protocol device). The device selects the target network and parent node (usually the gateway coordinator) from the neighbor table and sends an "association request message". This message carries the device's core identity and capability information, including a 64-bit long address (the device's unique identifier), device capability parameters (such as whether it supports routing and power type), and protocol version identifier, ensuring that the coordinator can complete the initial identity verification.

[0028] Step 4: Coordinator verifies and allocates network resources (Execution entity: Gateway Coordinator). After receiving the "Association Request Message," the coordinator verifies the device's protocol version compatibility and its own network resources (such as short address availability). If the verification passes, it returns an "Association Response Message" to the device, which contains two key configurations: the target network's PANID (ensuring the device accesses the correct network) and the 16-bit short address assigned to the device (used for efficient addressing within the network). If the verification fails, it returns a rejection response explaining the reason (such as insufficient resources).

[0029] Step 5: Device Broadcasts Network Entry Announcement (Executor: Non-standard Zigbee Protocol Devices) The device receives and parses the "Association Response Message," confirms the acquisition of the short address, and then broadcasts a "Device Announcement Message" to the entire network. This message contains the device's long address, the assigned short address, and device capability information, declaring that it has successfully joined the network and simultaneously informing other nodes in the network (such as routers and other terminals) of the new device information to avoid communication conflicts.

[0030] Step 6: The coordinator reads device attributes and completes identification (Execution entities: gateway coordinator + device). After receiving the "Device Declaration Message," the coordinator sends an "Attribute Read Request Message" to the device, specifying the core attributes under the device's "Basic Cluster" to be read. The device replies with key information via an "Attribute Response Message," including the manufacturer name, Zigbee cluster library version, application version, and model identifier. The gateway accurately identifies the device product type based on the "Manufacturer Name + Model Identifier," providing a basis for subsequent non-standard protocol parsing rule matching. This completes the standardized network access interaction.

[0031] S2. According to the protocol rules loaded by the parsing engine, the protocol parsing module receives the data distributed by the Zigbee communication submodule, performs CRC check and sequence number deduplication, and then performs reverse parsing of the data points of the Hex format raw data. Specifically, this process is based on the protocol rules loaded by the parsing engine (including built-in templates and rules generated from OTA upload protocol files). It performs a standardized parsing process of "verification → matching → extraction → conversion → calculation → verification" on the raw hexadecimal data, which includes the following 6 consecutive steps: The specific steps for reverse analysis of data points are as follows; Step 1: Data Reception and Preprocessing Preparation (Execution Entity: Protocol Parsing Module) The protocol parsing module receives the hexadecimal raw data dispatched by the Zigbee communication submodule, and synchronously obtains the device identifier (such as media access control address), cluster identifier and attribute identifier corresponding to the data, providing basic information for subsequent verification and rule matching; at the same time, it initializes the parsing cache, clears the residual data from the previous round of parsing, and avoids cross-interference.

[0032] Step 2: Cyclic Redundancy Check (CRC Check) (Execution Entity: Protocol Parsing Module) Based on the check algorithm defined in the protocol rules (such as CRC16_MODBUS), the integrity of the received hexadecimal raw data frame is checked: the CRC value of the data frame (excluding the frame tail CRC field) is calculated and compared with the frame tail CRC field of the data frame itself; if the two are consistent, the data transmission is determined to be undamaged and proceed to the next step; if they are inconsistent, it is determined to be an invalid data frame, which is directly discarded and an exception log is recorded.

[0033] Step 3: Serial Number Deduplication and Filtering (Execution Entity: Protocol Parsing Module) Extract a 16-bit custom serial number from the original data frame (extracted according to the serial number field position defined in the protocol rules), and query the serial number cache library in conjunction with the device identifier: If there is no record of the "device identifier + serial number" combination in the cache library, it is determined to be the first reception, and the serial number is stored in the cache library (managed by the timestamp expiration cleanup mechanism), and proceed to the next step; if the record of the combination already exists, it is determined to be a duplicate data packet, and it is discarded directly to avoid duplicate parsing and occupying system resources.

[0034] Step 4: Precise Matching of Protocol Rules (Execution Entity: Parsing Engine) The parsing engine uses a combination of "cluster identifier + attribute identifier" to quickly search for matching parsing rules from a pre-set hash table (the hash table is built when the protocol rules are loaded, ensuring millisecond-level lookup efficiency). Parsing rules include core configurations: data start offset bytes, data length, data type (e.g., 16-bit integer, 8-bit unsigned integer, 32-bit floating-point number), byte order (big-endian or little-endian), scaling factor, offset, numerical range limit, and verification rules. If no matching rule is found, an exception handling mechanism is triggered, the original data is stored in the parsing queue, and a protocol file is requested from the cloud.

[0035] Step 5: Raw data extraction and standardization transformation (Execution entity: parsing engine) Data Extraction: According to the matched protocol rules, extract the byte segment starting from the "starting offset byte" and corresponding to the "data length" from the data payload of the hexadecimal raw data frame (e.g., if the rule specifies a starting offset of 6 bytes and a data length of 2 bytes, then extract the 6th-7th bytes of the original data); Byte Order Conversion: Perform byte order adjustment on the extracted byte segment according to the byte order (big-endian or little-endian) defined by the rules (e.g., for little-endian byte order, adjust the original data with the low byte first and high byte last to the standard byte order); Data Type Conversion: Convert the adjusted byte segment to the data type defined by the rules (e.g., convert 2 bytes of hexadecimal data to a 16-bit integer); Scaling and Offset Calculation: Apply the scaling factor and offset in the rules, and calculate the final business data using the formula "Standardized Business Data = Converted Original Data × Scaling Factor + Offset" (e.g., original data 21, scaling factor 0.1, offset -40, calculated result is -37.9℃).

[0036] Step 6: Data Validation and Output (Execution Entity: Parsing Engine) Validate standardized business data according to the validation rules in the protocol: First, numerical range validation (determining whether the business data is between the minimum and maximum values ​​defined in the rules, such as temperature -40℃ to 85℃); second, optional secondary validation (such as re-validating data integrity according to the validation algorithm specified in the rules); if the validation passes, output standardized, business-readable data (including numerical values ​​and units) and transmit it to the data preprocessing module; if the validation fails, mark the data as abnormal, retain the original data and the reason for the abnormality, and report it to the cloud for subsequent traceability.

[0037] S3. The data preprocessing module performs field extraction, data normalization, and cache queue storage processing on the parsed data; The specific data preprocessing module focuses on "redundancy removal, standardization, and stable storage." It performs coherent processing on the standardized business data output by the protocol parsing module to ensure consistent data format and reliable quality, laying the foundation for subsequent data encapsulation and cloud uploading. This process includes three main stages: "field extraction → data normalization → cache queue storage," each with clearly defined execution logic and operational details. Specific processing steps of data preprocessing Step 1: Field Extraction (Execution Entity: Data Preprocessing Module) Extraction criteria: Based on the "data point definition" (such as data point name and attribute identifier) ​​in the protocol rules loaded by the parsing engine, the effective monitoring indicators in the parsed data are accurately identified.

[0038] Core operations: Extract core business fields from the parsing results, including device identifier (a unique identifier bound to the original data), monitoring indicator name (such as temperature, humidity, equipment status), original value, unit, and collection timestamp; at the same time, remove redundant information (such as temporary calculated values, duplicate fields, and invalid placeholders during the parsing process), and retain only the core data necessary for subsequent processing.

[0039] Example: If the parsing result contains "Device ID=zigbee_0x1234, Temperature value=21.1, Temperature unit=℃, Humidity value=40, Humidity unit=%, Temporary check value=0xABCD, Acquisition time=2023-08-20T14:23:15+08:00", then after extraction, retain "Device ID, Temperature value, Temperature unit, Humidity value, Humidity unit, Acquisition time" and discard "Temporary check value".

[0040] Step 2: Data Normalization (Execution Entity: Data Preprocessing Module) For the extracted core data, a dual operation of "timestamp standardization + numerical unit unification" is performed to eliminate data format differences between different non-standard devices: (1) Timestamp standardization: Processing target: Local timestamps reported by the device (which may contain different time zones, such as East 8 zone and West 5 zone) or non-standard format times (such as "20230820142315" "14:23:152023-08-20").

[0041] Operation logic: First, identify the original time zone and format of the timestamp, and convert it to Coordinated Universal Time (UTC) format according to the ISO8601 standard; when converting the time zone, calculate the offset based on the time zone information recorded in the device attributes (or the time zone preset by the protocol rules), and uniformly adjust it to "YYYY-MM-DDTHH:MM:SSZ" when converting the format.

[0042] Example: Convert the time "2023-08-20T14:23:15+08:00" in East 8 time zone to UTC format "2023-08-20T06:23:15Z"; convert the non-standard format "20230820142315" (East 8 time zone) to "2023-08-20T06:23:15Z".

[0043] (2) Unified numerical units: Processing objects: Non-standard unit data reported by different non-standard devices (e.g., temperature may be in Fahrenheit or Celsius, and humidity may be in a custom unit "RH" or percentage).

[0044] Operational logic: Based on the "standard units" defined in the protocol rules (such as temperature uniformly in °C, humidity uniformly in %, and voltage uniformly in V), perform unit conversion; the conversion formula is based on the preset conversion relationship in the protocol rules, such as Fahrenheit to Celsius: °C = (℉-32) × 5 / 9, and custom humidity unit to percentage: percentage = custom value × scaling factor).

[0045] Example: 70℉ Fahrenheit is converted to 21.1℃, and a custom humidity value "80 (RH)" (scaling factor 0.5) is converted to 40%.

[0046] Step 3: Cache queue storage (Execution subject: data preprocessing module) Queue type: First-in-first-out (FIFO) cache queue is used to ensure the consistency of data processing time order and avoid out-of-order uploads.

[0047] Storage operation: The normalized complete data (including device identifier, UTC timestamp, standardized unit value, and monitoring indicator name) is stored in the queue in the order of collection time; each data is accompanied by a "storage timestamp" for subsequent expiration cleanup.

[0048] Queue management mechanism: Dynamic expansion: The queue's default capacity supports 1000 data entries. When the amount of stored data reaches 80% of the capacity, it will automatically expand to 1.5 times the current capacity to avoid data overflow. Expired data cleanup: Regularly scan the queue and automatically delete data that has been stored for more than 24 hours and has not been encapsulated and uploaded, thus freeing up storage space; Overflow protection: If a sudden surge in data causes the queue to reach its maximum expansion limit, the latest data will be retained first, while the overflow status will be marked and reported to the cloud to ensure that core data is not lost.

[0049] Data retrieval: The data encapsulation module reads data from the queue according to the "first-in, first-out" principle. After successful reading, the data is marked as "processed". Processed data is cleaned up periodically thereafter.

[0050] The S4 data encapsulation module formats the preprocessed data in the cache queue into JSON format, generating a data packet containing the device's unique identifier, UTC timestamp, and standardized business data, which is then uploaded to the cloud via the cloud communication submodule of the network communication module.

[0051] The data encapsulation module, with "structured, standardized, and highly reliable" as its core objectives, completes the JSON format encapsulation of preprocessed data and its cloud upload. It is a crucial link connecting local data processing at the gateway with cloud data reception, specifically comprising four main stages: "Cached Data Reading → JSON Structured Encapsulation → Data Packet Verification → Dual-Channel Cloud Upload." Each stage has clearly defined execution logic, operational specifications, and exception handling: Specific processing content of data encapsulation and cloud upload: 1. Stage 1: Cache Queue Data Reading (Execution Entity: Data Encapsulation Module) Reading Rules: Strictly adhere to the "First-In, First-Out (FIFO)" principle, reading complete preprocessed data in the "Unprocessed" state from the data preprocessing module's cache queue. Before reading, verify the validity of the queue data (e.g., whether there is data frame corruption or missing fields) to ensure that the read data contains all core fields: "device unique identifier, UTC timestamp, and standardized business data (including indicator name, value, and unit)". - Status Marking: Upon successful reading, immediately update the status of the data in the cache queue to "Pending Encapsulation" to avoid repeated readings. If reading fails (e.g., data frame corruption or missing fields), mark the data as "Read Abnormal," record the reason for the abnormality (e.g., "UTC timestamp format error" or "business data value is empty"), and trigger the retry mechanism (maximum 3 retries, 1-second interval). If the retry fails, report the abnormality log to the cloud.

[0052] Data filtering: During the reading process, expired data with a storage time of more than 24 hours in the cache queue is automatically filtered to avoid invalid encapsulation occupying system resources.

[0053] Step 2: JSON format structured encapsulation (Execution subject: data encapsulation module) Encapsulation Specifications: Follows the predefined JSON Schema standard (adapting to unified cloud parsing rules) to ensure a fixed structure and parsable fields in the generated JSON data packet. It includes four core mandatory fields (without redundancy), the specific definitions and examples of which are as follows: The field name is “device_unique_id”, the field type is string, and it is a globally unique identifier assigned to the device by the gateway. It is composed of “Zigbee short address + device MAC address” to ensure that the cloud can accurately associate the device identity. The example value is “0x1234_00124B0022334455”. The field name is "utc_timestamp", the field type is string, and it strictly follows the UTC timestamp format of the ISO8601 standard. The format is "YYYY-MM-DDTHH:MM:SSZ". It is used to unify the time base of cloud data and eliminate time zone differences. The example value is "2025-12-11T08:30:15Z". The field name is "business_data", and the field type is an array. It is used to hold standardized business data. Each element in the array corresponds to a monitoring indicator. Each element contains three subfields: "metric_name", "value", and "unit". "metric_name" is a string type, representing the name of the monitoring indicator (such as temperature or humidity). "value" is a numeric type (integer or floating-point number), representing the standardized value of the monitoring indicator. "unit" is a string type, representing the standard unit of the monitoring indicator (such as ℃, %, or V). It supports flexible encapsulation of single or multiple indicators. In the example, this field contains two monitoring indicators, temperature and humidity, and the corresponding content is [{"metric_name":"temperature","value":25.6,"unit":"℃"},{"metric_name":"humidity","value":45.2,"unit":"%"}]; The field name is "packet_meta", the field type is object, and it is used to store the metadata of the data packet (for cloud verification). It contains three subfields: "length", "gen_time", and "sign". "length" is an integer representing the byte length of the JSON data packet; "gen_time" is a string representing the generation time of the data packet (gateway local time, in the format "YYYY-MM-DDTHH:MM:SS±HH:MM"); and "sign" is a string representing the digest signature of the data packet (generated based on the AES-128-CCM encryption algorithm, used for cloud verification of data packet integrity). An example value is {"length":186,"gen_time":"2025-12-11T16:30:15+08:00","sign":"0x789ABC"}. Example of a complete JSON data packet: json{"device_unique_id":"0x1234_00124B0022334455","utc_timestamp":"2025-12-11T08:30:15Z","business_data":[{"metric_name":"temperature","value":25.6,"unit":"℃"},{"metric_name":"humidity","value":45.2,"unit":"%"}],"packet_meta":{"length":186,"gen_time":"2025-12-11T16:30:15+08:00","sign":"0x789ABC"}} Encapsulation Adaptability: Supports flexible encapsulation of single metrics (such as temperature data only) and multiple metrics (such as temperature and humidity + device operating status), adapting to the monitoring dimension differences of different non-standard Zigbee devices. Furthermore, the naming rules and data types of all fields are completely unified, ensuring that the cloud does not need to develop dedicated parsing logic for different manufacturers' devices; data parsing can be completed directly through the preset JSON Schema. Step 3: Data packet integrity and legality verification (Execution subject: data encapsulation module) - Dual verification logic: (1) JSON syntax verification: Verify the syntax legality of the data packet through the built-in JSON parser (such as whether there are syntax errors or field type mismatches) to ensure that the cloud can parse it normally; (2) Field integrity verification: Verify whether the required fields (device_unique_id, utc_timestamp, business_data) are missing, and whether the field values ​​conform to the preset rules (such as whether utc_timestamp is a valid ISO8601 format, and whether the value of business_data is a numeric type); - Security signature (optional): Based on the AES-128-CCM encryption algorithm defined by the security configuration field, generate a digest signature for the JSON data packet, store it in the sign field of packet_meta, and use it to verify that the data packet has not been tampered with by the cloud; - Exception handling: If the verification fails, discard the data packet, roll back the original preprocessed data to the cache queue (marked as "encapsulation failure"), and record the exception log (including the failure field and error type) for subsequent re-encapsulation.

[0054] Step 4: Upload to the cloud via the cloud communication submodule (Execution subject: data encapsulation module + cloud communication submodule) - Transmission architecture adaptation: Based on the "MQTT + HTTP / 2 dual-channel" architecture of the cloud communication submodule, a differentiated upload strategy is implemented: (1) Single data with high real-time requirements (such as device status changes, sensor real-time values): Upload via MQTT protocol, relying on MQTT's QoS1 service quality level (at least one delivery) to ensure real-time data availability; (2) Batch historical data (such as multiple data accumulated in the cache queue): Packaged and uploaded via HTTP / 2 protocol, utilizing HTTP / 2's multiplexing technology to improve concurrent transmission efficiency; - Upload process: (1) Connection verification: The cloud communication submodule first verifies the connection with the cloud. (1) Connection status (based on a 60-second heartbeat keep-alive mechanism). If the connection is lost, reconnection will be performed according to the exponential backoff strategy (the first reconnection interval is 2 seconds, and the interval doubles thereafter, with a maximum interval of 32 seconds); (2) Data packet sending: Send the verified JSON data packet to the specified topic / interface in the cloud according to the selected protocol; (3) Receipt confirmation: Wait for the cloud to return a "upload successful" receipt. After receiving the receipt, mark the corresponding data in the cache queue as "uploaded" and clean it up periodically; If no receipt is received (timeout of 10 seconds), trigger the retransmission mechanism (up to 3 retransmissions). If the retransmission fails, the data will be returned to the cache queue; - Status synchronization: After the upload is completed, synchronize the upload status (success / failure, number of data packets, transmission time) to the local log module of the gateway for local operation and maintenance and cloud monitoring. Supplementary Explanation (Demonstrating Technical Value and Adaptability) 1. Adaptability Logic: The unified JSON format encapsulation solves the problem of data structure differences between non-standard Zigbee devices from different manufacturers, ensuring that the cloud does not need to develop exclusive parsing logic for different devices; the dual-channel transmission architecture balances "real-time performance" and "transmission efficiency," adapting to the transmission needs of different types of data in IoT scenarios. 2. Reliability Guarantee: Full-process status marking, retry mechanism, and receipt confirmation prevent data loss or duplicate uploads; AES-128-CCM signature verification ensures that data packets are not tampered with during transmission and meets the encryption requirements of security configuration fields. 3. Closed-Loop Process: A complete closed loop is formed with the upstream data preprocessing module (cached queue reading) and the downstream cloud communication submodule (dual-channel upload), and the upload status is synchronized to both local and cloud, achieving full lifecycle traceability of data. Complete content of the integrated "Data Encapsulation Module" (excerpt) 4. Data Encapsulation Module 1. Core Function: Encapsulates standardized data in the cache queue of the data preprocessing module into a unified JSON format data packet, and reliably uploads it to the cloud through the cloud communication submodule, realizing standardized interaction between local data and the cloud.2. Core Processing Flow: The data encapsulation module formats the preprocessed data in the cache queue into JSON format, generating a data packet containing the device's unique identifier, UTC timestamp, and standardized business data. This packet is then uploaded to the cloud via the cloud communication submodule of the network communication module. Specifically, this includes the following four steps: 1. Step 1: Cache Queue Data Reading (Execution Entity: Data Encapsulation Module) - Reading Rules: Following the "First-In-First-Out (FIFO)" principle, reads the complete preprocessed data that is "unprocessed" in the cache queue and verifies the data's validity (field integrity, no corruption); - Status Marking: Successful reading is marked as "Pending Encapsulation," while failure triggers 3 retries (1-second interval). Failed retries are reported in the exception log; - Data Filtering: Automatically filters out expired data stored for more than 24 hours. 2. Step 2: JSON Format Structured Encapsulation (Execution Entity: Data Encapsulation Module) - Encapsulation Specifications: Follows the preset JSON Schema standard, including four mandatory fields: unique device identifier, UTC timestamp, business data body, and data packet metadata. Field types and naming are consistent. - Flexible Adaptation: Supports single / multiple indicator encapsulation to adapt to the monitoring dimension differences of different non-standard devices. - Example: (Complete JSON data packet example attached, omitted). 3. Step 3: Data Packet Integrity and Legality Verification (Execution Entity: Data Encapsulation Module) - Dual Verification: JSON syntax verification + mandatory field integrity verification to ensure that the data packet conforms to the cloud parsing rules. - Secure Signature: Generates a data packet digest signature based on the AES-128-CCM algorithm to prevent tampering during transmission. - Exception Handling: If verification fails, the data is rolled back to the cache queue, and an exception log is recorded. 4. Step 4: Upload to the cloud via the cloud communication submodule (Execution entity: data encapsulation module + cloud communication submodule) - Dual-channel strategy: real-time data uses MQTT (QoS1), batch data uses HTTP / 2 (multiplexing); - Upload process: connection verification → data packet sending → receipt confirmation → status synchronization. Upon disconnection, reconnection is exponentially avoided, and 3 retransmissions are triggered upon timeout; - Status management: successfully uploaded data is marked as "uploaded," and failed data is returned to the cache queue. 3. Exception fallback mechanism: If the cloud remains unreachable (reconnection fails after more than 32 seconds), the cache queue automatically expands to its maximum capacity (default 10,000 records), prioritizing the retention of the latest data; after the cloud connection is restored, the backlogged data in the queue is automatically uploaded in batches to ensure no data loss.

[0055] The above are merely embodiments of the present invention. The circuits, electronic components, and modules involved are all prior art, fully achievable by those skilled in the art, and require no further explanation. The content protected by this application does not involve improvements to the software and methods. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all prior art in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the application.

Claims

1. A method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway, characterized in that: Includes the following steps: S1. Non-standard Zigbee protocol devices initiate network access requests, and the Zigbee communication submodule executes the standardized network access interaction process; The Zigbee communication submodule sequentially performs preamble / SFD detection, vendor code extraction and matching, and non-standard protocol feature identification on the data frames sent by the device. S2. According to the protocol rules loaded by the parsing engine, the protocol parsing module receives the data distributed by the Zigbee communication submodule, performs CRC check and sequence number deduplication, and then performs reverse parsing of the data points of the Hex format raw data. S3. The data preprocessing module performs field extraction, data normalization, and cache queue storage processing on the parsed data; The S4 data encapsulation module formats the preprocessed data in the cache queue into JSON format, generating a data packet containing the device's unique identifier, UTC timestamp, and standardized business data, which is then uploaded to the cloud via the cloud communication submodule of the network communication module.

2. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 1, characterized in that: The standardized network access interaction process described in step S1 is as follows: The gateway's coordinator broadcasts a permission command to join the network; Once the device to be connected to the network enters the network access state, it scans the designated channels in the order of priority channel first and then secondary channel, and sends beacon requests in sequence. After receiving a beacon request, the coordinator on the same channel sends a beacon response message. The devices waiting to join the network filter out the compatible networks and send an association request message. The association request message carries the device type, address and device capability information. The coordinator determines whether to allow network access based on the current resource status. If allowed, it returns an associated response message, which includes the target network's personal area network identifier and the short address assigned to the device. After a device successfully joins the network, it broadcasts a device announcement message to declare that it has joined the network. The device announcement message includes the device's long address, short address, and device capability information. The coordinator sends an attribute read request message to read the attributes of the device's underlying cluster. The device replies with the manufacturer name and model identifier through the attribute response message. The gateway identifies the device product type based on the manufacturer name and model identifier.

3. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 2, characterized in that: The data frame structure processed by the Zigbee communication submodule includes a preamble, frame control, vendor code, cluster identifier, data payload, and cyclic redundancy check.

4. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 3, characterized in that: The preamble and start-of-frame delimiter detection is used to verify the legality of physical layer frames and locate the start position of data. Specifically, it scans the binary stream and matches a 4-byte standard preamble and a 1-byte start-of-frame delimiter. If the detection is successful, it returns the start position of the frame after skipping the preamble and start-of-frame delimiter. If no preamble and start-of-frame delimiter that meet the requirements are detected, the frame is determined to be invalid and is discarded directly.

5. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 4, characterized in that: The vendor code is extracted from the manufacturer identifier field of the Zigbee application layer protocol frame. The extraction position is a preset fixed offset position of the application layer identifier in the application layer protocol frame. During extraction, the two bytes of data at this position are combined into a 16-bit vendor code by left-shifting the high-order bits by 8 bits and concatenating them with the low-order bits.

6. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 5, characterized in that: The protocol parsing module's parsing engine supports parsing protocol files in two standardized formats: JSON and XML. The protocol file contains two core fields: device configuration file and security configuration. The device configuration file defines the manufacturer identifier, device type, protocol version, custom cluster, data points, and reverse control instructions. The data point definition includes name, data type, byte order, scaling factor, offset, unit, starting offset bytes, and verification rules. The security configuration field defines the AES-128-CCM encryption algorithm and the RSA-2048 digital signature algorithm.

7. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 6, characterized in that: The reverse control command includes a cluster identifier, a command identifier, and parameter configuration. The parameter configuration includes parameter name, encoding format, scaling factor, and offset. The parsing engine can generate a control message that the device can recognize based on the reverse control command. The control message is sent to the target device through the Zigbee communication submodule to achieve reverse control.

8. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 1, characterized in that: The cyclic redundancy check and serial number deduplication mechanism are as follows: The integrity of data frames is checked using a cyclic redundancy check algorithm. If the check fails, the data frame is discarded. A serial number cache is established to record the device media access control address and a 16-bit custom serial number. Custom sequence numbers are generated incrementally by the sender and used cyclically. The data frame corresponding to the first occurrence of a custom sequence number enters the parsing queue, while data packets corresponding to custom sequence numbers that do not appear for the first time are determined to be duplicate data packets and discarded. The serial number cache library is configured with a maximum storage capacity and a timestamp expiration cleanup mechanism to automatically delete custom serial number records that exceed a preset duration.

9. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 1, characterized in that: The specific process of reverse parsing the data points is as follows: The parsing rule is searched for in the preset hash table based on the combination of cluster identifier and attribute identifier; Extract the raw data fields from the data frame payload according to the starting offset bytes and data length specified by the parsing rules; Perform byte order conversion on the original data field according to the byte order defined by the parsing rules; The scaling factor and offset in the application parsing rules are used to calculate the standard business data using the formula "business data = original data × scaling factor + offset". The validity of standard business data is verified by parsing the validation rules in the rules; the validated business readable data is then output.

10. The method for accessing non-standard Zigbee protocol devices and dynamically parsing data based on a Zigbee gateway as described in claim 1, characterized in that: The cloud communication submodule adopts a dual-channel transmission architecture of message queue telemetry transmission protocol + Hypertext Transfer Protocol version 2: The message queue telemetry transmission protocol is responsible for real-time transmission of device data and issuance of reverse control commands, and ensures message reachability through the quality of service level. Hypertext Transfer Protocol version 2 is responsible for batch data synchronization and remote protocol update protocol file transfer.

Images