A configurable recursive internet of things data frame construction method and system for electric energy meters and electricity collection terminals

Abstract

The application discloses a configurable recursive Internet of Things data frame construction method and system for electric energy meters and power consumption collection terminals, and relates to the technical field of Internet of Things communication and embedded systems. The method is based on a configuration table driving mechanism, describes a JSON data frame structure by defining configuration items containing structure nodes and data nodes, stores real-time collection data by using a state buffer, constructs a referenced data object tree in a recursive traversal mode, only saves data reference information without data copying in the traversal process, and finally generates a target data frame through serialization. The corresponding system comprises a configuration table management module, a state buffer, a recursive JSON construction module and a serialization output module. The application realizes flexible configuration and dynamic update of data formats, supports efficient construction of complex nested JSON structures, significantly reduces resource consumption of embedded devices, and improves the adaptability and maintainability of Internet of Things terminals in variable business scenarios.

Description

Technical Field

[0001] This invention belongs to the field of Internet of Things (IoT) data communication technology, specifically relating to a configurable recursive IoT data frame construction method and system for electricity meters and electricity consumption collection terminals. Background Technology

[0002] The Internet of Things (IoT) technology has developed rapidly in recent years. Various terminal devices collect environmental and business data through sensors and upload the data to the cloud for analysis and processing using network protocols. Among them, the MQTT protocol, due to its lightweight and asynchronous characteristics, has become one of the mainstream communication methods for data reporting by IoT devices such as electricity meters and electricity consumption collection terminals. Therefore, how to efficiently and reliably organize and encapsulate the collected multi-channel sensor data to meet both the format requirements of customized business needs and the resource-constrained operating environment of embedded terminals has become a key issue that needs to be addressed in practical deployments.

[0003] In traditional solutions, IoT terminals such as electricity meters and electricity consumption data acquisition terminals typically define the structure of the data to be uploaded using hard-coding. Specifically, the device firmware pre-writes code to construct specific JSON objects, for example, by directly calling library functions such as cJSON to organize field names and data values ​​into strings with a fixed hierarchical relationship before uploading. While this approach is simple to implement, it has two significant drawbacks in practical applications: First, because the data format is fixed within the code, when cloud-based business requirements change, requiring adjustments to field names or nesting relationships, the firmware must be modified and recompiled, resulting in poor system flexibility and maintainability. Second, limited by the memory and computing resources of embedded devices, traditional solutions struggle to efficiently support the construction of multi-layered nested and complex JSON data structures, often only generating relatively simple flat or shallow nested structures.

[0004] To enhance the flexibility of JSON data construction, existing technologies have developed methods that describe data structures through configuration and generate JSON messages using recursive parsing. For example, Chinese invention application CN118317008A proposes a multi-protocol, multi-interface data distribution system based on an industrial edge gateway. This system configures a JSON structure template through a custom data format module and calls the CJSON library in conjunction with a recursive algorithm to parse the data layer by layer down to the lowest level, thereby achieving configurable data format.

[0005] However, the above solutions still have significant shortcomings when facing IoT terminal scenarios with highly limited resources. First, their configuration templates typically need to fully describe the entire JSON structure, lacking a substructure reuse mechanism. When multiple reporting formats contain the same sub-nodes, if the field names or data sources of that sub-node need to be adjusted, all related templates must be modified one by one, making it impossible to achieve unified configuration and dynamic updates. This results in low format adaptation efficiency and difficulty in coping with frequently changing business needs. Second, the solution requires converting all data to basic types during JSON construction and performing multiple data copies during assembly. Especially when dealing with deeply nested and complex data, it frequently allocates and releases memory, resulting in high memory and CPU overhead. For embedded devices with only limited RAM, such resource consumption not only affects system stability but also causes data upload delays, impacting real-time transmission performance. Summary of the Invention

[0006] This invention proposes a configurable recursive IoT data frame construction method and system for electricity meters and electricity consumption acquisition terminals. Its purpose is to flexibly adapt to the diverse customization requirements of business terminals for data formats, while effectively reducing the memory and CPU resource overhead in the process of constructing complex nested JSON data, thereby achieving efficient and reliable data upload in resource-constrained hardware environments.

[0007] The technical solution of this invention is as follows:

[0008] A configurable recursive Internet of Things (IoT) data frame construction method for electricity meters and electricity consumption data acquisition terminals includes the following steps:

[0009] Step S1: Define configuration items in the configuration table. Each configuration item corresponds to a node in the JSON data frame. Configuration items must include at least the following two types:

[0010] Structure node configuration items: These correspond to objects and arrays in JSON format, containing association information pointing to other configuration items;

[0011] Data node configuration items: These correspond to key-value pairs in JSON format and contain information about how the data is retrieved.

[0012] Step S2: Store the collected data in real time in a statically pre-allocated state buffer with a predefined data structure;

[0013] Step S3: After receiving the data frame construction request, the construction function is called to recursively traverse the data object tree under the drive of the configuration table.

[0014] When traversing to the configuration item of the structure node, the construction function is recursively called on the configuration items corresponding to each of its child nodes according to the associated information contained therein, so as to generate a nested data structure;

[0015] When traversing to a data node configuration item, the corresponding data source is associated with the data acquisition method information contained therein, and the reference information of the target data is saved to the current node; when the data source is a state buffer, the reference information is a pointer to the target data in the state buffer;

[0016] Step S4: Serialize the referenced data object tree into a data frame and write it to a statically pre-allocated output buffer; during serialization, read the target data from the data source according to the saved reference information and write it to the static output buffer.

[0017] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption collection terminals: the fields of the configuration table are used to define the attributes of the nodes and the behavior of the nodes when they are constructed; during traversal, the processing logic is determined according to the field values ​​configured in the current record.

[0018] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals, the configuration table includes at least the following fields:

[0019] Field name DataName: A string type used to define the key name corresponding to this node in the JSON data frame;

[0020] The framing function CJsonFunc is used to define the method used when adding the current node to the parent node; the object association information NextTable is an array used to store association information pointing to other configuration items.

[0021] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals: during traversal in step S3, if the framing function CJsonFunc in the currently traversed configuration item indicates that the current node is an object, then the current node is added to the parent node as an object through the framing function CJsonFunc, and then the construction function is recursively called on each element in the object association information NextTable to obtain the members of the current node.

[0022] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals: during traversal in step S3, if the framing function CJsonFunc in the currently traversed configuration item indicates that the current node is an array, then the current node is added to the parent node as an array through the framing function CJsonFunc, and then the construction function is recursively called on each element in the object association information NextTable to obtain the array elements of the current node.

[0023] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals, the configuration table also includes the following fields:

[0024] DataType: Defines the data source for the target data of the current node;

[0025] State Buffer Index ValueType: Defines the target data for the current node in the state buffer; valid only when the data source type DataType indicates that the data source is a state buffer.

[0026] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals: during traversal in step S3, if the current node is a key-value pair, the current node is added to the parent node as a key-value pair through the framing function CJsonFunc, and the reference information of the target data is obtained through the data source type DataType and saved to the current node.

[0027] As a further improvement to the configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals, it also includes:

[0028] Step S0: Receive a configuration update instruction from an external source; dynamically update the configuration table in the device according to the update instruction, thereby changing the format of the JSON data frame without modifying and recompiling the device firmware.

[0029] A configurable recursive Internet of Things (IoT) data frame construction system for electricity meters and electricity consumption data acquisition terminals includes:

[0030] The configuration table management module is used to store and maintain configuration tables. Each configuration table defines multiple configuration items, and each configuration item corresponds to a node in a JSON data frame. Each configuration item includes at least structure node configuration items and data node configuration items. The structure node configuration items correspond to objects and arrays in JSON format and contain association information pointing to other configuration items. The data node configuration items correspond to key-value pairs in JSON format and contain data retrieval method information.

[0031] The state buffer is a statically pre-allocated memory area with a predefined data structure, used to store the acquired data in real time.

[0032] A recursive JSON construction module, communicating with the configuration table management module and the state buffer, is used to respond to a data frame construction request by calling a construction function to perform recursive traversal driven by the configuration table to generate a reference-based data object tree; wherein, the recursive JSON construction module is configured as follows:

[0033] When traversing to the configuration item of the structure node, the construction function is recursively called on the configuration items corresponding to each of its child nodes according to its association information to generate a nested data structure;

[0034] When traversing to a data node configuration item, the corresponding data source is associated with its data acquisition method information, and the reference information of the target data is saved to the current node; when the data source is a state buffer, the reference information is a pointer to the target data in the state buffer;

[0035] The serialization and output module, which is connected to the recursive JSON construction module, is used to serialize the referenced data object tree into a data frame and write it into a statically pre-allocated output buffer. During the serialization process, the target data is read from the data source according to the reference information stored in the node and written into the output buffer.

[0036] As a further improvement to the configurable recursive IoT data frame construction system for electricity meters and electricity consumption acquisition terminals, the configuration table management module also supports receiving configuration update instructions from external sources via wireless or wired communication interfaces and dynamically updating the configuration table accordingly, thereby changing the format of the JSON data frame without modifying and recompiling the system firmware.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This invention defines the structure and data source of each node in a JSON data frame through a configuration table, enabling flexible configuration and dynamic updates of the data format. This overcomes the shortcomings of traditional hard-coding methods that require recompiling the firmware when business requirements change, and significantly improves the adaptability and maintainability of the system in diverse IoT application scenarios.

[0039] 2. When constructing complex nested JSON structures, a predefined static state buffer is used to store real-time collected data, laying the foundation for subsequently determining the reference information of the target data. Furthermore, during the recursive traversal of the configuration table, only the reference information pointing to the target data is saved, rather than data copying, thereby significantly reducing memory usage and CPU overhead during the framing stage, making it particularly suitable for resource-constrained embedded terminals.

[0040] 3. By combining the configuration table with the recursive construction mechanism, it supports the unified description and reuse of structural nodes and data nodes, enabling the same substructure to share the same configuration in multiple data formats, thereby enhancing the modularity and configuration efficiency of the system while reducing maintenance complexity.

[0041] 4. During the serialization phase, the system directly reads data from the status buffer based on the reference information and writes it to the statically pre-allocated output buffer, avoiding dynamic memory allocation and multiple data copies, which further improves the real-time performance and operational stability of data framing.

[0042] 5. The system supports remote updates of the configuration table via OTA or serial port, enabling real-time adjustments to the data format and rapid business iteration without modifying the core firmware, thus providing technical support for the sustainable evolution of IoT terminals in long-term deployment. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the architecture of a configurable recursive IoT data frame construction system for electricity meters and electricity consumption acquisition terminals. Detailed Implementation

[0044] This invention provides a configurable recursive IoT data frame construction method and system for electricity meters and electricity consumption data acquisition terminals. It is suitable for resource-constrained embedded IoT terminals and can dynamically construct multi-layered nested JSON format data frames based on a configuration table, then upload them to a cloud server via the MQTT protocol. The invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] like Figure 1 As shown, the configurable recursive IoT data frame construction system for electricity meters and electricity consumption acquisition terminals includes a configuration table management module, a state buffer, a recursive JSON construction module, and a serialization and output module. The configuration table management module stores and maintains the configuration table, which defines multiple configuration items, each corresponding to a node in a JSON data frame. The state buffer is a statically pre-allocated memory area with a predefined data structure, used to store multi-channel sensor data collected in real time. The recursive JSON construction module communicates with the configuration table management module and the state buffer, responding to data frame construction requests and recursively traversing to generate a reference-based data object tree driven by the configuration table. The serialization and output module communicates with the recursive JSON construction module, serializing the reference-based data object tree into data frames and writing them to the statically pre-allocated output buffer.

[0046] The configuration items in the configuration table include at least structure node configuration items and data node configuration items. Structure node configuration items correspond to objects and arrays in JSON format, containing association information pointing to other configuration items; data node configuration items correspond to key-value pairs in JSON format, containing data retrieval method information. The fields in the configuration table are used to define the attributes of nodes and the behavior when nodes are constructed. Specifically, the configuration table includes at least a field name (DataName), a framing function (CJsonFunc), and object association information (NextTable). The field name (DataName) is a string type, defining the key name corresponding to the node in the JSON data frame. The framing function (CJsonFunc) defines the method used to add the current node to a parent node. The object association information (NextTable) is an array type, storing association information pointing to other configuration items.

[0047] Furthermore, the configuration table may also contain a data source type (DataType) and a state buffer index (ValueType). The data source type (DataType) defines the data source of the target data for the current node, such as from a collection module, system function, or configuration item. The state buffer index (ValueType) defines the corresponding data in the state buffer for the target data of the current node, and is only valid if the data source type (DataType) indicates that the data source is a state buffer.

[0048] The status buffer is designed as a multi-dimensional array structure, such as DoubleData[bLoop][ValueType], where bLoop is used to distinguish between multi-loop or multi-channel data, and ValueType is used to identify the index of the data type in the buffer. This buffer implements time-series caching and read / write separation of data during periodic updates by the acquisition module, avoiding data races and inconsistencies during frame assembly.

[0049] The core of the recursive JSON building module is the build function DataToCJson, whose function prototype can be designed as follows:

[0050] cJSON *DataToCJson(TDataTable *DataTable, char* name, int ValueType,BYTE Loop, cJSON *cJsonOut, QWORD qwTms, char *addr)

[0051] Here, DataTable is a pointer to the target configuration item, name and ValueType are used to handle parameter passing during the recursive process of array type objects, Loop is the number of loops contained in the device, qwTms is the timestamp, and addr is the device address.

[0052] Upon receiving a data frame construction request, the system calls the `DataToCJson` function to perform a recursive traversal driven by the configuration table. When traversing to a structure node configuration item, the node type is determined based on its framing function `CJsonFunc`. If the current node is an object, it is added to the parent node as an object using `CJsonFunc`, and then the construction function is recursively called on each element in `NextTable` to generate the members of the current object. If the current node is an array, it is added to the parent node as an array using `CJsonFunc`, and the construction function is recursively called on each element in `NextTable` to generate array elements. When traversing to a data node configuration item, the current node is added to the parent node as a key-value pair using the framing function `CJsonFunc`, and the reference information of the target data is obtained based on the data source type `DataType` and saved to the current node. When the data source is a state buffer, the reference information is a pointer to the target data in the state buffer (the address of the target data is found using the configured index `ValueType`); when the data source is a system function or configuration item, the corresponding function can be called directly or the configuration value can be read.

[0053] Throughout the recursive construction process, the system adopts a zero-copy data access strategy, that is, the frame assembly function directly references the data pointer in the state buffer without copying the data, and only saves the reference information of the data, thereby significantly reducing memory and CPU overhead.

[0054] The serialization and output module serializes the recursively constructed reference-based data object tree into a JSON string and writes it to a statically pre-allocated output buffer, JsonBuffer. During serialization, the system reads the target data from the corresponding data source based on the reference information stored in each node and writes it directly to the output buffer, avoiding multiple data copies and dynamic memory allocation, thus improving system efficiency and stability.

[0055] In addition, the system supports a dynamic configuration update mechanism. The configuration table management module can receive configuration update commands from external sources via OTA or serial port, dynamically updating the configuration table in the device. This allows for changes to the format of JSON data frames without modifying or recompiling the device firmware, enabling rapid iteration of business requirements.

[0056] The following specific embodiment illustrates the implementation process of the present invention. Assume that the following JSON data frame needs to be generated:

[0057] {

[0058] "meter": {

[0059] "addr": "12345678",

[0060] "loop0": {

[0061] "voltage": "220.000000",

[0062] "current": "5.230000"

[0063] },

[0064] "timestamp": 1723541920

[0065] }

[0066] }

[0067] First, define the configuration items for each node in the configuration table. For example:

[0068] TDataTable Table_Meter = {

[0069] .MqttData.DataName = "meter",

[0070] .CJsonFunc = cJSON_AddObjectToObject,

[0071] .NextTable = {&Table_Addr, &Table_Loop0, &Table_Timestamp}

[0072] };

[0073] TDataTable Table_Addr = {

[0074] .MqttData.DataName = "addr",

[0075] .CJsonFunc = cJSON_AddStringToObject,

[0076] .MqttData.DataType = eGetMeterAddr

[0077] };

[0078] TDataTable Table_Loop0 = {

[0079] .MqttData.DataName = "loop0",

[0080] .CJsonFunc = cJSON_AddObjectToObject,

[0081] .NextTable = {&Table_Voltage, &Table_Current}

[0082] };

[0083] TDataTable Table_Voltage = {

[0084] .MqttData.DataName = "voltage",

[0085] .CJsonFunc = cJSON_AddStringToObject,

[0086] .MqttData.DataType = eFromCollection,

[0087] .MqttData.ValueType = VALUE_VOLTAGE

[0088] };

[0089] TDataTable Table_Current = {

[0090] .MqttData.DataName = "current",

[0091] .CJsonFunc = cJSON_AddStringToObject,

[0092] .MqttData.DataType = eFromCollection,

[0093] .MqttData.ValueType = VALUE_CURRENT

[0094] };

[0095] TDataTable Table_Timestamp = {

[0096] .MqttData.DataName = "timestamp",

[0097] .CJsonFunc = cJSON_AddNumberToObject,

[0098] .MqttData.DataType = eGetTimeMs

[0099] };

[0100] Upon system startup, the configuration table is read from Flash. The acquisition module periodically updates the data in the status buffer, for example, storing the voltage value of loop 0 (220.0) in DoubleData[0][VALUE_VOLTAGE] and the current value (5.23) in DoubleData[0][VALUE_CURRENT]. When a data frame construction request is received, the system calls the DataToCJson(&Table_Meter, NULL, 0, 0, NULL, timestamp, addr) function to recursively traverse the configuration table from the root node, generating JSON objects layer by layer. When traversing to a data node, the system retrieves the pointer to the target data from the status buffer based on the DataType and ValueType, and saves the reference information. Finally, the serialization module converts the generated reference-based data object tree into a JSON string (replacing the reference information of the target data with the target data during this process), writes it to the static buffer, and publishes it to the specified Topic via the MQTT upload module.

[0101] This invention achieves efficient and flexible generation of complex nested JSON data frames on resource-constrained embedded devices by using configuration table-driven recursive construction, combined with state buffers and zero-copy strategies. It supports dynamic format updates and significantly improves the adaptability and maintainability of IoT terminals.

[0102] It should be noted that, as will be apparent to those skilled in the art, the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics thereof. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A configurable recursive Internet of Things (IoT) data frame construction method for electricity meters and electricity consumption data acquisition terminals, characterized in that, Includes the following steps: Step S1: Define configuration items in the configuration table. Each configuration item corresponds to a node in the JSON data frame. Configuration items must include at least the following two types: Structure node configuration items: These correspond to objects and arrays in JSON format, containing association information pointing to other configuration items; Data node configuration items: These correspond to key-value pairs in JSON format and contain information about how the data is retrieved. Step S2: Store the collected data in real time in a statically pre-allocated state buffer with a predefined data structure; Step S3: After receiving the data frame construction request, the construction function is called to recursively traverse the data object tree under the drive of the configuration table. When traversing to the configuration item of the structure node, the construction function is recursively called on the configuration items corresponding to each of its child nodes according to the associated information contained therein, so as to generate a nested data structure; When traversing to a data node configuration item, the corresponding data source is associated with the data acquisition method information contained therein, and the reference information of the target data is saved to the current node; When the data source is a state buffer, the reference information is a pointer to the target data in the state buffer; Step S4: Serialize the referenced data object tree into a data frame and write it to a statically pre-allocated output buffer; during serialization, read the target data from the data source according to the saved reference information and write it to the static output buffer.

2. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 1, characterized in that: The fields in the configuration table are used to define the attributes of nodes and the behavior of nodes when they are built; during traversal, the processing logic is determined based on the field values ​​configured in the current record.

3. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption data acquisition terminals as described in claim 1 or 2, characterized in that, The configuration table must contain at least the following fields: Field name DataName: A string type used to define the key name corresponding to this node in the JSON data frame; The framing function CJsonFunc is used to define the method used when adding the current node to the parent node; the object association information NextTable is an array used to store association information pointing to other configuration items.

4. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 3, characterized in that: During the traversal in step S3, if the framing function CJsonFunc in the currently traversed configuration item indicates that the current node is an object, then the current node is added to the parent node as an object through the framing function CJsonFunc. Then, the construction function is recursively called on each element in the object association information NextTable to obtain the members of the current node.

5. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 3, characterized in that: During the traversal in step S3, if the framing function CJsonFunc in the currently traversed configuration item indicates that the current node is an array, then the current node is added to the parent node as an array through the framing function CJsonFunc. Then, the construction function is recursively called on each element in the object association information NextTable to obtain the array elements of the current node.

6. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 3, characterized in that, The configuration table also contains the following fields: DataType: Defines the data source for the target data of the current node; State Buffer Index ValueType: Defines the target data for the current node in the state buffer; valid only when the data source type DataType indicates that the data source is a state buffer.

7. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 6, characterized in that: During the traversal in step S3, if the current node is a key-value pair, the current node is added to the parent node as a key-value pair through the frame function CJsonFunc, and the reference information of the target data is obtained through the data source type DataType and saved to the current node.

8. The configurable recursive IoT data frame construction method for electricity meters and electricity consumption acquisition terminals as described in claim 1, characterized in that, Also includes: Step S0: Receive a configuration update instruction from an external source; dynamically update the configuration table in the device according to the update instruction, thereby changing the format of the JSON data frame without modifying and recompiling the device firmware.

9. A configurable recursive Internet of Things (IoT) data frame construction system for electricity meters and electricity consumption acquisition terminals, characterized in that, include: The configuration table management module is used to store and maintain configuration tables. Each configuration table defines multiple configuration items, and each configuration item corresponds to a node in a JSON data frame. Each configuration item includes at least structure node configuration items and data node configuration items. The structure node configuration items correspond to objects and arrays in JSON format and contain association information pointing to other configuration items. The data node configuration items correspond to key-value pairs in JSON format and contain data retrieval method information. The state buffer is a statically pre-allocated memory area with a predefined data structure, used to store the acquired data in real time. A recursive JSON construction module, communicating with the configuration table management module and the state buffer, is used to respond to a data frame construction request by calling a construction function to perform recursive traversal driven by the configuration table to generate a reference-based data object tree; wherein, the recursive JSON construction module is configured as follows: When traversing to the configuration item of the structure node, the construction function is recursively called on the configuration items corresponding to each of its child nodes according to its association information to generate a nested data structure; When traversing to a data node configuration item, the corresponding data source is associated with its data acquisition method information, and the reference information of the target data is saved to the current node; when the data source is a state buffer, the reference information is a pointer to the target data in the state buffer; The serialization and output module, which is connected to the recursive JSON construction module, is used to serialize the referenced data object tree into a data frame and write it into a statically pre-allocated output buffer. During the serialization process, the target data is read from the data source according to the reference information stored in the node and written into the output buffer.

10. The configurable recursive IoT data frame construction system for electricity meters and electricity consumption acquisition terminals as described in claim 9, characterized in that: The configuration table management module also supports receiving configuration update instructions from external sources via wireless or wired communication interfaces, and dynamically updating the configuration table accordingly, thereby changing the format of the JSON data frame without modifying or recompiling the system firmware.

Images