Method and device for processing data in building system, equipment and storage medium

Abstract

This application relates to a method, apparatus, device, and storage medium for processing data in a building system. The method includes: parsing the data structure of data packets reported by IoT devices in the building system based on preset protocol parsing rules to obtain corresponding data information, and generating a structural description model based on the data information; writing the data in the data packets into a memory buffer pool based on a double buffering mechanism, and, if a raster structure matching the structural description model exists in the system, batch writing data from the memory buffer pool into the raster structure according to preset rules; if no raster structure matching the structural description model exists in the system, constructing a raster structure matching the structural description model based on field information in the data information, and batch writing data from the memory buffer pool into the raster structure according to preset rules; and performing data querying in the raster structure based on an index corresponding to a custom field.

Description

Technical Field

[0001] This application relates to the field of data processing in building systems, and more particularly to a method, apparatus, device, and storage medium for data processing in building systems. Background Technology

[0002] In building automation systems, equipment types are diverse and operating states are complex, involving protocol data that covers multiple dimensions, including sensor-collected data, control command issuance, equipment status feedback, and alarm information. This data is not only widely sourced but also varies in structure, often containing nested structures, variable fields, and a large amount of unstructured content, such as JSON messages, nested data blocks, and custom byte streams. Existing storage methods mostly employ relational databases or fixed-length structure models, offering good stability and storage efficiency when processing business data with fixed structures and clearly defined fields. However, facing real-world building automation applications with diverse protocols, frequent field changes, and complex data formats, these fixed-structure storage methods often struggle to adapt flexibly, leading to wasted space, low read / write performance, and insufficient support for unstructured data, thus limiting the system's scalability and real-time response capabilities.

[0003] There are currently no effective solutions to the aforementioned technical problems in the related technologies. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for processing data in a building system, in order to solve the problem that the data storage in the existing building system adopts a fixed structure storage method, which easily leads to space waste and low read / write performance.

[0005] In a first aspect, this application provides a data processing method for a building system, comprising: parsing the data structure of data packets reported by IoT devices in the building system based on preset protocol parsing rules to obtain corresponding data information, and generating a structural description model based on the data information; writing the data in the data packets into a memory buffer pool based on a double buffering mechanism, and, if a grid structure matching the structural description model exists in the system, writing data in batches from the memory buffer pool into the grid structure according to preset rules; if no grid structure matching the structural description model exists in the system, constructing a grid structure matching the structural description model based on field information in the data information, and writing data in batches from the memory buffer pool into the grid structure according to preset rules; and performing data querying in the grid structure based on an index corresponding to a custom field.

[0006] Secondly, this application provides a data processing device for a building system, comprising: a first processing module, configured to parse the data structure of data packets reported by IoT devices in the building system based on preset protocol parsing rules to obtain corresponding data information, and generate a structure description model based on the data information; a second processing module, configured to write the data in the data packets into a memory buffer pool based on a double buffering mechanism, and, if a grid structure matching the structure description model exists in the system, to write data in batches from the memory buffer pool into the grid structure according to preset rules; a third processing module, configured to construct a grid structure matching the structure description model based on field information in the data information, and to write data in batches from the memory buffer pool into the grid structure according to preset rules, if no grid structure matching the structure description model exists in the system; and a fourth processing module, configured to perform data querying in the grid structure based on an index corresponding to a custom field.

[0007] Thirdly, this application provides an apparatus comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the data processing method in the building system described in the first aspect of this application.

[0008] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the data processing method in the building system described in the first aspect of this application.

[0009] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application parses the data structure of data packets reported by IoT devices in a building system based on preset protocol parsing rules to obtain corresponding data information, and generates a structural description model based on the data information. Then, based on a double buffering mechanism, the data in the data packets is written into a memory buffer pool. If a grid structure matching the structural description model exists in the system, data is written in batches from the memory buffer pool into the grid structure according to preset rules. If no grid structure matching the structural description model exists in the system, a grid structure matching the structural description model is constructed based on the field information in the data information, and data is written in batches from the memory buffer pool into the grid structure according to preset rules. It can be seen that in this application, different protocol parsing rules parse the data structure of different types of data packets, thereby obtaining structural description models corresponding to different protocols. This allows for efficient adaptation and space optimization of the subsequently generated grid structure for multiple protocols and formats. Compared with the fixed-structure storage method used in existing building systems, the method in this application significantly improves read / write performance and optimizes space. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0013] Figure 1 A flowchart illustrating a data processing method in a building system, provided as an embodiment of this application;

[0014] Figure 2 An optional flowchart of a data processing method in a building system provided in an embodiment of this application;

[0015] Figure 3 A schematic diagram of the structure of a grid-based storage system based on a digital base provided in this application embodiment;

[0016] Figure 4 A flowchart illustrating the grid partitioning and storage method based on a digital base provided in this application embodiment;

[0017] Figure 5 A schematic diagram of the grid structure provided in the embodiments of this application;

[0018] Figure 6 This is a schematic diagram of the structure of a data processing device in a building system provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] To address the problems of wasted space and low read / write performance caused by the fixed storage structure in existing building systems, this application provides a data processing method for building systems, such as... Figure 1 As shown, the steps of this method include:

[0023] Step 101: Based on the preset protocol parsing rules, parse the data structure of the data packets reported by IoT devices in the building system to obtain the corresponding data information, and generate a structure description model based on the data information.

[0024] In this embodiment, the preset protocol parsing rules can be determined based on protocols such as BACnet, Modbus, and OPC UA. Furthermore, the protocol can also be a proprietary protocol or a custom throttling structure. Further, the data information in this embodiment can include field type, field level, nesting relationship, field frequency, etc. That is, in this embodiment, information such as field type, field level, nesting relationship, and field frequency can be extracted using the protocol structure parsing module. Therefore, in this embodiment, different protocol parsing rules can be used to parse the data structure of different types of data packets, thereby obtaining structural description models corresponding to different protocols. This allows for efficient adaptation and space optimization of the subsequently generated raster structure for multiple protocols and formats.

[0025] It should be noted that the structural description model in this application embodiment is a model of abstract data structure description information constructed by the system, which is used to guide the subsequent grid division and storage unit organization and layout.

[0026] Step 102: Based on the double buffering mechanism, write the data in the data packet into the memory buffer pool, and if there is a raster structure in the system that matches the structural description model, write the data in batches from the memory buffer pool into the raster structure according to the preset rules.

[0027] In the embodiments of this application, the double buffering mechanism refers to the system using two memory buffers (such as Buffer A and Buffer B) to work alternately in order to reduce write waiting time and improve concurrency performance.

[0028] Step 103: If there is no raster structure in the system that matches the structural description model, construct a raster structure that matches the structural description model based on the field information in the data information, and write data in batches from the memory buffer pool into the raster structure according to preset rules.

[0029] In this embodiment, field information may include the number of fields, data structure complexity, field access frequency, etc. Each grid can be understood as a pre-configured storage unit for a certain type of data structure. Therefore, each structure description model has a corresponding grid structure, i.e., dynamically constructed storage grids that match the protocol.

[0030] Step 104: Perform data retrieval in the raster structure based on the index corresponding to the custom field.

[0031] In this embodiment, the index corresponding to the custom field can be an inverted index for indexing discrete value fields, or a B+ tree index for indexing ordered fields. Different indexes serve different purposes; for example, an inverted index is suitable for quickly locating a set of records containing a specific value, while a B+ tree index is suitable for range queries and sorted queries.

[0032] Through steps 101 to 104 above, the data structure of data packets reported by IoT devices in the building system is parsed based on preset protocol parsing rules to obtain corresponding data information, and a structural description model is generated based on the data information. Then, based on a double buffering mechanism, the data in the data packets is written into a memory buffer pool. If a raster structure matching the structural description model exists in the system, data is written in batches from the memory buffer pool into the raster structure according to preset rules. If no raster structure matching the structural description model exists in the system, a raster structure matching the structural description model is constructed based on the field information in the data information, and data is written in batches from the memory buffer pool into the raster structure according to preset rules. It can be seen that in this application, different protocol parsing rules parse the data structure of different types of data packets, thereby obtaining structural description models corresponding to different protocols. This allows for efficient adaptation and space optimization of the subsequently generated raster structure for multiple protocols and formats. Compared to the fixed-structure storage method used in existing building systems, the method in this embodiment significantly improves read / write performance and achieves better space optimization.

[0033] In this embodiment of the application, the method of parsing the data structure of the data packets reported by IoT devices in the building system based on preset protocol parsing rules to obtain the corresponding data information, and generating a structure description model based on the data information, as involved in step 101 above, may further include:

[0034] Step 11: Use the plug-in adapter to identify the protocol type of the data packets in order to determine the corresponding protocol parsing rules;

[0035] Step 12: Extract at least one of the following data information from the data packet based on the protocol parsing rules: field type, field level, field nesting relationship, and field frequency;

[0036] In this specific example, the field type in the embodiments of this application refers to the specific type of data contained in the field, such as basic types: `int`, `float`, `boolean`, `string`; time types: `timestamp`, `datetime`, etc.; array or list types: such as `[int]` representing an integer array; structure types: nested objects or composite types; binary streams: such as some custom protocols that package fields into a `byte[]`.

[0037] Furthermore, the field hierarchy in this application reflects the nesting depth and parent-child relationship of fields in the message structure, such as when applied to formats like JSON or XML. For example (OPC UA message structure): `device.id` level 1, `device.metrics.temperature` level 2.

[0038]

[0039] In this application's embodiments, nesting refers to a situation where the value of a field is itself an object or array, reflecting the inclusion relationship between fields. For example, in the above example, `metrics` is a subfield of `device`, thus constituting nesting.

[0040] In this application's embodiments, field frequency refers to the frequency of a field's appearance in similar messages. The system can analyze this through sampling statistics. For example, high-frequency fields appear in almost every message, such as `timestamp`, `value`, and `device_id`. Low-frequency fields appear occasionally, such as `errorCode` and `comment`. High-frequency fields can be preferentially included in the "fixed area" of the grid to improve access efficiency.

[0041] Step 13: Serialize the extracted data information structure to generate the corresponding structure hash, and encapsulate the data information and structure hash into a structure description model.

[0042] In a specific example, the fields included in a structural description model are shown in Table 1.

[0043]

[0044]

[0045] Table 1

[0046] As can be seen, the structural description model in this application embodiment is a model of abstract data structure description information constructed by the system. It is used to guide the subsequent organization and layout of the grid structure, so that the data storage in the grid structure is organized and planned, thereby enabling the data to be effectively and reasonably classified and stored, and improving space utilization.

[0047] The method of writing data from a data packet into a memory buffer pool based on a double buffering mechanism, as mentioned in step 102 above, may further include:

[0048] Step 21: Write the data in the data packet into the first memory buffer;

[0049] Step 22: If the data in the first memory buffer meets the write threshold or reaches the timed trigger condition, write the data in the first memory buffer into the grid structure, and at the same time write the new data into the second memory buffer.

[0050] For steps 21 and 22 above, in a specific example, the first memory buffer is Buffer A, and the second memory buffer is Buffer B. Based on this, the system writes received data (such as a set of temperature and humidity data packets from a device) to Buffer A during runtime. When Buffer A meets the write threshold (such as 10MB or 1000 records) or reaches the timed trigger condition (such as 1 second), the system begins to batch flush the data in Buffer A to the backend storage (such as writing to the raster data block area). While Buffer A is being flushed, new data is immediately switched to Buffer B to avoid waiting for the write to complete, thereby eliminating write blocking. After Buffer A is written and cleared, it becomes the active buffer again, and the two alternate to maintain write continuity. It can be seen that the dual-buffering mechanism refers to the system using two memory buffers to work alternately to reduce write waiting time and improve concurrency performance.

[0051] In this embodiment of the application, the method of constructing a raster structure that matches the structural description model based on the field information in the data information in step 103 above, and writing data in batches from the memory buffer pool into the raster structure according to preset rules, may further include:

[0052] Step 31: Extract at least one of the following information from the field information: number of fields, data structure complexity, and field access frequency;

[0053] In this embodiment, the number of fields is used to determine the "number of columns" or the number of field slots in the raster. When there are fewer fields (<10), a flat structure can be used, which can be loaded into memory all at once. However, when there are more fields (>50), multi-level partitioning is required, such as a main field area and an auxiliary field area, to prevent the structure from becoming too large and causing access delays.

[0054] The data structure complexity in this application embodiment includes: nesting depth (e.g., field hierarchy), dynamic fields (e.g., variable-length arrays, JSON blocks), and type diversity (mixed-type fields, such as int+string+object). Complex data structures are handled in the following ways: 1) Using multi-level raster mapping: e.g., the first-level raster stores basic fields, and the second-level raster stores nested blocks; 2) Setting up variable-length field mapping pointer areas: for variable-length or nestable fields, offset pointers or block-level indexes are used. The data block area can be stored using a combination of embedded structure and reference structure.

[0055] In this embodiment, the frequency of fields determines their arrangement order and caching optimization strategy to improve access efficiency for high-frequency fields. High-frequency fields are placed at the beginning of the data block area for quick location; they can be accessed using a memory-mapped table or redundantly stored in the index area to improve retrieval performance. Low-frequency fields are arranged later or lazy-loaded.

[0056] Step 32: Determine the field layout and construct the metadata definition in the raster structure based on the extracted information, and initialize the storage structure in the raster structure based on the field layout and metadata definition. The storage structure includes the metadata area, index area, data block area and extension area.

[0057] As can be seen, the grid structure specifically consists of:

[0058] 1) Metadata area: Stores field structure (such as type, order, whether nested, whether variable length), mapping information, raster version, etc.

[0059] 2) Index: Stores the primary key (such as a combination of timestamp and device ID), offset, and pointers to variable-length fields for each piece of data. The index is used to speed up queries and write location.

[0060] 3) Data Block Area: The area where field values ​​are actually stored, using a fixed-length or a hybrid fixed-length + variable-length mode. Fields can be sorted by frequency, and nested fields can point to sub-blocks or the outer extension area.

[0061] 4) Extension Area: Used to store fields, additional metadata, logs, etc., added after protocol evolution. Supports dynamic expansion without reconstructing the original data.

[0062] As can be seen, in this embodiment of the application, by establishing an independent raster index table for each type of protocol data, access scheduling can be performed by combining dimensions such as data source, timestamp, and field mapping, thereby achieving fast data location and lock-free concurrent read and write.

[0063] Step 33: Write data in batches from the memory buffer pool into the corresponding raster structure using a structure matching method.

[0064] Step 33 refers to the batch data being uniformly written to the corresponding raster structure (i.e., raster cells with the same metadata definition and field index) in a structure matching manner, avoiding repeated structure checks and index positioning in a one-time operation.

[0065] As can be seen, the grid structure in this embodiment can independently define field mapping relationships and storage boundaries, and can adjust its size and distribution in real time according to data changes, thereby achieving efficient adaptation and space optimization for multi-protocol and multi-format data. This mechanism significantly improves structural adaptability and storage space utilization, avoiding problems such as redundant fields and frequent table structure changes that exist in traditional fixed-length structures.

[0066] The method of querying data in the raster structure based on the index corresponding to the custom field involved in step 104 of this application embodiment may further include:

[0067] Step 41: Generate at least one of the following indexes based on the custom field: inverted index, B+ tree index, hash index;

[0068] Step 42: Perform data query in the raster structure based on the index to obtain query results, wherein the data in the query results comes from one or multiple raster structures with a structural similarity greater than a first preset threshold.

[0069] As can be seen in the application embodiments, when each grid structure is created, the system automatically generates an index based on the fields defined in the protocol structure, which may specifically include:

[0070] 1) Inverted index: Used for discrete value fields, such as "device ID", "fault code", "device type", etc., suitable for quickly locating a set of records containing a specific value.

[0071] 2) B+ tree index: Used for ordered fields, such as "timestamp", "temperature value", "pressure value", etc., suitable for range queries and sorted queries.

[0072] 3) Hash index: Used for frequently queried fields, such as "device UUID" or "controller MAC".

[0073] Here's an example of a query type:

[0074] 1) Time-based queries: For example, a user might want to retrieve all air conditioning temperature data for a specific floor from 00:00 on June 1, 2025 to 00:00 on June 2, 2025. The system then uses a B+ tree index on the timestamp field to directly locate the range of records within that time interval. To avoid a full raster scan, only the data within the indexed time period is accessed, significantly improving query efficiency.

[0075] 2) The device-level aggregation algorithm calculates the average supply air temperature of all RTUs (Rooftop Air Conditioners) over the past 24 hours. It filters out the corresponding raster cells using the inverted index of "Device Type = RTU"; reads the target field "Supply Air Temperature" and calls the aggregation engine to calculate the average; if a device spans multiple raster cells, the query engine will call the cross-raster queryer to merge fields with consistent or similar structures before processing.

[0076] In this embodiment, the constructed grid structure can also be dynamically adjusted to expand, merge, or migrate the grid structure to meet the needs of real-time data. Based on this, such as Figure 2 The method described in this application embodiment may further include:

[0077] Step 201: Dynamically adjust the raster structure by real-time detection of data write volume, field change frequency, and raster load.

[0078] In this regard, the method of dynamically adjusting the raster structure by real-time detection of data write volume, field change frequency, and raster load in step 201 above can further include:

[0079] Step 51: If the amount of data stored in the raster structure exceeds the second preset threshold, the raster structure is split into multiple sub-raster structures.

[0080] Step 52: Merge grid structures with structural similarity less than the third preset threshold;

[0081] Step 53: In the event of a protocol version upgrade or a change in field hierarchy, remap to the new raster structure and smoothly migrate the data in the old raster structure to the new raster structure.

[0082] As can be seen, in this embodiment, by monitoring data write volume, field change frequency, and raster load in real time, the system can perform the following operations: Splitting: When a raster stores too much data or field redundancy increases, the system will automatically split it into multiple sub-rasters; Merging: If the structural differences between multiple rasteres are small, the system can merge them into a unified raster, saving storage space; Remapping: When the protocol version changes, it can automatically remap to the new raster template and smoothly migrate the original data. Driven by both the policy engine and the scheduling thread, the system ensures that it always maintains the optimal storage layout and data access efficiency.

[0083] The present application will now be explained and described in conjunction with specific embodiments of the present application. These specific embodiments provide a grid-based storage method and system based on a digital base, wherein, for example... Figure 3 As shown, the system includes:

[0084] The device / controller module is used to collect or upload raw protocol data.

[0085] The protocol parsing module is used to identify the data structure characteristics of different protocols.

[0086] The grid management module is responsible for matching or generating corresponding grid cells based on the parsed structure.

[0087] The data writing module is used to organize data into a raster and then efficiently write it to disk.

[0088] The underlying storage includes databases or caching systems that support persistent storage of structured and unstructured data.

[0089] Based on the above Figure 3 The system in this specific embodiment uses a grid-based storage method based on a digital base, such as... Figure 4 As shown, it includes the following steps:

[0090] Step 401, Protocol Structure Parsing: Receive data packets reported from various IoT devices (such as sensors, controllers, actuators, etc.) in the building system, and analyze the data structure based on preset or dynamically loaded protocol parsing rules. Specific protocols can include BACnet, Modbus, OPC UA, etc., and private protocols or custom byte stream structures are also supported. The protocol structure parsing module extracts information such as field type, field hierarchy, nesting relationships, and field frequency, and generates a structural description model to provide input for subsequent grid division.

[0091] Furthermore, the process for generating the structural description model is as follows:

[0092] 1) Data reception: The protocol structure parsing module receives one or more raw data packets.

[0093] 2) Protocol identification: Use the plug-in adapter to identify the protocol type (such as Modbus, BACnet, etc.) and determine which parsing rules to use.

[0094] 3) Field extraction: Use the corresponding parser to extract field names, types, and nesting structures; construct a list of fields and their hierarchical mapping tree.

[0095] 4) Frequency analysis: For continuous sample data, count the number of times a field appears and mark high-frequency fields; a window size such as 1000 rows can be set and a sliding window method can be used for calculation.

[0096] 5) Structure hash generation: After serializing the field structure, a structure hash (such as SHA256) is generated for the reuse and recognition of raster templates.

[0097] 6) Model encapsulation: The above information is encapsulated into a structural description model object for use by the grid division module.

[0098] Step 402, Raster Mapping and Partitioning: After receiving the structural description model, the raster partitioning module constructs a matching raster structure for the protocol data based on parameters such as the number of fields, data structure complexity, and field access frequency.

[0099] Each grid is a pre-configured storage unit for a specific type of data structure, such as... Figure 5 As shown, the raster structure can include: Metadata area: stores structural information such as field definitions, field order, and field types. Index area: records the primary key of the data (such as timestamp, device identifier) ​​and field quick location information. Data block area: used for actual data storage, supporting mixed configuration of fixed-length and variable-length fields. Extension area: reserved for dynamic adjustment needs such as protocol version upgrades and insertion of new fields.

[0100] To illustrate this, let's take an air conditioning controller device in a smart building system as an example. It periodically reports data packets with the following structure (using a custom protocol):

[0101]

[0102] Specific composition and information of each region within the grid structure:

[0103] 1) Metadata area

[0104] List of field names: The field names and hierarchical structure supported by this raster, such as timestamp, device_id, status.mode, status.setpoint.

[0105] Field type: The data type corresponding to each field (int, float, string, array, object). For example, timestamp → int64, status.setpoint → float32.

[0106] Field order: Storage order, which affects the arrangement of data blocks, for example 1. timestamp → 2. device_id → 3. status.mode.

[0107] Hierarchical structure information: Indicates nesting relationships and the affiliation of subfields. For example, status is an object, which includes mode, fan_speed, etc.

[0108] Whether it is a variable-length field: indicates whether it is a variable-length field (such as an array or a string). For example, alerts[*] → yes, device_id → yes.

[0109] Protocol identifier: Which protocol type is supported (for adaptation purposes), for example, `"custom_protocol_ac_v1"`.

[0110] 2) Index area

[0111] Primary key field value: The key field value used as the primary index (such as timestamp + device ID), for example `1718179200#AC_203A`.

[0112] Data block offset address: The starting offset position (byte offset) of the current data record in the data block area, for example, `offset:32840`.

[0113] Variable field index: Points to information such as block address, length, and count of a variable field. For example, alerts: offset = 32860, count = 1.

[0114] Hash value or digest: Used for data integrity verification or to speed up lookups, for example, hash(timestamp+device_id).

[0115] Conditional bitmap / label index: Optional, sets conditional labels for fields (such as whether to issue an alarm), for example, status.mode="cool" → bit=1.

[0116] 3) Data block area

[0117] In the actual data storage area, each record represents a mapping of uploaded data within this area. The field order and layout are defined by the metadata area.

[0118] For example, timestamp: `1718179200`.

[0119] 4) The extended area is used to support protocol upgrades, field additions, nested complex structure external storage, cross-version mapping, etc., which is equivalent to "the overflow space of the raster".

[0120] Step 403, Dynamic Adjustment and Scheduling Mechanism: Supports dynamic expansion, merging, and migration of the grid structure.

[0121] As can be seen, in this embodiment, by monitoring data write volume, field change frequency, and raster load in real time, the system can perform the following operations: Splitting: When a raster stores too much data or field redundancy increases, the system will automatically split it into multiple sub-rasters. Merging: If the structural differences between multiple rasteres are small, the system can merge them into a unified raster, saving storage space. Remapping: When the protocol version changes, it can automatically remap to the new raster template and smoothly migrate the original data.

[0122] In the case of grid splitting, taking the reported data of a certain type of air conditioning equipment as an example, the initial data structure had few fields, only including basic information such as temperature, set temperature, and operating status. However, with system upgrades, some equipment began to report fields such as energy consumption data and maintenance information.

[0123] Specific splitting process: The system monitors the field distribution in the raster. For example, raster ID-AC-V1 has a total of 30 fields, of which only 8 are high-frequency fields. Policy triggering: The policy engine determines that the "sparseness threshold" has been reached and triggers the split. Raster replication: Two new sub-raster templates are created: `Raster-AC-Standard` and `Raster-AC-Extended`. Data migration: Data from the original raster is migrated to the two new rasters, and data blocks are rewritten based on device field matching. Index update: The old index points to the new raster location, and the original raster is marked as "frozen" or pending cleanup.

[0124] Raster merging process: For example, if a system contains two fan coil unit protocols (Modbus v1 and v1.1) with very minor field differences, except for an additional "motor status" field, after prolonged operation, two similar raster structures exist, consuming resources. When multiple raster structures have high similarity (high consistency in field names, types, and order), or when raster storage density and write frequency are low, a unified template compatibility can be achieved through minor structural adjustments. The entire merging process involves: analyzing the field sets of the two raster templates and calculating the similarity (e.g., ≥90%); constructing a "compatible template," introducing new fields, and setting default values ​​for filling; building a new metadata area to integrate field definitions and hierarchical structures; data rewriting and migration: supplementing v1 data with default field values, maintaining the integrity of v1.1 data, and writing it into a unified new raster structure; index updates: resetting the offset to point to the new raster; marking the original raster as merged obsolete and periodically clearing it; and remapping: when protocol versions are upgraded or field hierarchies change.

[0125] Step 404, Writing and Caching: To improve write performance, the system introduces a double-buffering mechanism and a batch write strategy. Before data is written to the database, it is first written to a memory buffer pool and then flushed to the backend storage in batches according to time or batch thresholds. Multi-threaded concurrent writing is supported, and data validation and field integrity checks are embedded in the write path to ensure data consistency.

[0126] In this embodiment, the batch writing of data aims to reduce the impact of frequent write operations on the storage system and improve disk I / O utilization. The specific batch writing logic includes the following process:

[0127] 1) Data aggregation: The system aggregates multiple records from the same device type or the same data structure together. For example, data reported by 100 temperature and humidity sensors in a building within 1 second will be aggregated into a single batch.

[0128] 2) Batch data entry trigger conditions: Data count threshold: For example, trigger a write operation every 500 data entries. Time interval threshold: For example, force data entry every 1 second, regardless of the number of data entries.

[0129] 3) Write operation: Batch data will be uniformly written to the corresponding raster structure (i.e., raster cells with the same metadata definition and field index) in a structure matching manner, avoiding repeated structure checks and index location in a one-time operation.

[0130] Step 405, Querying and Indexing: Supports query indexing mechanisms based on structured fields. When each raster is created, an inverted index or B+ tree index is automatically generated according to the field definition to support needs such as time-period queries, device-based aggregation, and specific field filtering. It supports cross-raster query engines, aggregating structurally similar data and presenting it uniformly, facilitating data comparison and analysis across multiple protocols.

[0131] As can be seen, the specific implementation of this application provides a dynamic grid partitioning mechanism oriented towards protocol structures. This mechanism dynamically constructs a storage grid that matches the protocol structure by parsing information such as protocol field types, data lengths, and nesting relationships. Each grid can independently define field mapping relationships and storage boundaries, and its size and distribution can be adjusted in real time according to data changes, thereby achieving efficient adaptation and space optimization for multi-protocol and multi-format data. This mechanism significantly improves structural adaptability and storage space utilization, avoiding problems such as redundant fields and frequent table structure changes present in traditional fixed-length structures.

[0132] Furthermore, this application also provides a high-performance read / write scheduling mechanism based on raster indexes, which improves the processing capability of large-scale protocol data in high-frequency acquisition and real-time response scenarios. Specifically, by establishing an independent raster index table for each type of protocol data and combining data source, timestamp, field mapping, and other dimensions for access scheduling, it achieves rapid data location and lock-free concurrent read / write. Simultaneously, this mechanism supports batch writing, conditional retrieval, and range scanning, greatly improving the system's throughput and stability in high-concurrency environments.

[0133] Corresponding to the above Figure 1 This application also provides a data processing device for a building system, such as... Figure 6 As shown, the device includes:

[0134] The first processing module 602 is used to parse the data structure of the data packets reported by IoT devices in the building system based on preset protocol parsing rules, obtain the corresponding data information, and generate a structure description model based on the data information.

[0135] The second processing module 604 is used to write data in the data packet into the memory buffer pool based on the double buffering mechanism, and when there is a raster structure in the system that matches the structural description model, to write data from the memory buffer pool into the raster structure in batches according to preset rules.

[0136] The third processing module 606 is used to construct a grid structure that matches the structural description model based on the field information in the data information when there is no grid structure in the system that matches the structural description model, and to write data in batches from the memory buffer pool into the grid structure according to preset rules.

[0137] The fourth processing module 608 is used to perform data queries in the raster structure based on the index corresponding to the custom field.

[0138] In an optional embodiment of this application, the first processing module may further include: a first processing unit, configured to use a plug-in adapter to identify the protocol type of the data packet to determine the corresponding protocol parsing rules; a second processing unit, configured to extract at least one of the following data information from the data packet based on the protocol parsing rules: field type, field level, field nesting relationship, and field frequency; and a third processing unit, configured to serialize the extracted data information structure to generate a corresponding structure hash, and encapsulate the data information and the structure hash into a structure description model.

[0139] In an optional embodiment of this application, the second processing module in this application embodiment may further include: a fourth processing unit, used to write data in the data packet into a first memory buffer; and a fifth processing unit, used to write the data in the first memory buffer into a grid structure when the data in the first memory buffer meets the writing threshold or reaches the timed trigger condition, and at the same time write new data into a second memory buffer.

[0140] In an optional embodiment of this application, the third processing module in this application embodiment may further include: a sixth processing unit, used to extract at least one of the following information from the field information: number of fields, data structure complexity, and field access frequency; a seventh processing unit, used to determine the field layout and construct the metadata definition in the raster structure based on the extracted information, and initialize the storage structure in the raster structure based on the field layout and metadata definition, wherein the storage structure includes a metadata area, an index area, a data block area, and an extension area; and an eighth processing unit, used to write data in batches from the memory buffer pool into the corresponding raster structure in a structure matching manner.

[0141] In an optional embodiment of this application, the fourth processing module in this application embodiment may further include: a ninth processing unit, used to generate at least one of the following indexes based on a custom field: an inverted index, a B+ tree index, and a hash index; and a tenth processing unit, used to perform data querying in a raster structure based on the index to obtain query results, wherein the data in the query results comes from one or more raster structures with a structural similarity greater than a first preset threshold.

[0142] In an optional embodiment of this application, the apparatus may further include: a fifth processing module, used to dynamically adjust the grid structure by real-time detection of data writing volume, field change frequency and grid load in the grid structure.

[0143] In an optional embodiment of this application, the fifth processing module may further include: an eleventh processing unit, configured to split the raster structure into multiple sub-rasteres when the amount of data stored in the raster structure exceeds a second preset threshold; a twelfth processing unit, configured to merge raster structures with a structural similarity less than a third preset threshold; and a thirteenth processing unit, configured to remap to a new raster structure and smoothly migrate data from the old raster structure to the new raster structure when the protocol version is upgraded or the field level changes.

[0144] like Figure 7 As shown in the figure, this application provides a device including a processor 711, a communication interface 712, a memory 713, and a communication bus 714, wherein the processor 711, the communication interface 712, and the memory 713 communicate with each other through the communication bus 714.

[0145] Memory 713 is used to store computer programs;

[0146] In one embodiment of this application, when the processor 711 executes the program stored in the memory 713, it implements the data processing method in the building system provided in any of the aforementioned method embodiments, and its function is similar, so it will not be described again here.

[0147] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method in a building system as provided in any of the foregoing method embodiments.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0150] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0151] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing data in a building system, characterized in that, include: The data structure of data packets reported by IoT devices in the building system is parsed based on preset protocol parsing rules to obtain corresponding data information, and a structural description model is generated based on the data information. The data in the data packet is written into the memory buffer pool based on the double buffering mechanism. If a grid structure matching the structure description model exists in the system, the data is written into the grid structure in batches from the memory buffer pool according to the preset rules. If no raster structure matching the structural description model exists in the system, a raster structure matching the structural description model is constructed based on the field information in the data information, and data is written into the raster structure in batches from the memory buffer pool according to preset rules. Data queries are performed in the raster structure based on the index corresponding to the custom field.

2. The method according to claim 1, characterized in that, Based on preset protocol parsing rules, the data structure of data packets reported by IoT devices in the building system is parsed to obtain the corresponding data information, and a structural description model is generated based on the data information, including: The plugin adapter is used to identify the protocol type of the data packet in order to determine the corresponding protocol parsing rules; Based on the protocol parsing rules, at least one of the following data information is extracted from the data packet: field type, field level, field nesting relationship, and field frequency; The extracted data information is serialized to generate a corresponding structure hash, and the data information and the structure hash are encapsulated into the structure description model.

3. The method according to claim 1, characterized in that, The data in the data packet is written into the memory buffer pool based on a double buffering mechanism, including: Write the data in the data packet into the first memory buffer; When the data in the first memory buffer meets the write threshold or reaches the timed trigger condition, the data in the first memory buffer is written to the grid structure, and new data is written to the second memory buffer at the same time.

4. The method according to claim 1, characterized in that, Based on the field information in the data information, a raster structure matching the structural description model is constructed, and data is written in batches from the memory buffer pool into the raster structure according to preset rules, including: Extract at least one of the following information from the field information: number of fields, data structure complexity, and field access frequency; Based on the extracted information, the field layout is determined and the metadata definition in the raster structure is constructed. The storage structure in the raster structure is initialized based on the field layout and the metadata definition. The storage structure includes a metadata area, an index area, a data block area, and an extension area. Data is written in batches from the memory buffer pool to the corresponding raster structure using a structure matching method.

5. The method according to claim 1, characterized in that, Data queries are performed in the raster structure based on the index corresponding to the custom field, including: Generate an index based on at least one of the following: inverted index, B+ tree index, or hash index; Based on the index, data is queried in the raster structure to obtain query results, wherein the data in the query results comes from one or multiple raster structures with a structural similarity greater than a first preset threshold.

6. The method according to claim 1, characterized in that, The method further includes: The raster structure is dynamically adjusted by real-time monitoring of data write volume, field change frequency, and raster load.

7. The method according to claim 6, characterized in that, The raster structure is dynamically adjusted by real-time monitoring of data write volume, field change frequency, and raster load, including: If the amount of data stored in the grid structure exceeds a second preset threshold, the grid structure will be split into multiple sub-grids. Merge grid structures whose structural similarity is less than the third preset threshold; In the event of protocol version upgrades or field hierarchy changes, data is remapped to the new raster structure, and data in the old raster structure is smoothly migrated to the new raster structure.

8. A data processing device for a building system, characterized in that, include: The first processing module is used to parse the data structure of the data packets reported by IoT devices in the building system based on preset protocol parsing rules, obtain the corresponding data information, and generate a structure description model based on the data information. The second processing module is used to write the data in the data packet into the memory buffer pool based on the double buffering mechanism, and when there is a grid structure in the system that matches the structure description model, to write the data from the memory buffer pool into the grid structure in batches according to preset rules. The third processing module is used to construct a grid structure that matches the structure description model based on the field information in the data information when there is no grid structure in the system that matches the structure description model, and to write data in batches from the memory buffer pool into the grid structure according to preset rules. The fourth processing module is used to perform data queries in the raster structure based on the index corresponding to the custom field.

9. A data processing device for a building system, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to perform a data processing method in the building system according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The system stores computer-executable instructions for performing a data processing method in the building system according to any one of claims 1 to 7.

Images