Iot device access system supporting custom heterogeneous network protocol components

Abstract

The application relates to an Internet of Things device access system supporting customized heterogeneous network protocol components. The system comprises a protocol component library, wherein each protocol component interfaces a plurality of Internet of Things devices using a corresponding communication protocol and is configured with a metadata description file, a protocol component management module for performing life cycle management on the protocol components, adjusting running resource quotas according to an optimization strategy, a data analysis module for automatically matching and analyzing original data based on metadata fields according to analysis rules, and adjusting an analysis rule chain according to an optimization strategy, an interface output module for determining output channels and parameter output standardized data based on metadata fields, and adjusting the output channels and parameters according to an optimization strategy, and a scheduling optimization module for continuously collecting running state data of each module to generate an optimization strategy and feeding back the optimization strategy to each module. The system can realize fast access of heterogeneous devices and continuous self-optimization of the system in a multi-protocol device coexistence scene.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, and in particular to an IoT device access system that supports custom heterogeneous network protocol components. Background Technology

[0002] With the rapid development of IoT technology, technologies for large-scale IoT device access have emerged. These technologies are characterized by an explosive growth in the types and numbers of connected devices, which come from different manufacturers and follow various communication standards and protocols such as MQTT, CoAP, HTTP / HTTPS, LoRaWAN, Zigbee, NB-IoT, and various proprietary TCP / UDP protocols. This heterogeneity in device communication is becoming increasingly pronounced, leading to the current industry's reliance on traditional systems such as single-protocol gateways, multi-protocol hard-coded adapters, and lightweight adaptations based on enterprise service buses for IoT device access and data interaction. Traditional IoT device access methods typically involve writing corresponding hard-coded adapters for each protocol to enable access and data parsing for heterogeneous devices.

[0003] However, in traditional IoT device access methods, the protocol adaptation code is tightly coupled with the platform's core business logic. When a new protocol or new device needs to be accessed, the platform's core code needs to be modified and redeployed. This results in a long development cycle, high maintenance costs, and changes to the protocol parsing logic can easily affect the stable operation of core services. The system has poor scalability and cannot meet the needs of rapid access and flexible expansion of IoT devices. Summary of the Invention

[0004] Therefore, it is necessary to provide an IoT device access system that supports custom heterogeneous network protocol components to address the aforementioned technical issues.

[0005] An IoT device access system supporting custom heterogeneous network protocol components, the system comprising: The protocol component library includes multiple protocol components. Each protocol component is used to interface with multiple IoT devices that use the corresponding communication protocol to complete device connection establishment and raw data transmission and reception. Each protocol component is configured with a metadata description file. The metadata description file includes data format feature fields, parsing rule reference fields, output strategy reference fields, and resource profile fields. The protocol component management module is used to perform lifecycle management of loading, instantiation, mounting and unloading of protocol components in the extensible protocol component library, and to determine the running resource quota of the corresponding protocol component based on the resource profile field, and to adjust the running resource quota according to the received optimization strategy. The data parsing module is used to automatically match the corresponding parsing rules based on the data format feature fields and parsing rule reference fields in the metadata description file, to parse and process the original data of the device accessed by the protocol component, to obtain standardized data, and to adjust the corresponding data parsing rule chain according to the received optimization strategy. The interface output module is used to determine the data output channel and output parameters based on the output strategy reference field in the metadata description file, output the standardized data through the output channel according to the output parameters, and adjust the corresponding data output channel and output parameters according to the received optimization strategy. The scheduling optimization module is used to continuously collect device connection load data from the protocol component management module, parsing performance data from the data parsing module, and transmission operation data from the interface output module. Based on the collected operation status data, it generates corresponding optimization strategies and feeds the optimization strategies back to the protocol component management module, the data parsing module, and the interface output module, respectively.

[0006] The aforementioned IoT device access system, which supports custom heterogeneous network protocol components, enables plug-and-play use of heterogeneous protocols as standardized components by independently encapsulating protocol components and declaring data format characteristics, parsing rule references, output strategy references, and resource profiles through metadata description files. This requires no modification to the core system code. The protocol component management module manages the entire lifecycle of protocol components, including loading, instantiation, mounting, and unmounting, supporting dynamic expansion and non-stop deployment of new protocols. The metadata description file drives the data parsing module to automatically match parsing rules and drives the interface output module to determine output channels and parameters, achieving automatic inter-layer linkage between protocol access, data parsing, and data output, reducing the manual configuration threshold. The scheduling and optimization module continuously collects device connection load, parsing performance, and transmission operation data and generates optimization strategies to feed back to each module, enabling adaptive dynamic adjustment of operating resource quotas, parsing rule chains, and output channel parameters. This embodiment of the invention enables rapid access of heterogeneous devices and continuous self-optimization of system operation status in scenarios such as industrial IoT where multiple protocol devices coexist, effectively improving device access efficiency and system stability. Attached Figure Description

[0007] Figure 1 This is a block diagram of an IoT device access system that supports custom heterogeneous network protocol components in one embodiment. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0009] In one embodiment, such as Figure 1 As shown, an IoT device access system supporting custom heterogeneous network protocol components is provided, including: The protocol component library includes multiple protocol components. Each protocol component is used to connect to multiple IoT devices that use the corresponding communication protocol to complete the device connection establishment and raw data transmission and reception. Each protocol component is configured with a metadata description file. The metadata description file includes data format feature fields, parsing rule reference fields, output strategy reference fields, and resource profile fields. The protocol component management module is used to perform lifecycle management of loading, instantiation, mounting and unloading of protocol components in the extensible protocol component library, determine the running resource quota of the corresponding protocol component based on the resource profile field, and adjust the running resource quota according to the received optimization strategy. The data parsing module is used to automatically match the corresponding parsing rules based on the data format feature fields and parsing rule reference fields in the metadata description file, to parse and process the raw data of the device accessed by the protocol component, to obtain standardized data, and to adjust the corresponding data parsing rule chain according to the received optimization strategy. The interface output module is used to determine the data output channel and output parameters based on the output strategy reference field in the metadata description file, output standardized data through the output channel according to the output parameters, and adjust the corresponding data output channel and output parameters according to the received optimization strategy. The scheduling optimization module is used to continuously collect device connection load data from the protocol component management module, parsing performance data from the data parsing module, and transmission operation data from the interface output module. Based on the collected operation status data, it generates corresponding optimization strategies and feeds the optimization strategies back to the protocol component management module, data parsing module, and interface output module, respectively.

[0010] In the aforementioned IoT device access system that supports custom heterogeneous network protocol components, the independent encapsulation of protocol components and the declaration of data format characteristics, parsing rule references, output strategy references, and resource profiles in metadata description files enable heterogeneous protocols to be used plug-and-play in the form of standardized components without modifying the core system code. The protocol component management module performs full lifecycle management of protocol components, including loading, instantiation, mounting, and unmounting, supporting dynamic expansion and non-stop deployment of new protocols. The metadata description file drives the data parsing module to automatically match parsing rules and drives the interface output module to determine output channels and parameters, enabling automatic inter-layer linkage between protocol access, data parsing, and data output, reducing the manual configuration threshold. The scheduling and optimization module continuously collects device connection load, parsing performance, and transmission operation data and generates optimization strategies to feed back to each module, enabling adaptive dynamic adjustment of operating resource quotas, parsing rule chains, and output channel parameters. This embodiment of the invention enables rapid access of heterogeneous devices and continuous self-optimization of system operation status in scenarios such as industrial IoT where multiple protocol devices coexist, effectively improving device access efficiency and system operation stability.

[0011] In one embodiment, the system further includes a visual collaborative management and control platform. This platform has a built-in scenario-based template library containing multiple scenario templates. Each scenario template includes protocol components, parsing rule chains, and output strategies adapted to the corresponding scenario. The visual collaborative management and control platform is used to complete protocol component configuration, parsing rule orchestration, output method selection, and global operational status monitoring within a unified operation interface. The configuration interface of the visual collaborative management and control platform includes a protocol component definition area, a parsing rule orchestration area, and an output method configuration area. These three areas are linked through fields in the metadata description files of the protocol components. Any configuration change in any area automatically updates the corresponding fields in the metadata description files and triggers parameter updates and preview refreshes in the associated areas. The visual collaborative management and control platform displays the real-time connection count, message throughput, parsing success rate, and resource utilization of each protocol component in the form of a dashboard. The dashboard includes filtering controls based on protocol type, device type, and business scenario.

[0012] In this embodiment, the visual collaborative management and control platform enables non-professionals to complete the entire device access process within a single interface, avoiding access failures caused by inconsistent configurations across multiple regions, quickly locating operational anomalies under specific protocols or scenarios, and directly reusing existing scenario configurations to complete the access of new devices. This achieves low-code management of the entire process from device access configuration to operation monitoring, significantly reducing the system's usage threshold and maintenance costs, and improving device access efficiency.

[0013] In one embodiment, a protocol component template or a blank template is selected from a scenario-based template library. The protocol component template is a component template pre-configured with communication parameters and data frame structure, while the blank template is a general template without configured parameters. Communication parameters are filled in using a visual form provided by the visual collaborative management platform. Communication parameters include protocol type, communication address, port, connection method, and timeout. The structure of each field of the data frame transmitted when the protocol component communicates with the IoT device is defined using drag-and-drop operation provided by the visual collaborative management platform, as well as the byte offset and data type of each field. The data frame is a structured data unit organized according to the corresponding communication protocol, and the byte offset is the byte offset of the corresponding field relative to the start position of the data frame, used to locate and extract the data of that field. Based on the above configuration, the code skeleton, metadata description file, and configuration interface of the protocol component are automatically generated and encapsulated into a protocol component that conforms to the predetermined interface specification.

[0014] In this embodiment, when a user drags a field component into the data frame editing area in the drag-and-drop interface, the system automatically calculates the byte offset of the current field based on the byte lengths of all previously configured fields. If the total byte length of the preceding fields is N bytes, then the starting byte offset of the current field is N. If the user needs to customize the offset, they can also manually specify it through a form input method. The drag-and-drop interface presents the data frame structure with a visual byte layout diagram. According to the order of the fields and their respective byte lengths, the byte area occupied by each field in the data frame is displayed sequentially in rectangular blocks of different colors. Each field block is labeled with the field name, the starting position of the byte offset, the byte length, and the data type, and the byte boundaries are marked with a ruler, allowing the user to intuitively view the complete byte layout of the data frame.

[0015] When a user selects a data type for a field, the system automatically determines the byte length of the field based on the preset mapping relationship between data types and byte lengths: 1 byte for an unsigned 8-bit integer, 2 bytes for a signed 16-bit integer, 4 bytes for a 32-bit floating-point number, 8 bytes for a 64-bit floating-point number, and the length of a byte array is specified by the user or automatically calculated based on the field description. The system automatically updates the field block size and byte offset label in the byte layout diagram based on the byte length of each field.

[0016] The system incorporates a template engine-based code generator, which provides pre-defined code templates for different communication protocol types (TCP Server, TCPClient, UDP, MQTT, CoAP, etc.). Through variable substitution, user-filled communication parameters (protocol type, communication address, port, connection method, timeout, etc.) and data frame structure definitions are injected into template placeholders, generating a complete protocol component source code skeleton. The generated code skeleton includes device connection establishment and disconnection logic, data frame transmission and reception processing logic, connection keep-alive and timeout reconnection mechanisms, and a basic exception handling and logging framework. User-configured communication parameters are mapped to initialization parameters in the code through corresponding fields in the metadata description file, and the field definitions of the data frame structure are mapped to the specific implementations of the data frame serialization and deserialization methods in the code, ensuring that the generated code can correctly parse and encapsulate data according to the user-defined data frame structure.

[0017] The generated code skeleton, metadata description file, and configuration interface are associated and encapsulated through a unified component encapsulator. The code skeleton is compiled or packaged into protocol component modules that conform to the predetermined interface specifications. The metadata description file is embedded in the component package as the component's configuration file. The configuration interface is registered to the component management panel of the visual collaborative management platform, making it convenient for users to view and adjust component parameters at runtime.

[0018] In one embodiment, each protocol component is an independently encapsulated software module that conforms to a predetermined interface specification. Custom protocol components are added to the protocol component library through the loading operation of the protocol component management module and removed from the protocol component library through the unloading operation of the protocol component management module.

[0019] In this embodiment, by designing the protocol component as an independently encapsulated software module that conforms to a predetermined interface specification, the protocol component can be decoupled from the core business logic of the system, and changes to the protocol component will not affect the stable operation of the core services of the system. By adding custom protocol components to the protocol component library through the loading operation of the protocol component management module and removing them from the protocol component library through the uninstallation operation, dynamic expansion and removal of protocol components can be realized. New protocol access can be supported without modifying the core code of the system, thereby improving the flexibility and scalability of the system.

[0020] In one embodiment, the protocol component management module uses a hot loading mechanism when loading protocol components, deploying new protocol components to the running system by releasing upgrade packages, and the loading process does not require system downtime.

[0021] In this embodiment, the protocol component management module uses a hot loading mechanism when loading the protocol component. The new protocol component is deployed to the running system by releasing an upgrade package. This ensures that the deployment process of the new protocol component does not interrupt the normal operation of the system, avoiding the service interruption caused by the need to shut down the system to update when accessing a new protocol in the traditional solution, and improving the continuous service capability and availability of the system.

[0022] In one embodiment, the data format feature field is used to declare the encoding method, frame structure, and byte order of the data processed by the corresponding protocol component; the parsing rule reference field is used to declare the parsing rule identifier associated with the default protocol component; the output strategy reference field is used to declare the default output channel and default output parameters of the data of the corresponding protocol component; and the resource profile field is used to declare the maximum number of connections, concurrency capability, and resource consumption weight of the corresponding protocol component.

[0023] In this embodiment, the data format feature field declares the encoding method, frame structure, and byte order of the data processed by the protocol component, providing prior information on the data format to the parsing engine and supporting automatic matching of parsing rules. The parsing rule reference field declares the default associated parsing rule identifier, enabling the protocol component to automatically associate with the corresponding parsing rule after loading, eliminating the need for manual binding. The output strategy reference field declares the default output channel and default output parameters, enabling the protocol component to automatically determine the data output method after loading. The resource profile field declares the maximum number of connections, concurrency capability, and resource usage weight, providing a basis for determining and dynamically adjusting the protocol component's operating resource quota, thus achieving reasonable resource allocation.

[0024] In one embodiment, the metadata description file further includes a scenario adaptation field; the scenario adaptation field is used to declare the business scenario tags applicable to the corresponding protocol component, so that the scenario-based template library can retrieve and match the corresponding scenario template based on the business scenario tags.

[0025] In this embodiment, by adding a scenario adaptation field to the metadata description file to declare the business scenario tags applicable to the protocol components, the protocol components can be classified and managed according to business scenarios. By using a scenario-based template library to retrieve and match the corresponding scenario templates based on the business scenario tags, users can quickly locate the protocol components and configuration schemes that are adapted to the business scenario when accessing a new device, reducing the workload of scenario screening and configuration trial and error, and improving the configuration efficiency of device access in specific scenarios.

[0026] In one embodiment, the parsing rules in the data parsing module exist in the form of a rule chain, which is composed of multiple parsing nodes connected together. The parsing nodes include field extraction nodes, data type conversion nodes, verification nodes, filtering nodes, and mapping nodes. The rule chain is arranged in a visual drag-and-drop manner through a visual collaborative management platform, and takes effect in real time after arrangement. When the rule chain parses the original data and generates new fields, the data parsing module synchronizes the name, data type, and semantic tags of the new fields to the interface output module. The interface output module includes the new fields in the output range of standardized data according to the name, data type, and semantic tags of the new fields, and determines the corresponding output channel and serialization format.

[0027] In this embodiment, complex data parsing logic is flexibly constructed using modular node combinations, allowing users to configure and adjust parsing rules without writing parsing code. At the same time, it enables automatic linkage between the parsing layer and the output layer, and can adapt the output of newly added fields without manual configuration, thereby improving the system's adaptability to changes in data structure and the response efficiency to changes in parsing rules.

[0028] In one embodiment, the data parsing module supports parsing and processing binary data, text data, structured data, and unstructured data. It has built-in byte offset extraction operators, regular expression matching operators, field mapping operators, numerical calculation operators, and data enrichment operators. Data cleaning, transformation, and enrichment are achieved through the combination of operators.

[0029] In this embodiment, the data parsing module supports the parsing and processing of binary data, text data, structured data, and unstructured data, covering the parsing needs of various heterogeneous data formats in IoT devices. Through built-in byte offset extraction operators, regular expression matching operators, field mapping operators, numerical calculation operators, and data enrichment operators, and by combining operators, data cleaning, transformation, and enrichment are achieved. This allows for flexible adaptation to the complex data processing needs of devices with different protocols, improving the system's ability to process heterogeneous data.

[0030] In one embodiment, the optimization strategy generated by the scheduling optimization module includes a resource scheduling strategy, a rule optimization strategy, and an output adaptation strategy. The resource scheduling strategy is generated based on the comparison result of device connection load data and a preset load threshold, and is used to adjust the operating resource quota of the protocol component. The rule optimization strategy is generated based on the comparison result of parsing performance data and a preset parsing threshold, and is used to adjust the parsing rule chain. The output adaptation strategy is generated based on the comparison result of transmission operation data and a preset transmission threshold, and is used to adjust the output channel and output parameters.

[0031] In this embodiment, by dividing the optimization strategy into three categories—resource scheduling strategy, rule optimization strategy, and output adaptation strategy—and generating them based on the comparison results of device connection load data with preset load thresholds, parsing performance data with preset parsing thresholds, and transmission operation data with preset transmission thresholds, the system can adaptively optimize the three dimensions of protocol component load, parsing rule efficiency, and output channel status. By adjusting the running resource quota, parsing rule chain, and output channel and output parameters through the three types of strategies, dynamic optimization of the entire system link can be achieved, maintaining stable and efficient operation of the system under scenarios such as fluctuations in device access volume, changes in parsing efficiency, and changes in output load.

[0032] In one embodiment, the scheduling optimization module is further configured to divide the runtime load of the protocol component into multiple discrete states, statistically analyze the transition frequency between each discrete state based on historical runtime data, and construct a state transition probability matrix based on the transition frequency; predict the probability distribution of the protocol component entering each load state in future time steps based on the current load state and the state transition probability matrix; when the probability of the protocol component entering an overload state in the future exceeds a preset threshold, the scheduling optimization module creates a protocol component instance in advance, adjusts the thread pool size, or increases the runtime resource quota.

[0033] In this embodiment, the continuously changing load is transformed into a modelable state transition problem. By predicting the load change trend and pre-allocating resources before the actual overload occurs, the limitations of traditional passive response scheduling can be overcome. This upgrades resource scheduling from passive response to proactive prediction, improving system stability and resource utilization efficiency in scenarios such as batch access of industrial IoT devices and periodic fluctuations in access volume.

[0034] In one specific embodiment, such as Figure 1 As shown, the protocol component layer enables dynamic access to heterogeneous protocols, the parsing engine layer performs visual parsing of network data, the unified interface layer provides standardized output of data from multiple scenarios, and the intelligent scheduling and optimization module collects real-time operational status data from the first three layers. After global analysis, it feeds optimization strategies back to each layer, driving adaptive tuning and thus achieving mutual empowerment and global optimization among the modules. The specific components of the system are described below: 1. Protocol Component Layer Dynamic Protocol Component Library: The system maintains an extensible protocol component library. Each protocol component is an independent software module (such as a JAR file) that conforms to a predefined interface specification.

[0035] Component description file: Each protocol component is accompanied by a metadata description file (such as XML or JSON format), which declares the component's name, version, supported protocol types (such as "TCP_Server", "UDP_Client", "MQTT_Client"), required configuration parameters, and other information.

[0036] To support closed-loop collaboration and intelligent scheduling, the metadata description file of this invention further expands upon the original basic fields such as name, version, protocol type, and configuration parameters by the following fields: (1) Data format feature field (dataFormatFeature): declares the encoding method, frame structure, byte order, field separator and other format features of the data processed by this protocol component, so that the parsing engine can automatically match the corresponding parsing rules; (2) AdaptationScenario: Declares the business scenario tags (such as industrial automation, energy monitoring, etc.) applicable to this component, for retrieval and matching by the scenario template library; (3) Parsing rule reference field (parseRuleRef): declares the default parsing rule chain associated with this component, realizing inter-layer linkage between components and parsing rules; (4) Output Strategy Reference Field (outputStrategyRef): Declares the default output method and adaptation rules for the component's data, enabling the linkage between the component and the output layer; (5) Resource Profile field: declares the resource usage characteristics of the component (such as maximum number of connections, concurrency capability, CPU weight, etc.) for the intelligent scheduling and optimization module to perform load scheduling.

[0037] Here is an example of a metadata description file (JSON format): { "componentName": "ModbusTCP_Adapter", "version": "1.2.0", "protocolType": "TCP_Client", "dataFormatFeature": { "encoding": "binary", "frameStructure": "header(4B)+payload+CRC(2B)", "byteOrder": "big-endian" }, "adaptationScenario": ["Industrial Automation", "Energy Monitoring"], "parseRuleRef": "modbus_register_mapping_v1", "outputStrategyRef": ["mqtt_publish", "mysql_write"], "resourceProfile": {"maxConnections": 1000, "cpuWeight": 2} } Component lifecycle management: The system provides full lifecycle management for dynamic protocol components, including loading, instantiation, mounting, and unmounting. Administrators can support new protocol components by releasing upgrade packages.

[0038] 2. Analyze the engine layer Configurable Parsing Rules: Each protocol component generates raw data after receiving device data, which needs to be parsed. The system provides a powerful parsing engine with black-box encapsulated parsing rules for easy customer use. The parsing engine offers a visual rule chain orchestration interface, allowing users to drag and drop parsing nodes (such as field extraction, data type conversion, validation, filtering, mapping, etc.) to form a parsing rule chain and preview the parsing effect in real time, without needing to write parsing code. The parsing engine supports processing various payload formats, including binary, text, structured, and unstructured data. It has built-in rich data processing operators such as field extraction, data type conversion, numerical calculation, string processing, conditional filtering, multi-source data merging, and field enrichment, enabling complex data cleaning, transformation, and enrichment through rule chain combinations.

[0039] A linkage and binding relationship is established between parsing rules, protocol components, and output strategies. When the metadata description file of a protocol component declares data format characteristics, the parsing engine automatically matches and recommends corresponding parsing rules; when parsing rules generate new fields or data views, the adaptation rules of the output layer can be automatically linked to achieve seamless inter-layer collaboration of "component access → data parsing → interface output".

[0040] Specifically, the parsing engine layer maintains a metadata field registry. When the parsing rules parse the original data and generate new fields or data views, the system automatically updates the field definition information in the registry, including metadata such as the name, data type, byte offset, and semantic tags of the new fields.

[0041] The output layer detects changes in parsing results by monitoring changes in metadata field registry entries or through a periodic metadata synchronization mechanism. When a new field is detected, the output layer's adaptation rules automatically incorporate it into the field mapping table of the standardized data model. Based on the field's data type and semantic tags, it automatically matches the default output channel and serialization format, achieving automatic mapping and expansion from parsing results to output data without manual intervention to adjust output configurations. After the linkage adjustment is completed, the system displays the newly added output fields and their automatic adaptation results with highlighted markers on the output configuration interface of the visual collaborative management platform for user review and confirmation. Users can choose to accept the automatic adaptation results or manually adjust the output mapping relationship. If the user does not operate within the specified time, the automatic adaptation rules take effect by default.

[0042] 3. Unified Interface Layer Data processed by the parsing engine is transformed into a unified, standardized data model (usually in JSON format) within the system. The unified interface layer provides data to upper-layer applications in two main ways: Message middleware: Publishes standardized data to message queues (such as MQTT topics) for other microservice systems to subscribe to and consume.

[0043] REST API Interface: This interface exposes standardized data via HTTP, allowing upper-layer applications to query and retrieve parsed device data on demand. The system automatically generates a standardized data query interface for each connected device. The interface path includes device identifier and data category parameters. Upper-layer applications can use GET requests to query the latest device data or historical data within a specified time range, and POST requests to write control commands or configuration parameters, which are then forwarded by the unified interface layer to the corresponding protocol components for distribution. The REST API interface provides both JSON and XML data output formats, supports paginated queries, field filtering, and time range filtering, meeting the on-demand device data retrieval needs of external systems in different scenarios.

[0044] 4. Intelligent Scheduling and Optimization Module The intelligent scheduling and optimization module horizontally connects the protocol component layer, parsing engine layer, and unified interface layer, driving global adaptive optimization by collecting real-time operational data from each layer. This module mainly includes the following four sub-modules: (1) Intelligent load scheduling of protocol components: Real-time monitoring of operational metrics such as device access volume, number of connections, message throughput, and CPU / memory usage of each protocol component, combined with the resource profile declared in the component description file, dynamically adjusts the concurrency of component instances, thread pool size, and resource quota; when the device access volume of a certain protocol suddenly increases, the component instance is automatically horizontally scaled up, and when the access volume decreases, it is automatically scaled down, thereby reducing resource waste while ensuring access performance. Specifically, the system continuously collects real-time operational metrics such as the number of device connections, message throughput, and CPU / memory usage of each protocol component during runtime, and compares and evaluates the current operational metrics with the threshold values ​​of the resource profile fields declared in the component metadata description file; When the number of device connections for a protocol component approaches the maximum connection threshold declared in the resource profile, the system automatically creates a new instance for that component to achieve horizontal scaling. The new instance inherits the metadata description file configuration of the same protocol component, and newly connected device connections are allocated to the new instance through a load balancing strategy. When the number of device connections falls below the threshold, the system automatically reclaims excess instances to release resources. The thread pool size is dynamically adjusted based on the current message throughput and average message processing latency. When throughput increases and processing latency exceeds a preset baseline, the thread pool size is increased to improve concurrent processing capabilities. Resource quota adjustments are calculated based on the ratio between the actual CPU and memory utilization and the CPU weight declared in the resource profile. This determines the updated resource limit parameters for each component instance at the container or virtual machine level, and updates are issued through the container orchestration interface or the operating system-level resource control group (cgroup) interface. The above dynamic adjustment process is uniformly decided by the resource scheduling strategy submodule in the scheduling optimization module. The adjustment results are recorded in real-time in the system operation log and displayed in the monitoring panel of the visual collaborative management platform.

[0045] (2) Intelligent optimization of parsing rules: Continuously count the execution time, parsing success rate, and anomaly rate of each parsing rule chain to identify rule nodes with low parsing efficiency or frequent errors; automatically merge, trim and rearrange the rule chains according to data format characteristics and parsing logs to generate an optimized parsing rule chain version, and hot update it to the parsing engine after verification to achieve adaptive improvement of parsing performance.

[0046] Specifically, the system continuously collects execution logs for each parsing rule chain, including metrics such as execution time, input / output data volume, parsing success rate, and exception type for each parsing node. Based on information such as encoding method, frame structure, and byte order declared in the data format feature field, combined with execution statistical analysis in the parsing logs, the system automatically identifies optimizable node combinations in the rule chain: when two adjacent consecutive parsing nodes (such as a field extraction node and a data type conversion node) always appear in a fixed order in historical execution and no intermediate results are referenced by other nodes, the system automatically merges them into a combined node to reduce data transmission overhead between nodes; when a parsing node in the rule chain outputs empty results or results consistent with the input in multiple consecutive executions, it is identified as a redundant node and automatically pruned; when the execution time statistics of each parsing node in the rule chain show that some parsing nodes with high execution time can be rearranged to filter invalid data in advance, the system automatically rearranges and optimizes the node execution order. The optimized parsing rule chain version is first verified in a sandbox environment using historical data playback. After verification, it is hot-updated and deployed to the parsing engine to replace the original rule chain and take effect in real time. The entire optimization and update process does not affect the parsing tasks that are currently being executed.

[0047] (3) Dynamic adaptation of interface output: Monitor the subscription status, consumption rate and backlog of downstream consumers (message middleware, database, business system, etc.), and dynamically adjust the data output routing, batch size, compression strategy and retry mechanism; when the downstream consumption capacity is insufficient, automatically enable the degradation or buffering strategy to avoid message loss or delay caused by data backlog.

[0048] (4) Global status monitoring and alarm: Unified status collection and visual monitoring of protocol components, parsing rules and data output across the entire chain. When abnormalities such as component abnormality, parsing failure rate exceeding the threshold, and output backlog are detected, alarms are triggered in a timely manner, and alarm information and optimization suggestions are pushed to the visual collaborative management and control platform.

[0049] (5) Markov Chain-Based Protocol Component Load State Prediction and Proactive Scheduling: Based on intelligent load scheduling of protocol components, a Markov chain is introduced to perform probabilistic modeling and state transition prediction of the operating load state of protocol components. The operating load of each protocol component is divided into multiple discrete states, including idle state, normal state, busy state, and overload state. The transition frequency between each state is statistically analyzed based on historical operating data, and a state transition probability matrix is ​​constructed through maximum likelihood estimation. During system operation, the scheduling optimization module predicts the probability distribution of the component entering each load state at each time step in the future based on the current load state of the protocol component and using the one-step and multi-step transition probability formulas of the Markov chain. When the probability of a protocol component entering the overload state at a future time exceeds a preset threshold, the system proactively triggers resource pre-allocation before the actual overload of the component occurs, preventing overload by creating new component instances in advance, adjusting the thread pool size, or increasing resource quotas. Meanwhile, by using the multi-step transition probability of Markov chains to extrapolate the medium- and long-term load trends of protocol components, a forward-looking decision-making basis is provided for resource quota planning and dynamic adjustment of the number of component instances. This upgrades resource scheduling from a passive response to an active prediction, further improving the system's operational stability and resource utilization efficiency in scenarios with fluctuating device access volumes.

[0050] The optimization decisions of the four sub-modules mentioned above are not executed independently. Instead, they follow a closed loop of "data collection → intelligent analysis → strategy generation → strategy distribution → effect feedback → further optimization," feeding the optimization results back to the protocol component layer, parsing engine layer, and unified interface layer. This allows each layer to continuously adapt and optimize during operation, forming a complete closed loop of "access-parsing-output-optimization." For example, when the parsing engine layer reports frequent parsing failures for a certain type of private protocol data, the intelligent scheduling and optimization module generates parsing rule correction suggestions, sends them back to the visual collaborative management platform, prompts the user to adjust the parsing rules, and hot-updates the corrected rules to the parsing engine, thus forming a closed-loop optimization with inter-layer linkage.

[0051] 5. Visualized Collaborative Management and Control Platform The visual collaborative management platform provides users with a unified low-code operation entry point, enabling non-professionals to complete the entire process of "component configuration → parsing rule editing → output method selection → global status monitoring" within a single interface, and visually presenting the collaborative effects between different layers. The platform mainly includes the following capabilities: (1) Integrated visual configuration interface: The definition of protocol components, the arrangement of parsing rules, and the configuration of output methods are integrated into the same visual interface. Users can complete the configuration in a low-code way by dragging and dropping, filling in forms, and selecting from drop-down menus, without having to write underlying code.

[0052] In the protocol component definition area on the left side of the visual configuration interface, users can select or create protocol components by dragging and dropping. After configuring communication parameters and data frame structures, the interface automatically displays a preview of the metadata description file associated with the protocol component. In the parsing rule orchestration area in the middle of the interface, the system automatically recommends matching parsing rule templates based on the data format characteristics of the configured protocol components. Users can further customize the rule chain by dragging and dropping, and the interface displays a real-time preview of the parsing results during the orchestration process. In the output method configuration area on the right side of the interface, the system automatically recommends output channels and parameters based on the output strategy reference fields declared by the current protocol component. Users can adjust these through drop-down selection or form filling. The data linkage relationship between the above three configuration areas is maintained by the built-in linkage data model of the interface. Changes in the configuration of any area automatically trigger parameter updates and preview refreshes in the associated areas, ensuring that users complete the entire component access configuration process within the same interface. After configuration, the system automatically generates a complete protocol component package and deploys it to the runtime environment. The deployment progress is displayed in real-time in the status bar at the bottom of the interface.

[0053] (2) Visual presentation of inter-layer collaboration: The complete data flow and collaborative relationship of “component access → data parsing → interface output → optimization feedback” are visualized by means of data flow diagrams, closed-loop link diagrams, etc., as well as the running status, performance indicators and before and after optimization of each layer, so that users can intuitively understand the mutual empowerment effect of each module.

[0054] (3) Scenario-based component and parsing rule template library: The platform has built-in component and parsing rule templates for typical scenarios such as industrial automation, energy monitoring, smart buildings, and vehicle networking. Each scenario template contains a combination configuration of protocol components, parsing rule chains, and data output strategies adapted to the scenario. When users connect new devices, they only need to select a template with a similar scenario from the template library and make a few parameter adjustments to it to quickly complete the connection, which greatly reduces the access threshold.

[0055] The system also includes the function of unified monitoring and management of the entire life cycle operation status of the protocol components through the aforementioned visual collaborative management and control platform. The platform centrally displays key operation indicators such as the real-time connection number, message throughput, parsing success rate and resource utilization of each protocol component in the form of a dashboard. It supports data filtering and comparative analysis by multiple dimensions such as protocol type, device type and business scenario, and automatically triggers alarms and pushes optimization suggestions to the platform's operation panel when abnormal indicators are detected, enabling users to complete the entire process from device access configuration to operation monitoring and management within a unified interface.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An IoT device access system supporting custom heterogeneous network protocol components, characterized in that, The system includes: The protocol component library includes multiple protocol components. Each protocol component is used to interface with multiple IoT devices that use the corresponding communication protocol to complete device connection establishment and raw data transmission and reception. Each protocol component is configured with a metadata description file. The metadata description file includes data format feature fields, parsing rule reference fields, output strategy reference fields, and resource profile fields. The protocol component management module is used to perform lifecycle management of loading, instantiation, mounting and unloading of protocol components in the extensible protocol component library, and to determine the running resource quota of the corresponding protocol component based on the resource profile field, and to adjust the running resource quota according to the received optimization strategy. The data parsing module is used to automatically match the corresponding parsing rules based on the data format feature fields and parsing rule reference fields in the metadata description file, to parse and process the original data of the device accessed by the protocol component, to obtain standardized data, and to adjust the corresponding data parsing rule chain according to the received optimization strategy. The interface output module is used to determine the data output channel and output parameters based on the output strategy reference field in the metadata description file, output the standardized data through the output channel according to the output parameters, and adjust the corresponding data output channel and output parameters according to the received optimization strategy. The scheduling optimization module is used to continuously collect device connection load data from the protocol component management module, parsing performance data from the data parsing module, and transmission operation data from the interface output module. Based on the collected operation status data, it generates corresponding optimization strategies and feeds the optimization strategies back to the protocol component management module, the data parsing module, and the interface output module, respectively.

2. The system according to claim 1, characterized in that, The system also includes a visual collaborative management and control platform, which has a built-in scenario-based template library. The scenario-based template library includes multiple scenario templates, and each scenario template contains protocol components, parsing rule chains, and output strategies adapted to the corresponding scenario. The visual collaborative management and control platform is used to complete protocol component configuration, parsing rule arrangement, output method selection, and global operation status monitoring in a unified operation interface; The configuration interface of the visual collaborative management and control platform includes a protocol component definition area, a parsing rule arrangement area, and an output method configuration area. The various areas are linked through fields in the metadata description file of the protocol component. When the configuration of any area changes, the corresponding field in the metadata description file is automatically updated, triggering parameter updates and preview refreshes in the associated areas. The visual collaborative management and control platform displays the real-time connection count, message throughput, parsing success rate, and resource utilization of each protocol component in the form of a dashboard. The dashboard includes filtering controls based on protocol type, device type, and business scenario.

3. The system according to claim 2, characterized in that, The protocol component configuration includes: Select a protocol component template or a blank template from the scenario-based template library. The protocol component template is a component template with pre-configured communication parameters and data frame structure, and the blank template is a general template without configured parameters. The communication parameters are filled in using the visual form provided by the visual collaborative management platform; the communication parameters include protocol type, communication address, port, connection method, and timeout. The structure of each field of the data frame transmitted when the protocol component communicates with the IoT device is defined by drag-and-drop operation provided by the visualization collaborative management platform, as well as the byte offset and data type of each field. The data frame is a structured data unit organized according to the corresponding communication protocol, and the byte offset is the byte offset of the corresponding field relative to the start position of the data frame, which is used to locate and extract the data of the field. Based on the above configuration, the code skeleton, metadata description file, and configuration interface of the protocol component are automatically generated and packaged into a protocol component that conforms to the predetermined interface specification.

4. The system according to claim 1, characterized in that, The protocol component management module uses a hot loading mechanism when loading protocol components. It deploys new protocol components to the running system by releasing upgrade packages, and the loading process does not require system downtime.

5. The system according to claim 1, characterized in that, The data format feature field is used to declare the encoding method, frame structure, and byte order of the data processed by the corresponding protocol component; The parsing rule reference field is used to declare the parsing rule identifier that is associated with the corresponding protocol component by default; The output strategy reference field is used to declare the default output channel and default output parameters of the corresponding protocol component data; The resource profile field is used to declare the maximum number of connections, concurrency capability, and resource consumption weight of the corresponding protocol component.

6. The system according to claim 2, characterized in that, The metadata description file also includes a scenario adaptation field; the scenario adaptation field is used to declare the business scenario tags applicable to the corresponding protocol components, so that the scenario-based template library can retrieve and match the corresponding scenario templates based on the business scenario tags.

7. The system according to claim 2, characterized in that, The parsing rules in the data parsing module exist in the form of a rule chain, which is composed of multiple parsing nodes. The parsing nodes include field extraction nodes, data type conversion nodes, validation nodes, filtering nodes, and mapping nodes. The rule chain is arranged in a visual drag-and-drop manner through the visual collaborative management and control platform, and takes effect in real time after arrangement. When the rule chain parses the original data to generate new fields, the data parsing module synchronizes the name, data type, and semantic tag of the new fields to the interface output module. The interface output module includes the new field in the output range of standardized data based on the name, data type and semantic tag of the new field, and determines the corresponding output channel and serialization format.

8. The system according to claim 1, characterized in that, The data parsing module supports parsing and processing binary data, text data, structured data, and unstructured data. It has built-in byte offset extraction operators, regular expression matching operators, field mapping operators, numerical calculation operators, and data enrichment operators. Data cleaning, transformation, and enrichment can be achieved through the combination of these operators.

9. The system according to claim 1, characterized in that, The optimization strategies generated by the scheduling optimization module include resource scheduling strategies, rule optimization strategies, and output adaptation strategies. The resource scheduling strategy is generated based on the comparison result between the device connection load data and the preset load threshold, and is used to adjust the operating resource quota of the protocol components. The rule optimization strategy is generated based on the comparison results between the parsing performance data and the preset parsing threshold, and is used to adjust the parsing rule chain; The output adaptation strategy is generated based on the comparison result between the transmission operation data and the preset transmission threshold, and is used to adjust the output channel and output parameters.

10. The system according to claim 1, characterized in that, The scheduling optimization module is also used to divide the operating load of the protocol component into multiple discrete states, count the transition frequency between each discrete state based on historical operating data, and construct a state transition probability matrix based on the transition frequency. Based on the current load state and the state transition probability matrix, predict the probability distribution of the protocol component entering each load state in future time steps; When the probability of a protocol component entering an overload state in the future is predicted to exceed a preset threshold, the scheduling optimization module creates a protocol component instance in advance, adjusts the thread pool size, or increases the runtime resource quota.

Images