IoT Terminal Reporting Test Method and System Based on Multi-Protocol Unified Simulation Engine
By adopting an IoT terminal reporting test method based on a multi-protocol unified simulation engine, the problem of insufficient multi-protocol unified simulation and automated assertion capabilities in existing technologies is solved. This method enables efficient and visualized multi-protocol testing and report generation, improving test coverage and reproducibility.
Patent Information
- Application Number
- CN202610313668.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-30
AI Technical Summary
Existing IoT platforms lack the capabilities for unified simulation of multiple protocols, visual configuration, flexible scenario orchestration, and automated assertions, resulting in low test coverage, high orchestration thresholds, and difficulty in achieving unified management and efficient testing of multiple protocols.
The method for testing IoT terminal reporting based on a unified simulation engine with multiple protocols is adopted. The unified engine loads the object model definition file, parses attributes, events and services, and sets test scenario parameters in combination with a visual configuration interface. It generates virtual devices and performs protocol selection and exception injection, supports concurrent testing of multiple protocols, and has built-in automatic assertion and multi-format report generation.
It achieves unified management of multiple protocols, lowers the threshold for test orchestration, improves test coverage and reproducibility, supports the generation of reports in multiple formats, ensures that the behavior of virtual devices is consistent with that of real devices, and has full-link interaction capabilities and automated verification functions.
Smart Images

Figure CN122316919A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of Internet of Things (IoT) simulation testing technology, and in particular to an IoT terminal reporting test method and system based on a multi-protocol unified simulation engine. Background Technology
[0002] Currently, various IoT platforms and large-scale distributed IoT systems generally deploy a core module for device access and data reporting, used to uniformly manage the connection, authentication, data transmission, and protocol adaptation of terminal devices. This module typically possesses a variety of basic functions, including device authentication, secure encrypted transmission, data routing and forwarding, traffic control, anomaly detection, and log auditing, and is a fundamental component for the stable operation of an IoT platform.
[0003] In existing technologies, commonly used implementations in the industry rely heavily on mature open-source protocol stacks or gateway frameworks. These include access gateways based on single protocols such as MQTT, CoAP, HTTP, WebSocket, and TCP, or multi-protocol access platforms encapsulated on top of these protocols. These frameworks typically provide functions such as device connection management, session persistence, message publishing and subscription, topic routing, QoS (Quality of Service) control, message caching and retransmission, and message distribution and storage interfaces with the platform backend.
[0004] To adapt to different types of terminal devices and business scenarios, existing systems often introduce multi-protocol support at the access layer. However, this is mostly implemented as an independent module or through gateway plugins, lacking a unified scheduling and management mechanism between protocols. In the testing phase, the common practice is to use real hardware devices or simple single-protocol simulation tools for functional verification and performance testing. These tools typically only cover a limited number of protocol types and scenarios, and have limited capabilities in areas such as batch device simulation, complex object model-driven operations, abnormal scenario injection, automated assertions, and multi-format report generation. Therefore, existing technologies still have significant room for improvement in areas such as unified multi-protocol simulation, visual configuration, and flexible scenario orchestration. Summary of the Invention
[0005] The purpose of this invention is to provide a method and system for testing IoT terminal reporting based on a multi-protocol unified simulation engine.
[0006] The technical solution adopted in this invention is:
[0007] The IoT terminal reporting test method based on a multi-protocol unified simulation engine includes the following steps:
[0008] The IoT product object model definition file is loaded through a unified simulation engine, and the product object model index of attributes, events and services is parsed and established.
[0009] The configuration parameters for the test scenario are received through a visual configuration interface. These parameters include protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. Protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP.
[0010] Initialize the test scenario according to the configuration parameters, generate a virtual device that conforms to the corresponding product model, and initiate connection authentication with the IoT platform;
[0011] During the test run, simulated reporting data conforming to the corresponding product model is generated based on the product model index, data generation strategy, and event probability model. During the simulated reporting process, network anomalies or data payload anomalies are probabilistically injected according to the event probability model. The data is then reported to the IoT platform through at least one selected protocol.
[0012] Listen for and process service call instructions issued by the IoT platform; the virtual device corresponding to the product object model executes and updates the local device status.
[0013] Collect connection, reporting, response, and anomaly metrics data during the testing process; perform automated assertion verification on the collected metrics data based on predefined assertion rules;
[0014] Generate and export a test report based on the assertion results and indicator data.
[0015] Furthermore, the product object model definition file (JSON / YAML) describes the names, data types, value ranges, required constraints, and units of attributes, events, and services for each type of IoT terminal device. Virtual devices are generated by parsing the product object model file.
[0016] Furthermore, the event probability model is used to define the probability of normal events, abnormal events, and alarm events being triggered within the test time window, respectively.
[0017] Furthermore, the implementation of listening to and processing service call instructions issued by the IoT platform includes: parsing instruction parameters, simulating service execution according to the definition of the product object model file, and reporting the execution results and associated attribute changes to the platform to form a closed-loop interaction.
[0018] Furthermore, the implementation of automated assertion validation based on predefined assertion rules includes:
[0019] Based on the attribute value range and event sequence constraints defined in the product object model file, the first assertion rule is automatically generated; and custom assertion rules defined by the user through scripts or expressions are received and executed.
[0020] Furthermore, the implementation of generating and exporting test reports includes: generating reports containing assertion results, key indicator time series plots, and links to raw data, and supporting export in at least one of the following formats: HTML, Excel, TXT, and PDF.
[0021] Furthermore, the method also includes saving the complete configuration information, random seed, and device interaction trajectory of the test scenario to support one-click reproduction and playback of the test scenario.
[0022] Furthermore, by enabling tenant isolation (namespaces / resource quotas / access control) in the engine, devices, connections, and reports of different tenants are not affected by each other, and parallel load testing is supported.
[0023] Furthermore, the scheduler distributes virtual devices and connection loads across multiple nodes, automatically scaling up and down based on target concurrency and SLA, and supports task migration and breakpoint continuation testing for failed nodes.
[0024] Furthermore, certificates and keys are securely managed and rotated, sensitive data is stored with de-identification, and logs and reports include access auditing to meet organizational compliance and auditing needs.
[0025] An IoT terminal reporting test system based on a multi-protocol unified simulation engine includes:
[0026] The product object model management module is used to load and parse the object model definition files of IoT products and establish an index of product object models for attributes, events and services.
[0027] The visual configuration module provides a configuration interface for receiving configuration parameters for the test scenario. These parameters include: protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. The protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP.
[0028] The protocol adaptation and communication module is used to enable connection and data communication with the IoT platform based on multiple protocols.
[0029] The device simulation and scheduling engine is used to generate virtual device instances based on configuration parameters and product object model indexes, and to schedule virtual devices to perform data reporting, event triggering and service response.
[0030] The anomaly injection module is used to inject network or data payload anomalies probabilistically according to a strategy during the testing process;
[0031] The assertion and reporting engine is used to collect test metrics, perform automated verification based on assertion rules, and generate test reports in multiple formats.
[0032] Furthermore, the system also includes a scenario archiving and reproduction module, which is used to save the configuration, random seed and execution trajectory of the test scenario, and provides a scenario replay interface.
[0033] The present invention adopts the above technical solutions and has the following beneficial effects compared with the prior art: (1) The present invention natively supports multiple mainstream protocols such as MQTT, CoAP, HTTP, WebSocket, and TCP within the same engine, and can flexibly switch or run concurrently in a single scenario, greatly improving the efficiency of cross-protocol testing. (2) The present invention introduces a model-driven mechanism, which can automatically parse the product model file and generate full-link simulation logic of attributes, events, and services to ensure that the behavior of virtual devices is consistent with that of real devices; combined with the visual configuration interface, users can intuitively set parameters such as reporting cycle, event type, probability of anomalies and alarms, and network anomaly injection, reducing the threshold of test orchestration. (3) During the execution process, the system supports probabilistic scheduling of normal, abnormal, and alarm events, and can virtual devices execute and receive services issued by the platform to form a closed-loop interaction; built-in automatic assertion and custom assertion DSL realize multi-dimensional automatic verification of performance, correctness, and sequence. (4) This invention has the ability to generate and archive multi-format reports, and can output reports in HTML, Excel, TXT, PDF and other formats, and save scene configurations and random seeds to achieve traceability of problems and reproducibility of tests. These advantages make this invention superior to the closest prior art in terms of test coverage, flexibility, controllability and maintainability. Attached Figure Description
[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;
[0035] Figure 1 This is a schematic diagram of the test scenario initialization and connection authentication process of the present invention;
[0036] Figure 2 This is a schematic diagram of the data processing flow for the simulated reporting method of the present invention;
[0037] Figure 3 This is a schematic diagram illustrating the process of handling service call commands and exception injection issued by the Internet of Things platform in this invention.
[0038] Figure 4 This is a schematic diagram of the automatic assertion and report generation process of the present invention. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0040] This invention proposes an IoT terminal reporting test method, system, and medium that supports a unified simulation engine for multiple protocols. Within a unified simulation engine, it simultaneously supports multiple protocols such as MQTT, CoAP, HTTP, WebSocket, and TCP. Through a visual interface, it configures reporting cycles, event types, object model definitions, and anomaly and alarm probabilities, automatically driving virtual devices to perform end-to-end interactions, including attribute reporting, event triggering, and service response. The system incorporates automatic assertion and multi-format report generation modules, achieving fully automated verification and result visualization during the testing process. This significantly lowers the testing threshold, improves test coverage and reproducibility, and truly achieves unified multi-protocol operation, programmable scenarios, verifiable results, and traceable reports.
[0041] like Figures 1 to 4 As shown in any of the accompanying drawings, this invention discloses a testing method for IoT terminal reporting based on a multi-protocol unified simulation engine, comprising the following steps:
[0042] The unified simulation engine loads the IoT product model definition file, parses and builds the product model index of attributes, events and services; further, the product model definition file (JSON / YAML) describes the name, data type, value range, required constraints and units of attributes, events and services for each type of IoT terminal device, so as to generate virtual devices by parsing the product model file.
[0043] The configuration parameters for the test scenario are received through a visual configuration interface. These parameters include protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. Protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP.
[0044] Initialize the test scenario according to the configuration parameters, generate a virtual device that conforms to the corresponding product model, and initiate connection authentication with the IoT platform;
[0045] During the test run, simulated reporting data conforming to the corresponding product model is generated based on the product model index, data generation strategy, and event probability model. During the simulated reporting process, network anomalies or data payload anomalies are probabilistically injected according to the event probability model. The data is then reported to the IoT platform through at least one selected protocol.
[0046] The system listens for and processes service call commands issued by the IoT platform. The virtual device corresponding to the product's object model executes the command and updates its local device status. Specifically, when the platform issues a service call, the system parses the command parameters, the virtual device executes the command and updates its local status, and reports the execution result and attribute change receipt according to the object model definition. The system writes the attribute changes after service execution into the device state machine to ensure that subsequent reports are consistent with the platform's expectations. Constraint checks are performed on dependencies (such as temperature and wind speed).
[0047] Collect connection, reporting, response, and anomaly metrics data during the testing process; perform automated assertion verification on the collected metrics data based on predefined assertion rules; specifically, collect metrics such as connection success rate, handshake latency, reporting throughput, average / tail latency, ACK hit rate, anomaly distribution, and service success rate, and aggregate them in multiple dimensions by device / protocol / scenario.
[0048] Generate and export a test report based on the assertion results and indicator data.
[0049] Specifically, the visual configuration interface allows users to configure the scenario name, duration, number of concurrent virtual devices, reporting cycle range (fixed or interval), time base (absolute or relative), and random seed to ensure test reproducibility.
[0050] Furthermore, virtual devices that conform to the corresponding product object model are generated in batches according to the product dimension, generating DeviceId / ClientId / username and key (or certificate path), supporting multi-product mixing, different firmware version tags and tenant isolation identifiers.
[0051] Furthermore, the implementation of connection authentication with the IoT platform includes: initiating the connection process of the corresponding protocol (MQTT CONNECT, HTTP / WebSocket handshake, CoAP DTLS session, TCP pure connection), recording handshake delay, handshake failure code and retry count; subscribing to downlink topics or registering instruction callbacks for protocols that require downlink interaction (such as calling Topic for MQTT subscription service, registering message type for WebSocket), and caching session information.
[0052] Furthermore, the implementation of simulated data reporting includes:
[0053] Based on the data generation strategy, a generator (fixed value, random range, normal / Poisson distribution, regular expression, script, history playback) is specified for each attribute / event, and units, precision and boundary constraints are bound.
[0054] During the simulated reporting process, network anomalies or data payload anomalies are injected probabilistically based on an event probability model. Specifically, network anomalies (latency, packet loss, jitter, disconnection, out-of-order delivery, replay) and payload anomalies (missing fields, out-of-bounds delivery, malformed JSON) are injected. Anomalies are applied to connections and payloads according to probability; upon triggering, exponential backoff reconnection, message caching and resending, and automatic rate degradation are executed, and the recovery time and amount of lost data are recorded.
[0055] Specifically, a high-precision timer or time wheel is started to control device-level / protocol-level reporting throttling (rate cap, burst bucket) to prevent the platform from being overwhelmed by instantaneous peaks. Based on the object model, attributes / event packets are grouped into JSON or binary formats; schema verification and data cleaning are performed, and signing, compression, or serialization (such as CBOR / Protobuf) is performed when necessary. QoS / Confirm / ACK modes are specified for different messages (such as MQTT QoS0 / 1 / 2, HTTP idempotent keys), and TraceId / message sequence number is injected to achieve end-to-end traceability. When sending reported data, a timeout timer is started, and ACK / receipt listening is performed; the sending time, ACK time, retransmission count, and final status (success / failure / discard) are recorded.
[0056] Furthermore, the event probability model is used to define the probability and time window of normal events, abnormal events, and alarm events during testing, respectively.
[0057] Furthermore, the implementation of listening to and processing service call instructions issued by the IoT platform includes: parsing instruction parameters, simulating service execution according to the definition of the product object model file, and reporting the execution results and associated attribute changes to the platform to form a closed-loop interaction.
[0058] Specifically, when the platform issues a service call, it parses the instruction parameters, the virtual device executes the call and updates its local state, and reports the execution results and attribute change receipts according to the object model definition. The attribute changes after service execution are written into the device state machine to ensure that subsequent reports are consistent with the platform's expectations; constraint verification is performed on dependencies (such as temperature and wind speed). Based on the error codes returned by the platform (authentication failure, topic not found, load verification failure, quota exceeded), the system retryes, bypasses, or escalates alarms according to the strategy, and automatically corrects the generated strategy or throttling threshold.
[0059] Furthermore, the implementation of automated assertion validation based on predefined assertion rules includes:
[0060] Based on the attribute value range and event sequence constraints defined in the product object model file, the first assertion rule is automatically generated; and custom assertion rules defined by the user through scripts or expressions are received and executed.
[0061] Specifically, assertion templates are automatically generated based on the object model and scenario (ACK latency ≤ threshold, packet loss rate ≤ threshold, attribute values within range, event order not out of order, service call closed loop completion), and judgment thresholds are bound to them. Users define complex assertions (cross-device / cross-protocol / time window aggregation) through visualization or script expressions, and the engine parses and executes them in real-time streams and offline batches. When an assertion fails or the SLA is exceeded, an alarm is generated, and suggestions are given based on root cause characteristics (e.g., reducing reporting frequency, improving QoS, extending timeout, correcting field precision).
[0062] Furthermore, the implementation of generating and exporting test reports includes: generating reports containing assertion results, key indicator time series plots, and links to raw data, and supporting export in at least one of the following formats: HTML, Excel, TXT, and PDF.
[0063] Furthermore, the method also includes saving the complete configuration information, random seed, and device interaction trajectory of the test scenario to support one-click reproduction and playback of the test scenario.
[0064] Specifically, the system archives scene configurations, random seeds, version numbers, device lists, and interaction trajectories, supporting one-click replay and A / B comparison to achieve problem reproduction and regression verification.
[0065] Furthermore, by enabling tenant isolation (namespaces / resource quotas / access control) in the engine, devices, connections, and reports of different tenants are not affected by each other, and parallel load testing is supported.
[0066] Furthermore, the scheduler distributes virtual devices and connection loads across multiple nodes, automatically scaling up and down based on target concurrency and SLA, and supports task migration and breakpoint continuation testing for failed nodes.
[0067] Furthermore, certificates and keys are securely managed and rotated, sensitive data is stored with de-identification, and logs and reports include access auditing to meet organizational compliance and auditing needs.
[0068] Furthermore, the protocol adapter, data generator, exception injector, assertion, and reporting all adopt pluggable interfaces, supporting the rapid access and reuse of new private protocols or industry protocols.
[0069] An IoT terminal reporting test system based on a multi-protocol unified simulation engine includes:
[0070] The product object model management module is used to load and parse the object model definition files of IoT products and establish an index of product object models for attributes, events and services.
[0071] The visual configuration module provides a configuration interface for receiving configuration parameters for the test scenario. These parameters include: protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. The protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP.
[0072] The protocol adaptation and communication module is used to enable connection and data communication with the IoT platform based on multiple protocols.
[0073] The device simulation and scheduling engine is used to generate virtual device instances based on configuration parameters and product object model indexes, and to schedule virtual devices to perform data reporting, event triggering and service response.
[0074] The anomaly injection module is used to inject network or data payload anomalies probabilistically according to a strategy during the testing process;
[0075] The assertion and reporting engine is used to collect test metrics, perform automated verification based on assertion rules, and generate test reports in multiple formats.
[0076] Furthermore, the system also includes a scenario archiving and reproduction module, which is used to save the configuration, random seed and execution trajectory of the test scenario, and provides a scenario replay interface.
[0077] This invention unifies multiple mainstream IoT communication protocols, such as MQTT, CoAP, HTTP, WebSocket, and TCP, within a single simulation engine. It supports flexible switching or concurrent execution of multi-protocol tests within a single scenario, avoiding the drawbacks of fragmented and independently implemented multi-protocol systems in existing technologies. This invention introduces a device model-driven mechanism, which automatically parses product device model files and generates full-link simulation logic for attributes, events, and services, ensuring that the behavior of virtual devices is consistent with real devices. Through a visual configuration interface, users can intuitively set reporting cycles, event types, trigger probabilities for anomalies and alarms, network anomaly injection strategies, etc., significantly reducing the barrier to entry for test scenario orchestration. During test execution, the system supports probabilistic scheduling of normal events, abnormal events, and alarm events, and can enable virtual devices to execute and receive services issued by the platform, forming a closed-loop interaction. A built-in automatic assertion engine can automatically generate assertion rules for performance, correctness, and sequence based on the device model and scenario, and supports user-defined assertion DSLs to achieve automated verification of the testing process. This invention provides multi-format report generation capabilities, which can export test results, assertions, and key indicators to formats such as HTML, Excel, TXT, and PDF, and support chart-based display and original data traceability; it also has a result archiving and reproduction mechanism, saving scenario configurations, random seeds, and execution trajectories, which facilitates problem localization and regression testing.
[0078] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
Claims
1. A method and system for reporting test based on a multi-protocol unified simulation engine for an Internet of Things terminal, characterized in that, Includes the following steps: The IoT product object model definition file is loaded through a unified simulation engine, and the product object model index of attributes, events and services is parsed and established. The configuration parameters for the test scenario are received through a visual configuration interface. These parameters include protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. Protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP. Initialize the test scenario according to the configuration parameters, generate a virtual device that conforms to the corresponding product model, and initiate connection authentication with the IoT platform; During the test run, simulated reporting data conforming to the corresponding product model is generated based on the product model index, data generation strategy, and event probability model. During the simulated reporting process, network anomalies or data payload anomalies are probabilistically injected according to the event probability model. The data is then reported to the IoT platform through at least one selected protocol. Listen for and process service call instructions issued by the IoT platform; the virtual device corresponding to the product object model executes and updates the local device status. Collect connection, reporting, response, and anomaly metrics data during the testing process; perform automated assertion verification on the collected metrics data based on predefined assertion rules; Generate and export a test report based on the assertion results and indicator data. 2.The method of claim 1, wherein, The product model definition file describes the names, data types, value ranges, required constraints, and units of attributes, events, and services for each type of IoT terminal device. 3.The method of claim 1, wherein, The implementation of listening to and processing service call instructions issued by the IoT platform includes: parsing instruction parameters, simulating service execution according to the definition of the product object model file, and reporting the execution results and associated attribute changes to the platform to form a closed-loop interaction.
4. The test method for reporting of the multi-protocol unified simulation engine based Internet of Things terminal according to claim 1, characterized in that, The implementation of automated assertion validation based on predefined assertion rules includes: Based on the attribute value range and event sequence constraints defined in the product object model file, the first assertion rule is automatically generated; and custom assertion rules defined by the user through scripts or expressions are received and executed.
5. The test method for reporting based on the multi-protocol unified simulation engine of the Internet of Things terminal according to claim 1, characterized in that, The implementation of generating and exporting test reports includes: generating reports containing assertion results, key indicator time series plots, and links to raw data, and supporting export in at least one of the following formats: HTML, Excel, TXT, and PDF. 6.The method of claim 1, wherein, The method also includes saving the complete configuration information, random seeds, and device interaction trajectories of the test scenario to support one-click reproduction and playback of the test scenario.
7. The IoT terminal reporting test method based on a multi-protocol unified simulation engine according to claim 1, characterized in that, The unified simulation engine enables tenant isolation, configuring namespaces, resource quotas, and access controls for different tenants, ensuring that devices, connections, and reports of different tenants do not affect each other.
8. The IoT terminal reporting test method based on a multi-protocol unified simulation engine according to claim 1, characterized in that, The unified simulation engine distributes virtual devices and connection loads across multiple nodes through a scheduler, automatically scaling up and down based on target concurrency and SLA.
9. An IoT terminal reporting test system based on a multi-protocol unified simulation engine, wherein the IoT terminal reporting test method based on a multi-protocol unified simulation engine according to any one of claims 1 to 8 is characterized in that, The system includes: The product object model management module is used to load and parse the object model definition files of IoT products and establish an index of product object models for attributes, events and services. The visual configuration module provides a configuration interface for receiving configuration parameters for the test scenario. These parameters include: protocol selection, virtual device identity, data generation strategy, event probability model, and exception injection strategy. The protocol selection includes choosing at least one protocol from MQTT, CoAP, HTTP, WebSocket, and TCP. The protocol adaptation and communication module is used to enable connection and data communication with the IoT platform based on multiple protocols. The device simulation and scheduling engine is used to generate virtual device instances based on configuration parameters and product object model indexes, and to schedule virtual devices to perform data reporting, event triggering and service response. The anomaly injection module is used to inject network or data payload anomalies probabilistically according to a strategy during the testing process; The assertion and reporting engine is used to collect test metrics, perform automated verification based on assertion rules, and generate test reports in multiple formats.
10. The IoT terminal reporting test system based on a multi-protocol unified simulation engine according to claim 9, characterized in that, The system also includes a scenario archiving and reproduction module, which is used to save the configuration, random seed and execution trajectory of the test scenario, and provides a scenario replay interface.
