A distributed MQTT high-concurrency connection test method based on ZeroMQ

Abstract

The application relates to a ZeroMQ-based distributed MQTT high-concurrency connection test method, which comprises the following steps: a control node and a plurality of Worker nodes establish a communication link through ZeroMQ, and each Worker node is instructed to create a corresponding number of Worker processes according to the number of local CPU cores; the control node broadcasts preparation instructions and subtask parameters to all Worker processes, so as to instruct each Worker process to execute a preparation process according to the corresponding subtask parameters; and the control node generates start instructions of each Worker process based on the target execution time of each Worker node, and sends the start instructions to each Worker process, so as to instruct each Worker process to execute a high-concurrency connection test in parallel. The application can truly and accurately evaluate the connection processing performance of a large-scale MQTT cluster under extreme pressure with low-cost hardware resources.

Description

Technical Field

[0001] This application relates to the field of connection testing, and in particular to a method, system, computer device, and computer-readable storage medium for testing high-concurrency distributed MQTT connections based on ZeroMQ. Background Technology

[0002] With the popularization of IoT technology, the MQTT protocol is widely used in scenarios involving massive device access, making the high-concurrency connection processing capability of the MQTT broker a key performance indicator. However, existing testing techniques have significant shortcomings when facing ultra-large-scale concurrent scenarios:

[0003] Traditional tools (such as emqtt_bench) run on a single host and are limited by the number of file handles in the operating system and network protocol stack bottlenecks, making it difficult to break through the theoretical upper limit of the number of connections on a single machine and unable to meet the verification needs of more than one million.

[0004] Although existing distributed solutions involve multiple nodes working together, the asynchronous startup times result in connection requests being discretely distributed over time, making it impossible to truly reproduce the "connection storm" characteristics of devices coming online in a concentrated manner, leading to distortion of the measured peak rate.

[0005] Existing solutions mostly adopt a fixed task allocation model, failing to consider the performance differences of heterogeneous servers and the physical characteristics of multi-core CPUs. This not only leads to idle resources on high-performance nodes but also makes low-performance nodes the bottleneck, severely restricting the overall stress testing efficiency of the cluster.

[0006] Therefore, there is an urgent need for a distributed high-concurrency testing method that supports millisecond-level synchronization across nodes and can adapt to heterogeneous environment resources. Summary of the Invention

[0007] This application provides a method, system, computer device, and computer-readable storage medium for testing high-concurrency distributed MQTT connections based on ZeroMQ, in order to at least solve the problems in the related art.

[0008] In a first aspect, embodiments of this application provide a method for testing high-concurrency distributed MQTT connections based on ZeroMQ. The method is applied to a high-concurrency testing system including a control node and several worker nodes. The method includes:

[0009] The control node establishes a communication link with several worker nodes through ZeroMQ and instructs each worker node to create a corresponding number of worker processes based on the number of CPU cores on its machine.

[0010] The control node broadcasts preparation instructions and subtask parameters to all worker processes, instructing each worker process to execute the preparation process according to its corresponding subtask parameters;

[0011] After receiving the ready responses from all Worker processes, the control node compensates for the global unified startup time through a clock synchronization mechanism based on signal round-trip time, and determines the target execution time for each Worker node.

[0012] The control node generates start instructions for each worker process based on the target execution time corresponding to each worker node and sends them to each worker process to instruct each worker process to execute high-concurrency connection tests in parallel.

[0013] In some embodiments, a clock synchronization mechanism based on signal round-trip time is used to compensate for the globally unified execution time, and the target execution time for each Worker node is determined by including:

[0014] The first moment when the control node sends the preparation command, the second moment when the worker node receives the preparation command, and the third moment when the control node receives the ready response are determined.

[0015] The difference between the second time point and the first time point is calculated as the first transmission delay, and the difference between the third time point and the second time point is calculated as the second transmission delay;

[0016] The delay difference is obtained based on the first transmission delay and the second transmission delay, and half of the delay difference is determined as the clock offset of the Worker node relative to the control node;

[0017] The target execution time for each Worker node is obtained by superimposing the preset global unified startup time of the control node with the clock offset of each Worker node.

[0018] In some embodiments, the control node instructs each worker process to perform high-concurrency connection tests in parallel, including:

[0019] When the local clock reaches the target execution time, each worker process calls the emqtt_bench conn instruction to concurrently execute MQTT connections;

[0020] The connection log stream during the MQTT connection process is asynchronously sent back to the control node via ZeroMQ.

[0021] In some embodiments, when the computing performance of each Worker node differs, the process of generating the subtask parameters includes:

[0022] The control node receives the stress test parameters input by the user;

[0023] The control node sends a performance testing command, instructing the worker node to perform a short-term connection benchmark test to record the benchmark time required to establish a preset number of connections.

[0024] In addition, the ratio of the number of physical CPU cores of the Worker node to the baseline latency is determined as the performance weight of the Worker node, and the performance weight is returned to the control node;

[0025] Based on the performance weight and CPU core count of each worker node, the load test parameters are broken down into subtask parameters corresponding to each worker process.

[0026] In some embodiments, the load testing parameters are broken down into subtask parameters corresponding to each worker process, including:

[0027] The control node summarizes the performance weights reported by all Worker nodes and adds up the individual performance weights to obtain the global total weight.

[0028] Calculate the proportion of the performance weight of each Worker node to the total global weight;

[0029] The load testing parameters are weighted according to the weight ratio of each Worker node to determine the node task parameters undertaken by each Worker node. The node task parameters are then evenly distributed according to the number of physical CPU cores of each Worker node to obtain the sub-task parameters undertaken by each Worker process within the Worker node.

[0030] In some embodiments, the method further includes a heartbeat detection step:

[0031] Each Worker process sends a heartbeat packet containing the node identifier, process identifier, and current connection status to the control node at a preset frequency.

[0032] If the control node does not receive a heartbeat packet from any Worker process within a preset time threshold, it determines that the Worker process is abnormal and continues to execute the test tasks of the remaining Worker processes.

[0033] In some embodiments, asynchronously sending the connection log stream during the MQTT connection process back to the control node via ZeroMQ includes:

[0034] The Worker process obtains the standard output stream, extracts the established connection value and the elapsed time value, encapsulates them into a JSON format message, and sends it to the control node.

[0035] The control node receives the JSON format message and aggregates messages with the same elapsed time value to obtain the total number of global connections and connection rate under that elapsed time value, and then displays them.

[0036] In some embodiments, the method further includes:

[0037] The task synchronization and status registration between the control node and the worker node adopts the Router-Dealer mode;

[0038] The control node issues the preparation command and the start command in Pub-Sub mode;

[0039] The Worker process sends connection status logs and heartbeat packets back to the control node in Push-Pull mode.

[0040] Secondly, embodiments of this application provide a distributed MQTT high-concurrency connection testing system based on ZeroMQ. The system is applied to high-concurrency testing including a control node and several worker nodes. The system includes: a control module, a synchronization module, and a command generation module, wherein:

[0041] The control module is used to establish a communication link between the control node and several Worker nodes via ZeroMQ, so as to instruct each Worker node to create a corresponding number of Worker processes according to the number of CPU cores on its own machine.

[0042] The synchronization module is used to broadcast preparation instructions and subtask parameters to all Worker processes, so as to instruct each Worker process to execute the preparation process according to its corresponding subtask parameters.

[0043] Furthermore, after receiving the ready responses from all Worker processes, the global unified startup time is compensated through a clock synchronization mechanism based on signal round-trip time, and the target execution time corresponding to each Worker node is determined.

[0044] The instruction generation module is used to generate start instructions for each Worker process based on the target execution time corresponding to each Worker node and send them to each Worker process to instruct each Worker process to execute high-concurrency connection tests in parallel.

[0045] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0047] Compared to related technologies, the distributed MQTT high-concurrency connection testing method based on ZeroMQ provided in this application has the following advantages:

[0048] Employing a distributed architecture based on ZeroMQ, it overcomes the limitations of single-machine resources; it introduces clock compensation based on signal round-trip time (RTT) and a three-level synchronization protocol to control the execution error of all network nodes to the millisecond level. This ensures that massive connection requests are executed concurrently at the same time, realistically reproducing extreme instantaneous concurrency pressure and guaranteeing high fidelity of test results.

[0049] Worker processes are created based on the number of physical CPU cores, and a benchmark-based performance self-evaluation mechanism is introduced. By calculating the performance weight of each node and performing weighted task splitting, the problem of uneven resource utilization in heterogeneous cluster environments is solved, avoiding weak nodes from dragging down overall performance and significantly improving the global maximum concurrent output capability.

[0050] Through real-time asynchronous feedback and heartbeat detection mechanisms of the underlying log stream, fine-grained monitoring and aggregated display of the status of massive connections are achieved. Abnormal processes can be isolated in a timely manner, ensuring system stability and data accuracy during large-scale, long-term stress testing. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0052] Figure 1 This is a flowchart of a distributed MQTT high-concurrency connection testing method based on ZeroMQ according to an embodiment of this application;

[0053] Figure 2 This is a flowchart of another distributed MQTT high-concurrency connection testing method based on ZeroMQ according to an embodiment of this application;

[0054] Figure 3This is a structural block diagram of a distributed MQTT high-concurrency connection testing system based on ZeroMQ according to this embodiment;

[0055] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0057] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0058] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0059] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0060] Among related technologies, MQTT load testing tools mostly rely on single-machine operation and lack a precise synchronization mechanism in a distributed environment, which makes it impossible to truly simulate the "connection storm" scenario of a large number of devices going online instantly, and the resource utilization rate is low in heterogeneous server clusters.

[0061] In view of this, embodiments of this application provide a distributed MQTT high-concurrency connection testing method based on ZeroMQ. This method achieves high-fidelity and high-efficiency distributed load testing through precise clock synchronization compensation and dynamic task allocation based on performance weights.

[0062] Figure 1 This is a flowchart illustrating a distributed MQTT high-concurrency connection testing method based on ZeroMQ, according to an embodiment of this application. Figure 1 As shown, the process includes the following steps:

[0063] S101: The control node establishes a communication link with several worker nodes through ZeroMQ and instructs each worker node to create a corresponding number of worker processes based on the number of CPU cores on its machine.

[0064] Specifically, the test system in this embodiment adopts a star topology, including one controller node and several worker nodes. The controller node acts as the central coordinator and is responsible for global scheduling; the worker nodes act as execution units and are responsible for generating connection pressure.

[0065] First, the control node starts the ZeroMQ service port, opens the registration channel (e.g., using ZMQ_ROUTER mode) and the log receiving port (e.g., using ZMQ_PULL mode), and waits for the worker nodes to connect.

[0066] Furthermore, the Worker node script is started on each load testing machine. Upon startup, the Worker node script first calls a system interface (such as Python's psutil.cpu_count(logical=False)) to automatically detect the number of physical CPU cores N on the machine. It should be noted that this specifically refers to the number of physical cores, not the number of logical threads. This is to avoid resource contention that may arise from hyper-threading technology in high-concurrency network I / O-intensive tasks, ensuring that each core has exclusive access to computing resources.

[0067] Furthermore, based on the detected number of physical cores N, the Worker node uses multi-processing technology (such as the multiprocessing module) to start N independent Worker processes. Each Worker process is assigned a unique index number, worker_index, which will be used later to construct a unique client ID prefix to prevent connection conflicts. After each Worker process starts, it sends a registration request to the control node via ZeroMQ to establish a bidirectional communication link.

[0068] S102, the control node broadcasts preparation instructions and subtask parameters to all worker processes, instructing each worker process to execute the preparation process according to its corresponding subtask parameters.

[0069] In this embodiment, the number of Worker processes is dynamically adjusted according to the number of physical CPU cores of each Worker node, and the global load test parameters input by the user are evenly divided according to the number of Worker processes to generate sub-task parameters that each Worker process should undertake.

[0070] In the implementation described above in this embodiment, a uniform partitioning strategy is adopted. This strategy assumes that all Worker nodes in the cluster have the same unit computing power and focuses primarily on the full utilization of single-machine resources. Specifically, the concurrency granularity is dynamically adjusted based on the number of physical CPU cores detected on the target machine. This mechanism can effectively avoid idle multi-core processor resources or intensified thread competition caused by a mismatch between concurrency granularity and physical cores, thereby ensuring that a single Worker node can reach its theoretical maximum concurrent output.

[0071] More preferably, considering that in actual distributed deployment scenarios, Worker nodes (stress testers) often exhibit significant heterogeneity (i.e., different CPU models, clock speeds, cache sizes, or memory bandwidths). In heterogeneous environments, while the above-mentioned uniform partitioning strategy solves the resource utilization problem within a single machine, it still has limitations from the perspective of the entire cluster: weaker Worker nodes may become the bottleneck of the system due to full load, limiting the peak global connection rate; while stronger Worker nodes may be underloaded, resulting in wasted computing power and affecting the task synchronization effect between multiple nodes.

[0072] In this embodiment, the Worker node performs a self-evaluation during registration (or PREPARE phase), and the control node allocates subtask parameters proportionally and weighted according to the evaluation results. This addresses the problem of uneven resource utilization in heterogeneous environments by upgrading the static allocation of subtasks based on the number of cores to a dynamic allocation based on the number of cores combined with performance weights, further improving the system's resource utilization strategy.

[0073] To maximize the overall load testing capability of the cluster, this embodiment adopts a dynamic allocation strategy based on performance weights when generating subtask parameters.

[0074] Specifically, this step includes the following sub-steps:

[0075] S102.1, The control node receives the stress test parameters input by the user.

[0076] The load test parameters include the target Broker address, port, total global connections (Ctotal), and global connection rate (Rtotal).

[0077] S102.2, the control node sends a performance testing command, instructing the worker node to perform a short-term connection benchmark test.

[0078] It should be noted that this step is performed before the formal load test (or during the registration phase), and the Worker process performs a short-term benchmark test. For example, it runs emqtt_bench to establish a small number of connections (such as 1000) and records the benchmark time ΔTbase required to complete the task.

[0079] S102.3 Calculate performance weights and report them.

[0080] Worker nodes calculate their performance weight Wi based on the number of physical cores Ni and the baseline latency ΔTbase_i. The optional calculation formula is Wi = Ni / ΔTbase_i. It can be understood that a larger weight Wi indicates a stronger connection establishment capability per unit time. The worker process reports this weight value to the control node via ZeroMQ.

[0081] In this system, Ni determines the number of concurrent execution units (i.e., processes) that can be created on the Worker node. △Tbase_i quantifies the node's task processing efficiency and speed by actually running load testing tasks. The ratio of these two metrics serves as the performance weight for the Worker node, addressing the load distribution problem in heterogeneous clusters. It accurately reflects the Worker node's load testing capability per unit time. Through this weight, task allocation is based on the machine's actual load testing capability, thereby achieving precise load balancing across the cluster and maximizing overall efficiency.

[0082] S102.4, the control node splits tasks according to performance weights and broadcasts preparation instructions.

[0083] The control node aggregates the weights of all online Worker nodes and calculates the global total weight ∑W. Then, based on the weight ratio of each Worker node, the user-input global load testing parameters are weighted and split to determine the total node task parameters that each Worker node should undertake:

[0084] Subsequently, the control node distributes the total task parameters of the node equally to all corresponding Worker processes on the node based on the number of physical CPU cores automatically detected by the Worker node, thereby obtaining the sub-task parameters that each Worker process should ultimately undertake.

[0085] Wherein, the number of connections of a node Ci = Ctotal × (Wi / ∑W), and the rate Ri = Rtotal × (Wi / ∑W); the number of connections that each Worker process on node i should handle is Cprocess = Ci / Ni, and the rate that each Worker process on node i should handle is Rprocess = Ri / Ni.

[0086] This method introduces a dynamic task allocation strategy based on performance weights. Performance weights Wi are calculated through short-term benchmark self-evaluation of Worker nodes (stress testers), and global tasks are weighted and split accordingly. This further addresses the issue of uneven resource utilization in heterogeneous cluster environments, preventing weak nodes from becoming bottlenecks and avoiding resource idleness on high-performance nodes. It significantly improves the overall peak connection rate of the distributed stress testing system. Even when all Workers have the same number of CPU cores but different performance levels, the cluster's stress testing capacity can be maximized and globally optimized, thus more efficiently and realistically evaluating the target Broker's maximum processing capacity.

[0087] Finally, the control node broadcasts the PREPARE instruction for the above subtask parameters to all worker processes via ZeroMQ's publish-subscribe pattern (ZMQ_PUB). Upon receiving the instruction, the worker process preloads the testing tools (such as building the emqtt_bench command string), initializes the necessary resources, but pauses execution and remains in a silent waiting state.

[0088] S103 After receiving the ready responses from all Worker processes, the control node compensates for the global unified startup time through a clock synchronization mechanism based on signal round-trip time, and determines the target execution time for each Worker node.

[0089] It should be noted that due to network transmission latency and machine clock drift, starting up using a globally unified time can lead to inconsistent actions among nodes. This embodiment uses a three-level synchronization protocol combined with an RTT compensation algorithm to solve this problem.

[0090] Specifically, the steps include:

[0091] S103.1 Receive the ready response and record the timestamp.

[0092] Once the Worker process has completed its preparation work, it immediately sends a "READY" confirmation message back to the control node. The control node records three key time points for each Worker node:

[0093] t1: Local time when the control node sends the "PREPARE" command (or a dedicated time synchronization probe packet);

[0094] t2: The local time when the Worker node receives the instruction (included in the response message);

[0095] t3: Local time when the control node receives the "READY" response.

[0096] S103.2, Calculate the clock offset.

[0097] The difference between the second time point and the first time point is calculated as the first transmission delay, and the difference between the third time point and the second time point is calculated as the second transmission delay; the delay difference is obtained based on the first transmission delay and the second transmission delay, and half of the delay difference is determined as the clock offset of the Worker node relative to the control node;

[0098] Specifically, based on the aforementioned timestamps, the control node calculates the round-trip time (RTT), i.e., t3-t1; assuming the network's bidirectional delay is symmetrical, the unidirectional delay is RTT / 2. At this point, the clock offset of the Worker node relative to the control node, clock_offset, can be calculated using the following formula:

[0099]

[0100] This offset reflects how fast the Worker node's local clock is relative to the Control node's clock.

[0101] S103.3, Determine the target execution time with compensation.

[0102] The control node sets a future globally uniform start time Tstart (e.g., current time + 2 seconds). To compensate for clock differences, the control node calculates a target execution time with a compensation value for each Worker node i. :

[0103]

[0104] S104, the control node generates start instructions for each worker process based on the target execution time corresponding to each worker node and sends them to each worker process to instruct each worker process to execute high-concurrency connection tests in parallel.

[0105] The control node will calculate the... Encapsulated in the "START" command, it is sent to the corresponding Worker process via ZeroMQ.

[0106] After the Worker process receives the START command, it parses out... It continuously polls and checks the local system clock. When the local time reaches... At that instant, all Worker processes will trigger connection operations simultaneously.

[0107] The inventors in this case discovered through experiments that in a cluster without precise time synchronization, the clock error between nodes (which can reach several milliseconds) is usually significantly greater than the network transmission delay (generally less than 1ms). Therefore, this offset becomes the main factor affecting synchronization accuracy. Based on this, the control node sets a target execution time with a compensation value for the START instruction; this ensures that all worker processes start the connection task uniformly when their local clocks reach the corresponding time, ultimately controlling the network-wide execution time error within ±1ms, thereby forming a real, highly accurate concurrent surge of connection requests at the target Broker.

[0108] S105, the Worker process executes connection tests in parallel and asynchronously sends back connection log streams via ZeroMQ.

[0109] Once the local clock reaches the target execution time, the Worker process immediately invokes the underlying load testing tool (such as emqtt_bench conn) to execute the task. To achieve real-time monitoring, the Worker process uses the following mechanism to process data:

[0110] The worker process captures the standard output stream (stdout) of emqtt_bench in real time. Load testing tools typically output logs line by line, such as "connected:1000,used:1s". The worker process parses each line of logs, extracting key values ​​such as "number of established connections" (connected) and "used time" (used_seconds).

[0111] Subsequently, the worker process encapsulates this data, along with its own node ID and process ID, into a JSON message and asynchronously sends it to the control node via ZeroMQ's PUSH port. This streaming asynchronous reporting mechanism avoids frequent I / O blocking the load testing process, ensuring high performance in the test.

[0112] S106 controls the aggregation and display of log data on the node.

[0113] The control node receives JSON log streams from all worker processes via the PULL port. Due to potential network jitter, the control node performs time alignment and aggregation on the received messages based on the used_seconds field.

[0114] Specifically, at each full second, the control node sums the number of connections for all worker processes within the same `used_seconds`, calculating the current total global cumulative connections. Simultaneously, by calculating the difference in the total number of connections between adjacent full seconds, the real-time global connection rate is obtained. Finally, the control node displays the integrated load test results (e.g., [1s] Total: 9,850 | Rate: 9,850 / s) on the command-line interface, allowing users to monitor the overall load test status of the cluster in real time.

[0115] In addition, the system also implements a heartbeat detection mechanism throughout the load testing process. The worker process sends a heartbeat packet containing status information every second; if the control node does not receive a heartbeat from a process for a certain period of time (e.g., 3 seconds), it is determined to be abnormal and removed during the statistics, but the testing of other normal processes is not interrupted, thus ensuring the robustness of the system.

[0116] also, Figure 2 This is a flowchart of another distributed MQTT high-concurrency connection testing method based on ZeroMQ according to an embodiment of this application, such as... Figure 2 As shown, the system first undergoes initialization and registration. The control node runs controller_launcher.py to start the ZeroMQ service port and enters a waiting connection state. The worker nodes run worker_launcher.py, which, by detecting the number of physical CPU cores N on the local machine, creates N worker processes and assigns them a unique index worker_index. They then send registration information to the control node to establish a communication link.

[0117] Next, the task distribution and synchronization preparation phase begins. The control node receives the load testing parameters input by the user (such as the target Broker and the total number of connections), splits the task, and sends the PREPARE command to all Workers. The Worker process receives the subtask parameters, constructs the emqtt_bench conn command but does not execute it, and sends a READY signal back to the control node.

[0118] Finally, a precise synchronous startup is executed. After the control node collects the READY signal, it calculates the clock offset of each node, sets the target execution time, and sends a START command via ZeroMQ. The worker processes poll for their local time, and once the current time arrives, they concurrently execute the emqtt_bench conn command. During execution, the worker processes capture standard output in real time, encapsulate the data, and send a log stream via ZeroMQ. The control node receives the log stream, processes and displays the data, and continues until the test is deemed complete. It then sends a stop signal via ZeroMQ, and the worker processes terminate the emqtt_bench process.

[0119] Through steps S101 to S106 above, this application utilizes ZeroMQ to construct a high-efficiency distributed communication architecture, resolves resource contention issues through physical core binding, addresses uneven distribution of heterogeneous resources through benchmark-based dynamic weight allocation, and achieves millisecond-level task synchronization through an RTT clock compensation mechanism. The synergistic effect of these technologies enables this application to realistically and accurately evaluate the connection processing performance of large-scale MQTT clusters under extreme pressure with low-cost hardware resources.

[0120] On the other hand, this embodiment also provides a distributed MQTT high-concurrency connection testing system based on ZeroMQ. Figure 3 This is a structural block diagram of a distributed MQTT high-concurrency connection testing system based on ZeroMQ according to this embodiment, as follows: Figure 3 As shown, the system includes: a control module 30, a synchronization module 31, and an instruction generation module 32, wherein:

[0121] The control module 30 is used to establish a communication link between the control node and several worker nodes through ZeroMQ, so as to instruct each worker node to create a corresponding number of worker processes according to the number of CPU cores on its own machine.

[0122] The synchronization module 31 is used to broadcast preparation instructions and subtask parameters to all worker processes, so as to instruct each worker process to execute the preparation process according to its corresponding subtask parameters;

[0123] Furthermore, after receiving the ready responses from all Worker processes, the global unified startup time is compensated through a clock synchronization mechanism based on signal round-trip time, and the target execution time corresponding to each Worker node is determined.

[0124] The instruction generation module 32 is used to generate start instructions for each worker process based on the target execution time corresponding to each worker node and send them to each worker process to instruct each worker process to execute high-concurrency connection tests in parallel.

[0125] This system utilizes ZeroMQ to build a high-efficiency distributed communication architecture, resolves resource contention issues through physical core binding, addresses uneven distribution of heterogeneous resources through benchmark-based dynamic weight allocation, and achieves millisecond-level task synchronization through an RTT clock compensation mechanism. The synergistic effect of these technologies enables this application to realistically and accurately evaluate the connection processing performance of large-scale MQTT clusters under extreme pressure with low-cost hardware resources.

[0126] In one embodiment, Figure 4This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, this electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides the environment for the operating system to run, the computer programs are executed by the processor to implement a distributed MQTT high-concurrency connection testing method based on ZeroMQ, and the database stores data.

[0127] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for testing high-concurrency distributed MQTT connections based on ZeroMQ, characterized in that, The method is applied to a high-concurrency test system including a control node and several worker nodes, and the method includes: The control node establishes a communication link with several worker nodes through ZeroMQ and instructs each worker node to create a corresponding number of worker processes based on the number of CPU cores on its machine. The control node broadcasts preparation instructions and subtask parameters to all worker processes, instructing each worker process to execute the preparation process according to its corresponding subtask parameters; After receiving the ready responses from all Worker processes, the control node compensates for the global unified startup time through a clock synchronization mechanism based on signal round-trip time, and determines the target execution time for each Worker node. The control node generates a start command for each worker process based on the target execution time corresponding to each worker node and sends it to each worker process to instruct each worker process to execute high-concurrency connection tests in parallel. When the computing performance of each Worker node differs, the process of generating the subtask parameters includes: The control node receives the stress test parameters input by the user; The control node sends a performance testing command, instructing the worker node to perform a short-term connection benchmark test to record the benchmark time required to establish a preset number of connections. In addition, the ratio of the number of physical CPU cores of the Worker node to the baseline latency is determined as the performance weight of the Worker node, and the performance weight is returned to the control node; Based on the performance weight and CPU core count of each worker node, the load test parameters are broken down into subtask parameters corresponding to each worker process.

2. The method according to claim 1, characterized in that, The global unified execution time is compensated by a clock synchronization mechanism based on signal round-trip time, and the target execution time for each Worker node is determined as follows: The first moment when the control node sends the preparation command, the second moment when the worker node receives the preparation command, and the third moment when the control node receives the ready response are determined. The difference between the second time point and the first time point is calculated as the first transmission delay, and the difference between the third time point and the second time point is calculated as the second transmission delay; The delay difference is obtained based on the first transmission delay and the second transmission delay, and half of the delay difference is determined as the clock offset of the Worker node relative to the control node; The target execution time for each Worker node is obtained by superimposing the preset global unified startup time of the control node with the clock offset of each Worker node.

3. The method according to claim 2, characterized in that, The control node instructs each worker process to perform high-concurrency connection tests in parallel, including: When the local clock reaches the target execution time, each worker process calls the emqtt_benchconn instruction to concurrently execute MQTT connections; The connection log stream during the MQTT connection process is asynchronously sent back to the control node via ZeroMQ.

4. The method according to claim 1, characterized in that, The load testing parameters are broken down into sub-task parameters corresponding to each worker process, including: The control node summarizes the performance weights reported by all Worker nodes and adds up the individual performance weights to obtain the global total weight. Calculate the proportion of the performance weight of each Worker node to the total global weight; The load testing parameters are weighted according to the weight ratio of each Worker node to determine the node task parameters undertaken by each Worker node. The node task parameters are then evenly distributed according to the number of physical CPU cores of each Worker node to obtain the sub-task parameters undertaken by each Worker process within the Worker node.

5. The method according to claim 3, characterized in that, The method also includes a heartbeat detection step: Each Worker process sends a heartbeat packet containing the node identifier, process identifier, and current connection status to the control node at a preset frequency. If the control node does not receive a heartbeat packet from any Worker process within a preset time threshold, it determines that the Worker process is abnormal and continues to execute the test tasks of the remaining Worker processes.

6. The method according to claim 5, characterized in that, The process of asynchronously sending the connection log stream during the MQTT connection process back to the control node via ZeroMQ includes: The Worker process obtains the standard output stream, extracts the established connection value and the elapsed time value, encapsulates them into a JSON format message, and sends it to the control node. The control node receives the JSON format message and aggregates messages with the same elapsed time value to obtain the total number of global connections and connection rate under that elapsed time value, and then displays them.

7. The method according to claim 1, characterized in that, The method further includes: The task synchronization and status registration between the control node and the worker node adopts the Router-Dealer mode; The control node issues the preparation command and the start command in Pub-Sub mode; The Worker process sends connection status logs and heartbeat packets back to the control node in Push-Pull mode.

8. A distributed MQTT high-concurrency connection testing system based on ZeroMQ, characterized in that, The system is applied to high-concurrency testing involving a control node and several worker nodes. The system includes: a control module, a synchronization module, and an instruction generation module, wherein: The control module is used to establish a communication link between the control node and several Worker nodes via ZeroMQ, so as to instruct each Worker node to create a corresponding number of Worker processes according to the number of CPU cores on its own machine. The synchronization module is used to broadcast preparation instructions and subtask parameters to all Worker processes, so as to instruct each Worker process to execute the preparation process according to its corresponding subtask parameters. Furthermore, after receiving the ready responses from all Worker processes, the global unified startup time is compensated through a clock synchronization mechanism based on signal round-trip time, and the target execution time corresponding to each Worker node is determined. The instruction generation module is used to generate start instructions for each Worker process based on the target execution time corresponding to each Worker node and send them to each Worker process to instruct each Worker process to execute high-concurrency connection tests in parallel. When the computing performance of each Worker node differs, the process of generating the subtask parameters includes: The control node receives the stress test parameters input by the user; The control node sends a performance testing command, instructing the worker node to perform a short-term connection benchmark test to record the benchmark time required to establish a preset number of connections. In addition, the ratio of the number of physical CPU cores of the Worker node to the baseline latency is determined as the performance weight of the Worker node, and the performance weight is returned to the control node; Based on the performance weight and CPU core count of each worker node, the load test parameters are broken down into subtask parameters corresponding to each worker process.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

Images