Monitoring method and device for distributed testing, electronic device, storage medium

Abstract

The embodiment of the present disclosure provides a monitoring method and device for distributed testing, an electronic device and a storage medium, and relates to the technical field of digital data processing. The monitoring method for distributed testing comprises the following steps: in the process of performing testing on a tested node in distributed testing, periodically acquiring the data amount of log data of a test process run by the tested node; and in response to the data amount of the log data acquired at a time point in a current period being equal to the data amount of the log data acquired at a time point in a previous period, determining that the tested node reaches a stable state in the current period.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of electronic digital data processing technology, and more particularly to monitoring methods for distributed testing, monitoring devices for distributed testing, electronic devices, readable storage media, and computer program products. Background Technology

[0002] In scenarios such as distributed deep learning training, multi-machine performance testing, and distributed system integration testing, test tasks typically need to be executed collaboratively on multiple nodes under test in a distributed deployment. These nodes under test can be independent operating system instances, such as physical machines, virtual machines, or container instances, and multiple nodes under test can be connected via a network to form a distributed test system.

[0003] In distributed testing, each node under test runs a test process. These test processes communicate and collaborate to complete predetermined test tasks. Because test tasks are executed asynchronously across multiple nodes under test, the termination of a distributed test is not determined by the exit of a single process on a single node. For example, some test processes on certain nodes may exit prematurely while others continue execution; or the test logic may be complete, but the test process may not have exited due to resource cleanup or other reasons. These situations make it difficult to accurately perceive whether test activities on each node under test in the distributed system have actually stopped, thus complicating the determination of the overall termination status of the distributed test.

[0004] Therefore, it is necessary to monitor the execution process of distributed testing in order to accurately perceive the test activity status on each node under test in the distributed system, thereby providing a reliable basis for judging the end status of the entire distributed test and supporting operations such as test result collection, resource release, and subsequent task triggering. Summary of the Invention

[0005] Embodiments of this disclosure provide monitoring methods and apparatuses, electronic devices, readable storage media, and computer program products for distributed testing that can at least partially solve the problems described above or other problems in the art.

[0006] According to a first aspect of this disclosure, a monitoring method for distributed testing is provided, the method comprising: periodically acquiring the amount of log data of the test process running by the node under test during the execution of a test on the node under test in the distributed test; and determining that the node under test has reached a stable state in the current period in response to the fact that the amount of log data acquired in the previous period is equal to the amount of log data acquired in the current period.

[0007] According to a second aspect of this disclosure, a monitoring device for distributed testing is provided, the device comprising: a monitoring unit configured to periodically acquire the amount of log data of the test process run by the node under test during the execution of a test on the node under test in the distributed test; and a comparison unit configured to determine that the node under test has reached a stable state in the current period in response to the fact that the amount of log data acquired in the previous period is equal to the amount of log data acquired in the current period.

[0008] According to a third aspect of this disclosure, an electronic device is provided, the electronic device including a processor that can be used to implement the monitoring method for distributed testing in the first aspect and any possible implementation of the first aspect.

[0009] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the monitoring method for distributed testing as described in the first aspect and any possible implementation thereof.

[0010] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the monitoring method for distributed testing in the first aspect and any possible implementation thereof.

[0011] The monitoring method, apparatus, electronic device, readable storage medium, and computer program product for distributed testing provided by the embodiments of this disclosure acquire the amount of log data of the test process running by the node under test at different periods during the distributed testing process, and determine whether the node under test has reached a stable state based on whether the amount of log data acquired at different periods is equal, thereby achieving effective monitoring of the activity status of each node under test during the distributed testing process. Using the change in the amount of log data of the node under test as an observation indicator provides an objective and quantifiable basis for determining whether the node under test still has testing activity. This does not require intruding into the internal testing process of the node under test to obtain status information, nor does it require the test program to actively output an end signal, and has the characteristics of simple monitoring method and wide applicability.

[0012] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0013] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0014] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figures 1 to 7 These are flowcharts of a monitoring method for distributed testing provided according to exemplary embodiments of this disclosure; Figure 8 This is a block diagram of a monitoring device for distributed testing provided according to an exemplary embodiment of the present disclosure; Figure 9 This is a schematic block diagram of an electronic device provided according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0015] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0016] The term “exemplary” as used herein means “serving as an example, implementation method, or illustration.” Any implementation method described herein as “exemplary” is not necessarily to be construed as superior to or better than other implementation methods.

[0017] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, apparatuses, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0018] Some embodiments of this disclosure provide a monitoring method for distributed testing. Figure 1 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0019] like Figure 1 As shown, the monitoring method 1000 for distributed testing includes: Step S1: During the testing process on the node under test in the distributed test, the log data of the test process running on the node under test is obtained periodically.

[0020] Step S2: In response to the fact that the amount of log data acquired at a time point in the current period is equal to the amount of log data acquired at a time point in the previous period, it is determined that the node under test has reached a stable state in the current period.

[0021] In scenarios such as distributed deep learning training, multi-machine performance testing, and distributed system integration testing, test tasks typically need to be executed collaboratively on multiple nodes under test deployed in a distributed manner. Taking the DeepEP (Deep Expert Parallelism) communication library as an example, this library is specifically designed for MoE (Mixture-of-Experts) and Expert Parallelism (EP), providing high-throughput and low-latency all-to-all communication capabilities. Its core functionality is implemented by GPU (Graphics Processing Unit) kernels. Therefore, when testing DeepEP, communication connections need to be established between multiple nodes under test deployed with GPUs to perform cross-node distributed testing, where these nodes under test respectively assume the attributes of server nodes and client nodes during the test.

[0022] A node under test can be understood as an independent computing unit that undertakes testing tasks in distributed testing. A node under test can be a physical machine, a virtual machine instance, or a container instance. Taking a physical machine as an example, a single node under test may include at least one of a Central Processing Unit (CPU) and a GPU.

[0023] In a distributed testing system, there are multiple nodes under test, which communicate and collaborate via network connections. These nodes can be deployed in various ways, belonging to the same or different node types. Specifically, in a decentralized (All-to-All) mode, all nodes are equal, with no absolute distinction between "server nodes" and "client nodes." Each node can undertake some coordination work, thus belonging to the same node type. In a master-slave / standby mode, there is one master server and at least one standby server, belonging to the same node type and requiring monitoring. In a cluster mode, the distributed nodes form a service cluster, and it is necessary to ensure the completion of all test tasks for the entire cluster. For example, the nodes can be divided into two different types: a server group and a client group, each including at least one node under test.

[0024] Multiple nodes under test deployed in a distributed manner can form a distributed testing system through network connections. During distributed testing, each node under test runs a test process, which communicates with each other and works collaboratively to complete the predetermined test tasks.

[0025] Because the test tasks are executed asynchronously on multiple nodes under test, the end of the test is not determined by the exit of a single node's process. For example, the test process on some nodes may exit prematurely while others are still executing; or the test logic may be completed but the test process has not yet exited due to resource cleanup or other reasons. These situations make it difficult to accurately determine the end status of the entire distributed test.

[0026] In some instances, the completion status of distributed tests is determined manually. Testers need to log in to the server node to manually start the test process, and then log in to each client node to manually start the test process. During the execution of the distributed test, they need to periodically log in to each machine to check the process status. After the communication between the test processes ends, they also need to log in to each machine to view the log data and result files to manually determine whether the distributed test has truly ended. Finally, they need to collect logs and results from multiple machines for summary analysis. This approach is quite cumbersome, and in multi-test-node scenarios, the workload increases linearly with the number of test nodes.

[0027] In other instances, a fixed timeout is set to determine the end of the distributed test. When the preset timeout is reached, the distributed test is forcibly terminated regardless of whether it has actually ended. This approach is simple to implement, but setting the timeout too short may cause the distributed test to terminate before it finishes, while setting it too long may lead to unnecessary waiting during the distributed test.

[0028] In some instances, the completion status of a distributed test is determined by checking whether the test processes are still running. The distributed test is considered complete when all test processes have exited. However, some test processes may exit delayed due to resource cleanup after completing their test logic, or some may exit abnormally, meaning the distributed test has not truly ended. These situations can lead to inaccurate judgments.

[0029] Based on the above, it is necessary to monitor the execution process of distributed testing to determine whether the testing activities have actually ended, thereby providing a basis for operations such as test result collection, resource release, and subsequent task triggering. The embodiments of this disclosure provide a monitoring method 1000 for distributed testing, which can effectively monitor the activity status of each node under test during distributed testing by observing the amount of log data of the test processes running on the nodes under test, without intruding on the testing process.

[0030] Specifically, a test process can be understood as a program instance running on the node under test to perform specific test tasks. Test processes can be started by testers or automatically by the test framework. During execution, test processes generate runtime records, status information, error messages, and result data, which are typically written to log data.

[0031] Log data consists of records generated during the execution of a test process. Alternatively, log data can be a text file generated via redirection, stored locally on the node under test or on externally accessible storage media. The content of the log data reflects the running status and test results of the test process, and the amount of log data grows as the test process outputs data. When the amount of log data stops changing (also known as log stability), it usually means that the test process has stopped generating new output. The amount of log data can be understood as the number of bytes in the log data; this is a standard attribute provided by the operating system and can be obtained through file access interfaces.

[0032] The distributed testing process (hereinafter referred to as distributed testing) involving multiple nodes under test can be executed periodically. For example, the testing process may include multiple preset periods (hereinafter referred to as periods), where these periods can be understood as specific points in time or time intervals for observing / monitoring the status of the nodes under test during the distributed testing process. The specific settings for these multiple periods can be selected according to the needs of the actual testing scenario. For example, multiple periods can be set up after the distributed test has started and run for a period of time to ensure that the distributed test has entered a stable execution state.

[0033] The initial cycle that starts first among multiple cycles can be flexibly determined based on the needs of the actual test scenario, and is not necessarily related to the start time of the distributed test. It should be noted that the previous cycle described in this disclosure is not necessarily the initial cycle after the distributed test has started. In other words, "first cycle" and "second cycle", or "current cycle" and "previous cycle" are only used to indicate the chronological order of multiple cycles, and are not used to limit the absolute order of the previous cycle among multiple cycles.

[0034] Step S1, during the testing of the nodes under test in the distributed test, involves periodically acquiring the log data of the test processes running on the nodes under test. This can be understood as follows: multiple periods are set during the distributed test, and the log data of the test processes running on each of the distributed nodes under test is observed within these periods. These multiple periods include at least two periods that are sequentially adjacent, which can be the previous period and the current period following the previous period.

[0035] For a single node under test, it is necessary to obtain the number of bytes of log data from the test processes running on that node in the current period. Log data can be obtained through the file access interface provided by the operating system. Alternatively, for local nodes such as server nodes, direct access is possible; for example, the `os.path.getsize()` function can be used in Linux systems to obtain the amount of local log data. Furthermore, for remote nodes such as client nodes, commands can be executed on the remote node using protocols such as Secure Shell (SSH) to obtain the amount of log data. Regardless of whether local or remote access is used, the amount of log data obtained can be recorded as an observation value for that node in the current period.

[0036] Step S2, responding to the fact that the amount of log data acquired at a time point in the current period is equal to the amount of log data acquired at a time point in the previous period, determines that the node under test has reached a stable state in the current period. This can be understood as follows: for a single node under test, the amount of log data acquired in the previous period is compared with the amount of log data acquired in the current period. If they are equal, it means that in two adjacent periods in time sequence, the amount of log data corresponding to the node under test has not changed, or that the test process running on the node under test has not generated new log output during that time period, and the log data has become stable. Based on this, it can be determined that the node under test has reached a stable state in the current period. A stable state indicates that the test activities on the node under test have tended to stop, or that the test process running on the node under test no longer generates new log output.

[0037] It should be noted that equal log data volume can be understood as the number of bytes of log data acquired in two consecutive cycles being exactly the same. In other words, the difference in log data volume between two consecutive cycles is 0. Since log data volume is measured in bytes, and bytes are the smallest unit of granularity that the operating system can observe and modify, whether the log data volume is equal is a clear criterion and can serve as an objective basis for determining whether testing activities on the tested node have ceased.

[0038] Therefore, by periodically acquiring the log data volume of the test processes running on the tested node during distributed testing, and determining whether the tested node has reached a stable state based on whether the log data volume acquired in different periods is equal, it is possible to effectively monitor the activity status of multiple tested nodes in distributed testing. Using the change in log data volume as an observation indicator provides an objective and quantifiable basis for judging whether the tested node is still conducting test activities. This does not require intruding into the test process to obtain status information, nor does it require the test program to actively output a termination signal, and it has the characteristics of simple monitoring method and wide applicability.

[0039] Figure 2 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0040] like Figure 2 As shown, as an alternative, the monitoring method 1000 for distributed testing may also include: Step S3: Determine the end of the distributed test based on the fact that at least multiple tested nodes in the distributed test have reached a stable state in the current period.

[0041] In step S2, by comparing the amount of log data acquired in the previous period with the amount of log data acquired in the current period, it can be determined whether a single node under test has reached a stable state in the current period. A node under test that has reached a stable state has stopped generating new log outputs, indicating that the testing activities on that node have tended to cease. When multiple nodes under test in the distributed test have reached a stable state in the current period, it means that the testing activities on those nodes have all stopped, and at this point, it can be preliminarily considered that the entire distributed test has ended.

[0042] It should be noted that the end of distributed testing means that the test tasks on multiple tested nodes in a distributed deployment have actually ended, and all test activities on these nodes have ceased. At this point, it is safe to perform operations such as collecting test results, releasing computing resources, and triggering subsequent test tasks. The fact that multiple tested nodes have reached a stable state in the current period provides an objective and quantifiable basis for determining the end of distributed testing. This can serve as a sufficient or necessary condition for the end of distributed testing, and can be combined with other judgment mechanisms depending on the needs of the actual testing scenario.

[0043] Through step S3, the monitoring method 1000 for distributed testing provided by the embodiments of this disclosure can directly or indirectly determine the end of the entire distributed test based on the stable state of the tested node determined by the cessation of changes in the amount of log data. Compared with judgment methods that rely on fixed timeouts or single process states, this can identify the end time of the distributed test more promptly and accurately, reducing the probability of misjudgments caused by delayed exits or abnormal suspensions of the test process. Furthermore, the monitoring method 1000 for distributed testing does not require waiting for a fixed timeout, thereby improving the efficiency and resource utilization of distributed testing.

[0044] Figure 3 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0045] like Figure 3 As shown, in some embodiments of this disclosure, the monitoring method 1000 for distributed testing may further include: Step S2-1: In multiple cycles after the current cycle, the amount of log data of the tested node is obtained respectively; and in response to the fact that the amount of log data obtained in one of the multiple cycles is equal to the amount of log data obtained in the previous cycle of one of the multiple cycles, it is determined that the tested node has reached a stable state in one of the multiple cycles.

[0046] In this implementation, in order to further improve the reliability of the distributed test completion judgment, a configurable parameter of "predetermined number of times" is introduced, and the stable state of the tested node is confirmed by continuous verification of multiple cycles of a predetermined number of times.

[0047] Specifically, the predetermined number of times is a pre-set number of cycles for which stable state verification needs to be performed continuously before monitoring the distributed testing process. This parameter can be adjusted according to the needs of the actual testing scenario. For example, for scenarios where the accuracy of the distributed test completion judgment is critical, a larger predetermined number of times can be set to ensure that the tested node has indeed stopped generating new log output through multiple continuous verifications; for scenarios where the response speed of the distributed test completion judgment is critical, a smaller predetermined number of times can be set to quickly obtain the judgment result. This configurable parameter mechanism enables the monitoring method 1000 for distributed testing disclosed herein to adapt to various distributed testing scenarios ranging from minute-level distributed testing to hour-level distributed testing, demonstrating good versatility and practicality.

[0048] The specific value of the predetermined number of times can be configured in various ways, and this disclosure does not limit this. For example, the predetermined number of times can be set through command-line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios. For instance, the predetermined number of times can be specified in the command line by setting parameters such as `--stable-threshold`; in a CI / CD (Continuous Integration / Continuous Deployment) pipeline, it can be configured through the environment variable `DEEPEP_STABLE_THRESHOLD`; for multiple distributed testing scenarios that require batch management, it can also be uniformly set through JSON (JavaScript Object Notation) or YAML (YAML Ain't Markup Language) configuration files. This multi-layered parameter configuration mechanism enables the monitoring method 1000 for distributed testing to flexibly adapt to distributed testing scenarios of different scales, durations, and accuracy requirements. Users can choose the most suitable configuration method according to their actual needs without modifying the core monitoring logic.

[0049] Multiple cycles, ordered sequentially, are set after the current cycle. Alternatively, a predetermined time interval can exist between adjacent cycles. In other words, the log data volume of multiple tested nodes can be acquired in each of the multiple cycles with predetermined time intervals following the current cycle.

[0050] Optionally, the predefined time interval is a configurable parameter that represents the time span between adjacent periods, reflecting the frequency of the monitoring process. For scenarios with high requirements for the response speed of distributed test completion judgment, a smaller predefined time interval can be set to observe changes in log data volume more frequently, thereby capturing the stable state of the logs as soon as possible after the actual completion of the distributed test and shortening the detection latency. For scenarios that are sensitive to system overhead or where the distributed test execution time is long, a larger predefined time interval can be set to reduce the additional overhead caused by frequent monitoring and avoid interfering with the test process itself.

[0051] The specific value of the predetermined time interval can be configured in various ways, and this disclosure does not limit it. For example, the predetermined time interval can be set through command-line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios. For instance, the predetermined time interval can be specified in the command line by setting parameters such as `--check-interval`; in a CI / CD pipeline, it can be configured through the environment variable `DEEPEP_CHECK_INTERVAL`; for multiple distributed testing scenarios requiring batch management, the predetermined time interval can also be uniformly set through JSON or YAML configuration files. This configurable parameter mechanism enables the monitoring method 1000 used for distributed testing to flexibly adapt to distributed testing scenarios with different time scales and response requirements.

[0052] For example, monitoring can be set to occur every 10 seconds, with a predetermined number of times (up to 3). Alternatively, the previous period can be designated as the first period, the current period as the second period, and the periods following the current period can be designated as the third, fourth, and fifth periods, each with a predetermined time interval of 10 seconds. For each period (i.e., the third, fourth, and fifth periods), the log data volume of multiple nodes under test in the distributed test needs to be acquired and compared with the log data volume of the same node acquired in the previous period. In this way, the stable state of the nodes under test can be verified stage by stage.

[0053] It should be noted that the specific implementation method for obtaining the log data volume of the node under test in the embodiments of this disclosure can refer to the relevant description above, and will not be repeated here. Furthermore, this disclosure does not limit the specific method for obtaining the log data volume, as long as the data volume can be obtained at each periodic time point. In addition, as described in the above embodiments, "periodic" refers to periodically obtaining the log data volume at a preset time interval. Each period may include a time point for obtaining the log data volume, and there is a predetermined time interval between adjacent periods. By obtaining the log data volume at each periodic time point and comparing it with the log data volume obtained at the time point of the previous period, it is possible to accurately determine whether the log data volume has changed, thereby determining whether the testing activity of the node under test has stopped. In one embodiment, the log data volume of the node under test can be obtained at each periodic time point.

[0054] Alternatively, the predetermined number of cycles can be consecutive, or a predetermined number of cycles can be set consecutively after the current cycle, with each cycle adjacent to the previous one. Another option is that the predetermined number of cycles can be non-consecutive, as long as a predetermined number of cycles are set after the current cycle, with intervals between adjacent cycles. Furthermore, in practical applications, to ensure the real-time performance and accuracy of the stability assessment of the tested node, a continuous setting method can be adopted.

[0055] As another option, multiple cycles, a predetermined number of times, can be set within a pre-defined time window after the current cycle. For example, a 5-minute time window can be set, within which multiple cycles can be set at non-fixed or predetermined time intervals. The specific timing of the cycles can be flexibly determined according to actual needs. For example, multiple cycles can be set more intensively at the beginning of the time window to quickly respond to the end of the test, or multiple cycles can be set more intensively at the end of the time window to strengthen confirmation. Regardless of how the cycles are distributed within the time window, as long as the amount of log data of the tested node does not change within the predetermined number of consecutive cycles, the tested node can be confirmed to have reached a stable state.

[0056] Furthermore, the period setting can be correlated with the actual running status of the test process in distributed testing. For example, when a decrease in CPU utilization or reduction in network communication of a test process is detected, the monitoring interval can be automatically shortened to more precisely observe changes in the amount of log data corresponding to that test process; or when a test process is detected to be in a stable state for a long period of time, the monitoring interval can be appropriately lengthened to reduce unnecessary resource consumption.

[0057] Through step S2-1, the implementation method of this disclosure achieves continuous multiple verifications of the stable state of the node under test. Each additional verification means that the amount of log data of the node under test remains unchanged over a longer time span, thereby confirming with higher confidence that the testing activity on the node under test has indeed stopped. This continuous verification mechanism based on a predetermined number of verifications can effectively filter out accidental factors and instantaneous interference, significantly improving the robustness and reliability of the stability judgment of the node under test. By introducing a continuous period of a predetermined number of verifications and a configurable parameter mechanism, the universality and flexibility of the monitoring method 1000 for distributed testing are improved on the basis of improving the accuracy of the stability judgment of the node under test, and it can meet the diverse needs of different distributed testing scenarios.

[0058] Figure 4 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0059] like Figure 4As shown, in some embodiments of this disclosure, the monitoring method 1000 for distributed testing may further include: Step S3-1: Determine the end of the distributed test based on at least the fact that multiple tested nodes have reached a stable state in the current period, and that multiple tested nodes have reached a stable state in each of the multiple periods.

[0060] In step S2, by comparing the amount of log data acquired in the previous period with the amount of log data acquired in the current period, it can be determined whether a single node under test has reached a stable state in the current period. In step S2-1, multiple periods (e.g., the third period, the fourth period, etc.) are set after the current period, and by comparing the amount of the same log data acquired in adjacent periods, it is determined whether the node under test corresponding to that log data continues to maintain a stable state in these subsequent periods. When multiple nodes under test in the distributed test not only reach a stable state in the current period, but also continue to maintain a stable state in each of the subsequent predetermined number of periods, it means that the test process on these nodes has not generated new log outputs for multiple consecutive time intervals, and their test activities have been continuously stopped.

[0061] Based on this, the distributed test can be considered complete if at least multiple tested nodes reach a stable state in the current period, and also in each of multiple periods. This step uses the stable state information of the current period along with the stable state information of subsequent periods as the basis for determining the end of the distributed test. Specifically, this step requires that multiple tested nodes in the distributed test reach a stable state in the current period, and also reach a stable state in each of multiple periods a predetermined number of times after the current period, thereby confirming with higher confidence that the entire distributed test has ended.

[0062] In some embodiments of this disclosure, the above-mentioned judgment mechanism can be implemented using a stability counter. Specifically, a stability counter with an initial value of zero can be set; for each of multiple cycles, in response to multiple tested nodes reaching a stable state, the count of the stability counter is incremented; and in response to the count of the stability counter reaching or exceeding a predetermined number of times, the distributed test is determined to have ended.

[0063] In this implementation, the stability counter reaching or exceeding a predetermined number of counts indicates that multiple nodes under test in the distributed test have maintained a stable state for a predetermined number of consecutive periods, thus confirming the end of the distributed test. Introducing a stability counter to quantify the number of consecutive stable periods for a node under test is a simple implementation mechanism, requiring only the maintenance of a single integer variable to count the number of consecutive stable periods, eliminating the need for complex data structures or algorithms and reducing system overhead. Furthermore, the stability counter's count directly corresponds to the number of consecutive stable periods reached by the node under test; the count is updated each time a condition is met, and reaching a threshold triggers the termination decision. This logical relationship is clear and the causal relationship is explicit, facilitating real-time monitoring of test progress and enabling the output of intermediate states in automated testing processes.

[0064] As another implementation, the initial value of the stability counter can also be set to a predetermined number of times; for each of the multiple cycles, in response to the multiple tested nodes reaching a stable state, the count of the stability counter is reduced; and in response to the count of the stability counter decreasing to 0, the distributed test is determined to be over.

[0065] As another implementation method, the initial value of the stability counter can also be set to 1, which represents that multiple tested nodes in the distributed test have reached a stable state in the current period; for each of the multiple periods, in response to multiple tested nodes reaching a stable state, the count of the stability counter is incremented; and in response to the count of the stability counter reaching or exceeding a predetermined number of increments, the distributed test is determined to have ended.

[0066] In the above embodiments, by setting different initial values ​​for stability counters and counting increment methods (incrementing or decrementing), the number of times the tested node is continuously stable in the monitoring method 1000 used for distributed testing can be quantified. Those skilled in the art can choose an appropriate implementation method according to the needs of the actual testing scenario.

[0067] Optionally, in some embodiments of this disclosure, in order to achieve continuous multiple verifications of the stable state of the node under test, it is necessary to introduce a mechanism for successively comparing and recording the amount of log data, and to quantify it in conjunction with a stability counter.

[0068] Specifically, during continuous verification across multiple cycles, for a single node under test, the amount of log data acquired by that node in the previous cycle should be used as a reference value. The amount of data acquired in the current cycle should be compared with this reference value. For example, in the initial cycle, the amount of log data acquired by the node under test can be used as the initial reference value for subsequent comparisons. In each subsequent cycle (e.g., the current cycle, the third cycle, etc.), the amount of log data acquired by the node under test is acquired again, and compared with the reference value. If they are equal, it indicates that the node under test has reached a stable state in the current cycle; if they are not equal, it indicates that new data has been written to the node's log data during this period, and the test is still ongoing. In this case, the reference value can be updated to the amount of data acquired in the current cycle, and the counting of continuous stable states can be restarted.

[0069] By successively comparing and updating the reference value, every change in the amount of log data can be accurately captured, providing accurate input for the stability counter. As described above, the stability counter records the number of cycles in which multiple tested nodes in the distributed test consecutively reach a stable state. Optionally, in each cycle, in response to multiple tested nodes reaching a stable state, the stability counter increments by 1; in response to any tested node failing to reach a stable state in that cycle, or in other words, the amount of its log data changing relative to the reference value, the stability counter is reset to zero. When the stability counter reaches or exceeds a predetermined number of times, it indicates that multiple tested nodes in the distributed test have maintained a stable state for a predetermined number of consecutive cycles, thus confirming the end of the distributed test.

[0070] Therefore, through step S3-1, the implementation of this disclosure achieves the determination of the end of distributed testing based on multiple consecutive stable states of the tested node. By introducing a predetermined number of consecutive verifications, the end of distributed testing can be confirmed with higher confidence, reducing misjudgments caused by accidental factors or transient interference. In addition, the introduction of a stability counter simplifies and quantifies the continuous stability determination process, facilitating the integration of automated testing frameworks.

[0071] Figure 5 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0072] like Figure 5 As shown, in some embodiments of this disclosure, the monitoring method 1000 for distributed testing may further include: Step S3-2: Based on the fact that multiple tested nodes have reached a stable state in the current period, multiple tested nodes have reached a stable state in each of the multiple periods, and the test process run by at least one type of node has terminated, the distributed test is determined to be over.

[0073] As described above, in distributed testing, multiple nodes under test can exhibit various deployment modes; they can belong to the same type of node or to different types of node nodes. In other words, multiple nodes under test in distributed testing can include at least one type of node, and the division of node types can be flexibly defined according to the test architecture / deployment mode.

[0074] Taking cluster mode as an example, the multiple nodes under test in distributed testing can be divided into two different types of nodes: server-type nodes and client-type nodes. Server-type nodes are responsible for coordinating, distributing tasks, or summarizing results; client-type nodes are responsible for executing specific computation or communication tasks. Taking decentralized mode as an example, the multiple nodes under test in distributed testing are all peers. Each node under test has both the functions of a server, undertaking some coordination work, and the functions of a client, executing specific computation or communication tasks. Therefore, in this mode, the multiple nodes under test in distributed testing belong to the same type of node.

[0075] In step S2 above, by comparing the amount of log data acquired in the previous period with the amount of log data acquired in the current period, it is possible to determine whether a single node under test has reached a stable state in the current period. However, in some scenarios, log stability alone may still present uncertainties. For example, the test process may not generate new logs due to deadlock, suspension, or other reasons, but it may not have exited normally, meaning the distributed test has not actually ended. To at least address this issue, this step introduces a process state detection dimension, combining log stability with the process termination state of specific node types.

[0076] Specifically, based on the fact that multiple tested nodes in the distributed test have reached a stable state, it can be further determined that the test process run by at least one type of node has terminated. "Terminated" can also be understood as the test process run by that type of node no longer existing in the system, which may include the test process run by that type of node exiting normally or abnormally.

[0077] Alternatively, whether a test process has terminated can be checked using process checking commands provided by the operating system (e.g., pgrep). For local nodes under test, the process checking command can be executed directly; for remote nodes under test, the process status can be obtained by executing commands via SSH or other methods.

[0078] The termination of test processes for at least one type of node includes several possible decision logics. Taking cluster mode as an example, the distributed test can be determined to have ended based on at least several tested nodes reaching a stable state in the current period, and the termination of test processes for server-type nodes; alternatively, it can be based on at least several tested nodes reaching a stable state in the current period, and the termination of test processes for client-type nodes; furthermore, it can be based on at least several tested nodes reaching a stable state in the current period, and the termination of test processes for both server-type and client-type nodes. Taking decentralized mode as an example, since the multiple tested nodes in the distributed test belong to the same type of node, the termination of the distributed test can be determined based on at least several tested nodes reaching a stable state in the current period, and the termination of test processes for multiple tested nodes.

[0079] By combining log stability with process status, this implementation achieves a multi-dimensional comprehensive judgment of both log stability and process status. Compared to relying solely on log stability or solely on process status to determine the end of a distributed test, this implementation can effectively distinguish between complex situations such as "the test has ended but the test process has not exited" and "the test has not ended but the test process is abnormally suspended," significantly improving the accuracy and reliability of distributed test end determination.

[0080] It should be noted that the process state judgment mechanism in step S3-2 is used in conjunction with the aforementioned judgment based on continuous stable states. For example, it can be required that multiple tested nodes in a distributed test reach a stable state for multiple consecutive periods, and that the test process of at least one type of node has terminated, as a comprehensive judgment condition for the end of the distributed test. This multi-dimensional comprehensive judgment further improves the accuracy and robustness of the distributed test end judgment.

[0081] Furthermore, based on the fact that at least one of the tested nodes has not reached a stable state in the current period, the data volume of log data of the tested nodes is obtained in the third and fourth periods after the current period; based on the data volume of log data obtained respectively, the tested node that reaches a stable state in the fourth period is determined; and based on at least the fact that the tested nodes reach a stable state in the fourth period, the distributed test is determined to be over.

[0082] By comparing the amount of log data acquired in the previous period with the amount of log data acquired in the current period, it is possible to determine whether a single node under test has reached a stable state in the current period. However, in actual distributed testing, due to differences in the execution progress of test tasks on each node under test, some nodes may have reached a stable state in the current period, while others may not. To address this situation, this implementation provides a dynamic period advancement mechanism.

[0083] If at least one of the tested nodes is found to be not in a stable state in the current period, the monitoring process will not terminate but will continue. For example, in the third period after the current period, the log data volume of each tested node will be acquired. Subsequently, in the fourth period after the third period, the log data volume of each tested node will be acquired again. For a single tested node, by comparing the log data volume acquired in the third period with the log data volume acquired in the fourth period, it can be determined whether the tested node has reached a stable state in the fourth period. In other words, for any tested node in the distributed test, if the log data volume acquired in the third period is equal to the log data volume acquired in the fourth period, it can be determined that the tested node has reached a stable state in the fourth period.

[0084] After identifying the tested nodes that reached a stable state in the fourth cycle, it can be further determined whether all tested nodes in the distributed test reached a stable state in the fourth cycle. Based on this, the distributed test can be considered complete when at least several tested nodes have reached a stable state in the fourth cycle. It should be noted that the phrase "at least based on" means that this step can use the stable state information of the fourth cycle as the basis for determining the end of the distributed test, or it can be used in combination with other judgment conditions.

[0085] Therefore, the embodiments of this disclosure enable dynamic tracking of the completion status of distributed tests. When some tested nodes are stable while others are not, the monitoring process can automatically continue. This mechanism effectively addresses the inconsistency in execution progress among tested nodes in distributed tests, avoiding misjudgments or premature termination due to delays in individual tested nodes, and improving the robustness and adaptability of the monitoring method.

[0086] Furthermore, the settings for the third and fourth cycles are merely illustrative. In practical applications, more consecutive cycles (e.g., a fifth cycle, a sixth cycle, etc.) can be set as needed, until all tested nodes in the distributed test reach a stable state in a certain cycle. In other words, if some tested nodes in the distributed test are stable while others are not, the monitoring process can continue until a cycle in which all tested nodes in the distributed test are stable is found, and this cycle is used as the basis for determining the end of the distributed test.

[0087] Furthermore, this mechanism of continuously pushing forward until all tested nodes in the distributed test are stable can be combined with the aforementioned continuous stability verification mechanism based on a "predetermined number of times". For example, multiple tested nodes can be required not only to reach a stable state in a certain period, but also to maintain a stable state continuously in the subsequent predetermined number of periods before the distributed test is finally considered to be over.

[0088] As another implementation, a mechanism that continuously advances until all tested nodes in the distributed test are stable can be combined with a stability counter mechanism. For example, a stability counter with an initial value of zero can be set, and counting can begin in the period when all tested nodes have reached a stable state. For each subsequent period, the count is incremented in response to all tested nodes reaching a stable state; in response to any tested node not reaching a stable state, the counter is reset to zero, and the monitoring process continues until the counter value reaches a predetermined number of times.

[0089] Through the above methods, the implementation of this disclosure can not only handle situations where the execution progress of the tested nodes is inconsistent, but also filter out accidental factors and transient interference through a continuous and stable verification mechanism, thus confirming the end of the distributed test with higher confidence. This multi-layered judgment logic enables the monitoring method to adapt to more complex distributed testing scenarios, exhibiting good versatility and scalability.

[0090] Figure 6 This is a flowchart of a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0091] like Figure 6As shown, in some embodiments of this disclosure, the monitoring method 1000 for distributed testing may include two parallel monitoring dimension processes 100 and 200. Process 100 monitors the running status of the test processes running on the node under test; process 200 monitors the amount of log data of the test processes running on the node under test. By combining the process status information obtained from monitoring the running status of the test processes with the status information of the node under test obtained from monitoring the amount of log data, a multi-dimensional comprehensive judgment mechanism can be constructed, which can effectively improve the accuracy of the distributed test termination judgment.

[0092] In this implementation, as an option, the monitoring method 1000 for distributed testing may include: periodically detecting the running status of the test processes run by each of the multiple tested nodes; and determining that the distributed test has ended in response to detecting that the test processes run by each of the multiple tested nodes have terminated in any period.

[0093] Alternatively, the monitoring method 1000 for distributed testing may include: periodically detecting the running status of the test processes running on each of the multiple tested nodes; and, in response to detecting that at least one of the multiple test processes has not terminated in any period, determining that the distributed test has ended, based at least on the fact that the multiple tested nodes have reached a stable state in the current period.

[0094] For example, process 100 can monitor the running status of the test process running on the node under test, where the running status can include running (i.e., not terminated) and terminated. In multiple cycles including the previous cycle and the current cycle, it can be monitored whether the test process running on the node under test is still running or has been terminated.

[0095] Optionally, the running status of the test process can be detected using process checking commands provided by the operating system. For example, for a local node under test, commands such as pgrep can be used to check whether the test process it is running exists; for a remote node under test, process checking commands can be executed on the remote node under test via SSH or other means to obtain the running status information of the test process running on that node.

[0096] Based on process 100, the end of the distributed test can be determined independently based on the process status information, or the process status information can be combined with the status information of the tested node obtained from process 200 for a comprehensive judgment.

[0097] In the above comprehensive judgment process, a priority order can be set among different judgment criteria. Specifically, judgments based on process status information have higher priority: if the test processes running on multiple tested nodes in the distributed test are detected to have terminated in any period, the distributed test can be directly determined to be over without referring to the log stability information obtained in process 200. If process status information is insufficient to determine the end of the distributed test, the status information of the tested nodes obtained in process 200 can be taken into consideration. For example, the process termination status of at least one type of node can be combined with the status information of multiple tested nodes to jointly determine whether the distributed test has ended. This priority design ensures a rapid response when the test process exits normally, while improving the accuracy of the judgment by leveraging log stability when the process has not terminated.

[0098] For example, at the end of the current cycle, based on the acquired process status information, it can be determined whether at least one test process among the test processes running on multiple tested nodes has not terminated in multiple cycles. For instance, for any test process, based on the fact that it was detected as running in the previous cycle and is still running when detected in the current cycle, it can be determined that the test process has not terminated in either of these two cycles.

[0099] In this situation, since there are continuously running test processes, the termination of the distributed test cannot be determined solely based on the test process status. Therefore, the termination of the distributed test can be determined by initiating or combining the status information of the tested nodes obtained in process 200. This status information can be understood as whether the tested nodes have reached a stable state. For example, after determining that at least one test process has not terminated in either the previous or current cycle, the termination of the distributed test can be comprehensively determined based on whether multiple tested nodes have reached a stable state in the current cycle, as determined in step S2 above. If multiple tested nodes have reached a stable state in the current cycle, the termination of the distributed test can be determined based on this, even if there are continuously running test processes.

[0100] It should be noted that the above judgment occurs at the end of each cycle, at which point the process status information for each cycle, as well as the status information of the tested node for each cycle, have been obtained. This method of making a comprehensive judgment based on completed cycles makes full use of the acquired multi-dimensional information to achieve more reliable test completion detection.

[0101] Process 200 monitors the amount of log data from the test processes running on the node under test. The specific implementation of this process has been described in detail in steps S1 and S2 and related embodiments, and will not be repeated here. In short, process 200 determines whether the node under test has reached a stable state by comparing whether the amount of the same log data acquired at different periods is equal. When the amount of log data stops changing, it can be determined that the test activities running on the node under test have ceased.

[0102] By combining the process status information obtained from monitoring the running status of the test process with the status information of the tested node obtained from monitoring the data volume of log data, a multi-dimensional comprehensive judgment mechanism can be constructed. This multi-dimensional comprehensive judgment mechanism significantly improves the accuracy of distributed test completion detection and avoids misjudgment that may be caused by a single judgment basis. It is especially suitable for complex scenarios where the life cycle of the test process is inconsistent with the completion status of the test logic.

[0103] Furthermore, in some embodiments of this disclosure, the monitoring method 1000 for distributed testing may further include: in any period, determining that the distributed test has ended in response to the accumulated waiting time since the start of monitoring the distributed test reaching or exceeding the maximum waiting time.

[0104] Specifically, in the monitoring process of distributed testing, to prevent the monitoring process from continuing indefinitely due to anomalies such as deadlock, infinite loop, or communication interruption, this implementation introduces a timeout protection mechanism. This mechanism sets a "maximum waiting time" as the upper limit of the monitoring process time, ensuring that the monitoring process can be completed within a limited time under any circumstances, thereby avoiding long-term resource occupation.

[0105] The maximum wait time is a pre-set time threshold before starting distributed testing, used to limit the maximum duration of the entire monitoring process. This parameter can be flexibly configured according to the needs of the actual testing scenario. The specific value of the maximum wait time can be configured in various ways, such as command-line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios.

[0106] The timeout protection check occurs before monitoring the running status of the test processes on multiple nodes under test. This means that at the start of each monitoring cycle, it is first checked whether the accumulated waiting time since the start of monitoring has reached the maximum waiting time. In response to the accumulated waiting time reaching or exceeding the maximum waiting time, regardless of the current state of the test process or whether the nodes under test have reached a stable state, the distributed test can be determined to have ended, and the monitoring process terminated.

[0107] This sequential design of "checking timeouts first, then performing other monitoring" gives the timeout protection mechanism the highest priority. It can ensure that the monitoring process can still terminate in a timely manner and avoid indefinite waiting when other judgment mechanisms, such as process status detection and log stability detection, fail to trigger the termination judgment normally due to abnormal conditions.

[0108] Figure 7 This is a flowchart illustrating a monitoring method 1000 for distributed testing provided according to an exemplary embodiment of this disclosure.

[0109] like Figure 7 As shown, in one specific embodiment of this disclosure, the monitoring process of the monitoring method 1000 for distributed testing may include: Step 101: Start the distributed test. Step 102: Initialize the monitoring parameters, which may include: setting the maximum wait time, setting the predetermined number of cycles, and setting the predetermined time interval between cycles.

[0110] Step 103 involves obtaining the initial log data volume. During continuous verification across multiple cycles, for a single node under test, the log data volume obtained in the previous cycle for that node needs to be used as a reference value. The data volume obtained in the current cycle is then compared with this reference value. Based on this, the log data volume of the node under test can be obtained in the first initial cycle and used as the initial reference value for subsequent comparisons.

[0111] Step 104 enables periodic multi-dimensional monitoring, which may include: timeout protection monitoring, test process running status monitoring, and log data volume monitoring. After enabling periodic multi-dimensional monitoring, step 105 can be executed first to check timeout protection. Step 105 checking timeout protection may include: in response to the accumulated waiting time since the start of distributed test monitoring reaching or exceeding the maximum waiting time, executing step 112 to end the distributed test.

[0112] If the monitoring process does not time out, step 106, process status monitoring, can be executed. Based on the fact that the test processes running on each of the multiple tested nodes have terminated in the previous or current cycle, step 112, distributed test termination, is executed. For any test process, if it was detected as running in the previous cycle and is still running in the current cycle, it can be determined that the test process has not terminated in either cycle. Based on the fact that at least one test process has not terminated in both the previous and current cycles, step 107, stability counter initialization, is executed.

[0113] During the initialization process in step 107, the initial value of the stability counter can be set to zero. Then, step 108 is executed, in each of the multiple predetermined cycles following the current cycle, the amount of log data of the tested node is obtained, and it is determined whether the amount of log data obtained in the current cycle is equal to the amount of log data obtained in the previous cycle.

[0114] If the amount of log data acquired in the current period is equal to the amount of log data acquired in the previous period, it can be determined that the tested node has reached a stable state in the current period. If multiple tested nodes have reached a stable state, step 109, incrementing the stability counter, can be executed.

[0115] In step 110, it is determined whether the stability counter has reached or exceeded a predetermined number of counts. If the stability counter has reached or exceeded the predetermined number of counts, step 112 is executed to end the distributed test. If the stability counter has not reached the predetermined number of counts, step 111 can be executed to wait for subsequent cycles, and steps 108 to 110 are repeated.

[0116] Furthermore, during the loop from steps 108 to 110, at the beginning of each loop, the accumulated waiting time since the start of distributed testing can be determined. If the accumulated waiting time since the start of distributed testing reaches or exceeds the maximum waiting time, step 112, "End of Distributed Testing," is executed. If the accumulated waiting time since the start of distributed testing does not reach the maximum waiting time, the next loop can proceed.

[0117] Therefore, according to at least one embodiment of this disclosure, by acquiring the amount of log data of the test process running on the node under test at different periods during the distributed testing process, and determining whether the node under test has reached a stable state based on whether the amount of log data acquired at different periods is equal, it is possible to effectively monitor the activity status of multiple nodes under test in distributed testing. Using the change in the amount of log data as an observation indicator provides an objective and quantifiable basis for determining whether the node under test still has testing activities. This does not require intruding into the internal testing process to obtain status information, nor does it require the test program to actively output an end signal, and it has the characteristics of simple monitoring method and wide applicability.

[0118] Figure 8 This is a block diagram of a monitoring device 2000 for distributed testing provided according to an exemplary embodiment of the present disclosure.

[0119] like Figure 8As shown, some embodiments of this disclosure provide a monitoring device 2000 for distributed testing. The monitoring device 2000 for distributed testing may include a monitoring unit 300 and a comparison unit 400. The monitoring unit 300 is configured to periodically acquire the amount of log data of the test process running by the node under test during the execution of tests on the node under test in the distributed test; and the comparison unit 400 is configured to determine that the node under test has reached a stable state in the current period if the amount of log data acquired at a time point in the current period is equal to the amount of log data acquired at a time point in the previous period.

[0120] "During the execution of tests on the nodes under test in distributed testing, the amount of log data of the test processes run by the nodes under test is periodically obtained" can be understood as follows: Multiple periods are set during the distributed testing process, and the amount of log data of the test processes run by each of the multiple distributed nodes under test is observed within these periods. These multiple periods include at least two periods that are chronologically adjacent, which can be the previous period and the current period following the previous period. For a single node under test, it is necessary to obtain the number of bytes of log data of the test processes run by that node in the current period. Optionally, the log data can be obtained through the file access interface provided by the operating system.

[0121] The statement "If the amount of log data acquired at a given time point in the current period is equal to the amount of log data acquired at a given time point in the previous period, it is determined that the node under test has reached a stable state in the current period" can be understood as follows: For a single node under test, the amount of log data acquired in the previous period is compared with the amount of log data acquired in the current period. If they are equal, it means that in two adjacent periods in time sequence, the amount of log data corresponding to the node under test has not changed, or that the test process running on the node under test has not generated new log output during that time period, and the log data has become stable. Based on this, it can be determined that the node under test has reached a stable state in the current period. A stable state indicates that the test activities on the node under test have tended to stop, or that the test process running on the node under test no longer generates new log output.

[0122] It should be noted that equal log data volume can be understood as the number of bytes of log data acquired in two consecutive cycles being exactly the same. In other words, the difference in log data volume between two consecutive cycles is 0. Since log data volume is measured in bytes, and bytes are the smallest unit of granularity that the operating system can observe and modify, whether the log data volume is equal is a clear criterion and can serve as an objective basis for determining whether testing activities on the tested node have ceased.

[0123] Therefore, by acquiring the log data volume of the test processes running on the node under test at different periods during distributed testing, and determining whether the node under test has reached a stable state based on whether the log data volume acquired at different periods is equal, it is possible to effectively monitor the activity status of multiple nodes under test in distributed testing. Using the change in log data volume as an observation indicator provides an objective and quantifiable basis for judging whether the node under test is still conducting test activities. This does not require intruding into the internal testing process to obtain status information, nor does it require the test program to actively output a termination signal, and it has the characteristics of simple monitoring method and wide applicability.

[0124] In some embodiments of this disclosure, the monitoring device 2000 for distributed testing may further include a judgment unit (not shown), which is configured to determine the end of the distributed test based at least on the fact that multiple tested nodes in the distributed test have reached a stable state in the current period.

[0125] The monitoring device 2000 for distributed testing provided by the embodiments of this disclosure can directly or indirectly determine the end of the entire distributed test based on the stable state of the tested node determined by the cessation of changes in the amount of log data. Compared with judgment methods that rely on fixed timeouts or single process states, this can identify the end time of the distributed test more promptly and accurately, reducing the probability of misjudgments caused by delayed exits or abnormal suspensions of the test process. Furthermore, the monitoring device 2000 for distributed testing does not require waiting for a fixed timeout, thereby improving the efficiency and resource utilization of distributed testing.

[0126] In some embodiments of this disclosure, the monitoring unit 300 is further configured to acquire the amount of log data of the node under test in a plurality of cycles after the current cycle, respectively; and the comparison unit 400 is further configured to determine that the node under test has reached a stable state in one of the plurality of cycles in response to the fact that the amount of log data acquired in one of the plurality of cycles is equal to the amount of log data acquired in the previous cycle of one of the plurality of cycles.

[0127] In addition, the judgment unit is also configured to determine the end of the distributed test based on at least the fact that multiple tested nodes reach a stable state in the current period, and that multiple tested nodes reach a stable state in each of multiple periods.

[0128] In this implementation, in order to further improve the reliability of the distributed test completion judgment, a configurable parameter of "predetermined number of times" is introduced, and the stable state of the tested node is confirmed by continuous verification of multiple cycles of a predetermined number of times.

[0129] Specifically, the predetermined number of times is a pre-set number of cycles for which stable state verification needs to be performed continuously before monitoring the distributed testing process. This parameter can be adjusted according to the needs of the actual testing scenario. For example, for scenarios requiring high accuracy in determining the end of distributed testing, a larger predetermined number of times can be set to ensure that the tested node has indeed stopped generating new log output through multiple continuous verifications; for scenarios requiring high response speed in determining the end of distributed testing, a smaller predetermined number of times can be set to quickly obtain the judgment result. This configurable parameter mechanism enables the monitoring device 2000 for distributed testing disclosed herein to adapt to various distributed testing scenarios ranging from minute-level distributed testing to hour-level distributed testing, demonstrating good versatility and practicality.

[0130] The specific value of the predetermined number of times can be configured in various ways, and this disclosure does not limit this. For example, the predetermined number of times can be set through command line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios.

[0131] Multiple cycles, ordered sequentially, are set after the current cycle. Alternatively, a predetermined time interval can exist between adjacent cycles. In other words, the log data volume of multiple tested nodes can be acquired in each of the multiple cycles with predetermined time intervals following the current cycle.

[0132] Optionally, the predefined time interval is a configurable parameter that represents the time span between adjacent periods, reflecting the frequency of the monitoring process. For scenarios with high requirements for the response speed of distributed test completion judgment, a smaller predefined time interval can be set to observe changes in log data volume more frequently, thereby capturing the stable state of the logs as soon as possible after the actual completion of the distributed test and shortening the detection latency. For scenarios that are sensitive to system overhead or where the distributed test execution time is long, a larger predefined time interval can be set to reduce the additional overhead caused by frequent monitoring and avoid interfering with the test process itself.

[0133] The specific value of the predetermined time interval can be configured in various ways, and this disclosure does not limit it. For example, the predetermined time interval can be set through command line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios.

[0134] Alternatively, the predetermined number of cycles can be consecutive, or a predetermined number of cycles can be set consecutively after the current cycle, with each cycle adjacent to the previous one. Another option is that the predetermined number of cycles can be non-consecutive, as long as a predetermined number of cycles are set after the current cycle, with intervals between adjacent cycles. Furthermore, in practical applications, to ensure the real-time performance and accuracy of the stability assessment of the tested node, a continuous setting method can be adopted.

[0135] As another option, multiple cycles, a predetermined number of times, can be set within a pre-defined time window after the current cycle. For example, a 5-minute time window can be set, within which multiple cycles can be set at non-fixed or predetermined time intervals. The specific timing of the cycles can be flexibly determined according to actual needs. For example, multiple cycles can be set more intensively at the beginning of the time window to quickly respond to the end of the test, or multiple cycles can be set more intensively at the end of the time window to strengthen confirmation. Regardless of how the cycles are distributed within the time window, as long as the amount of log data of the tested node does not change within the predetermined number of consecutive cycles, the tested node can be confirmed to have reached a stable state.

[0136] Furthermore, the period setting can be correlated with the actual running status of the test process in distributed testing. For example, when a decrease in CPU utilization or reduction in network communication of a test process is detected, the monitoring interval can be automatically shortened to more precisely observe changes in the amount of log data corresponding to that test process; or when a test process is detected to be in a stable state for a long period of time, the monitoring interval can be appropriately lengthened to reduce unnecessary resource consumption.

[0137] Therefore, the implementation method of this disclosure enables continuous multiple verifications of the stable state of the node under test. Each additional verification means that the amount of log data of the node under test remains unchanged over a longer time span, thereby confirming with higher confidence that the testing activity on the node under test has indeed stopped. This continuous verification mechanism based on a predetermined number of verifications can effectively filter out accidental factors and instantaneous interference, significantly improving the robustness and reliability of the stability judgment of the node under test. By introducing a continuous period of a predetermined number of verifications and a configurable parameter mechanism, the universality and flexibility of the monitoring device 2000 used for distributed testing are improved while enhancing the accuracy of the stability judgment of the node under test, thus meeting the diverse needs of different distributed testing scenarios.

[0138] Based on this, the distributed test can be considered complete when at least multiple tested nodes reach a stable state in the current period, and also in each of multiple periods. This uses the stable state information from the current period along with the stable state information from subsequent periods as the basis for determining the end of the distributed test. Specifically, it requires that multiple tested nodes in the distributed test reach a stable state in the current period, and also reach a stable state in each of a predetermined number of periods after the current period, thereby confirming the completion of the entire distributed test with higher confidence.

[0139] In some embodiments of this disclosure, the determination unit is further configured to set a stability counter with an initial value of zero; for each of the multiple cycles, in response to the multiple tested nodes reaching a stable state, increment the count of the stability counter; and in response to the count of the stability counter reaching or exceeding a predetermined number of times, determine that the distributed test has ended.

[0140] In this implementation, the stability counter reaching or exceeding a predetermined number of counts indicates that multiple nodes under test in the distributed test have maintained a stable state for a predetermined number of consecutive periods, thus confirming the end of the distributed test. Introducing a stability counter to quantify the number of consecutive stable periods for a node under test is a simple implementation mechanism, requiring only the maintenance of a single integer variable to count the number of consecutive stable periods, eliminating the need for complex data structures or algorithms and reducing system overhead. Furthermore, the stability counter's count directly corresponds to the number of consecutive stable periods reached by the node under test; the count is updated each time a condition is met, and reaching a threshold triggers the termination decision. This logical relationship is clear and the causal relationship is explicit, facilitating real-time monitoring of test progress and enabling the output of intermediate states in automated testing processes.

[0141] In some embodiments of this disclosure, the plurality of tested nodes include at least one type node, and the determination unit is further configured to determine the end of the distributed test based on at least the fact that the plurality of tested nodes reach a stable state in the current period, the plurality of tested nodes reach a stable state in each of the plurality of periods, and the test process run by at least one type node has terminated.

[0142] In distributed testing, multiple nodes under test can exhibit various deployment modes; they can belong to the same type of node or to different types of node nodes. In other words, multiple nodes under test in distributed testing can include at least one type of node, and the classification of node types can be flexibly defined according to the test architecture / deployment mode.

[0143] By comparing the amount of log data acquired in the previous period with the amount of log data acquired in the current period, it is possible to determine whether a single node under test has reached a stable state in the current period. However, in some scenarios, log stability alone may still be uncertain. For example, a test process may not generate new logs due to deadlock, suspension, or other reasons, but it may not have exited normally, meaning the distributed test has not actually ended. To at least address this issue, the judgment unit can combine log stability with the process termination status of specific node types based on process state detection.

[0144] Specifically, based on the fact that multiple tested nodes in the distributed test have reached a stable state, it can be further determined that the test process run by at least one type of node has terminated. "Terminated" can also be understood as the test process run by that type of node no longer existing in the system, which may include the test process run by that type of node exiting normally or abnormally.

[0145] By combining log stability with process status, this implementation achieves a multi-dimensional comprehensive judgment of both log stability and process status. Compared to relying solely on log stability or solely on process status to determine the end of a distributed test, this implementation can effectively distinguish between complex situations such as "the test has ended but the test process has not exited" and "the test has not ended but the test process is abnormally suspended," significantly improving the accuracy and reliability of distributed test end determination.

[0146] The process state judgment mechanism is used in conjunction with the aforementioned judgment based on continuous stable states. For example, it can be required that multiple tested nodes in a distributed test reach a stable state for multiple consecutive periods, and that the test process of at least one type of node has terminated, as a comprehensive judgment condition for the end of the distributed test. This multi-dimensional comprehensive judgment further improves the accuracy and robustness of the distributed test termination judgment.

[0147] Furthermore, in some embodiments of this disclosure, the monitoring unit 300 is further configured to acquire the log data volume of the multiple tested nodes in the third and fourth periods after the current period, based on the fact that at least one of the tested nodes has not reached a stable state in the current period; the comparison unit 400 is further configured to determine the tested node that reaches a stable state in the fourth period based on the acquired log data volume; and the judgment unit is further configured to determine the end of the distributed test based on at least the fact that the multiple tested nodes have reached a stable state in the fourth period.

[0148] Based on this, the implementation method of this disclosure enables dynamic tracking of the completion status of distributed tests. When some tested nodes are stable while others are not, the monitoring process can automatically continue. This mechanism effectively addresses the inconsistency in execution progress among tested nodes in distributed tests, avoiding misjudgments or premature termination due to delays in individual tested nodes, and improving the robustness and adaptability of the monitoring method.

[0149] Additionally, the monitoring unit 300 may include a first sub-monitoring unit (not shown) and a second sub-monitoring unit (not shown). The first sub-monitoring unit monitors the running status of the test processes running on the node under test; the second sub-monitoring unit monitors the amount of log data of the test processes running on the node under test. By combining the process status information obtained from monitoring the running status of the test processes with the status information of the node under test obtained from monitoring the amount of log data, a multi-dimensional comprehensive judgment mechanism can be constructed, which can effectively improve the accuracy of distributed test termination judgment.

[0150] Specifically, in some embodiments of this disclosure, the monitoring unit 300 is further configured to: periodically detect the running status of the test processes run by each of the multiple tested nodes; and the judgment unit is further configured to: determine the end of the distributed test in response to detecting that the test processes run by each of the multiple tested nodes have terminated in any period.

[0151] In some other embodiments of this disclosure, the monitoring unit 300 is further configured to: periodically detect the running status of the test processes run by each of the multiple tested nodes; and the judgment unit is further configured to: in response to detecting that at least one of the multiple test processes has not terminated in any period, determine that the distributed test has ended, based at least on the fact that the multiple tested nodes have reached a stable state in the current period.

[0152] In the above comprehensive judgment process, a priority order can be set among different judgment criteria. Specifically, judgments based on process status information have higher priority: if the test processes running on multiple tested nodes in a distributed test are detected to have terminated within a certain period, the distributed test can be directly determined to be over without referring to the obtained log stability information. If process status information is insufficient to determine the end of the distributed test, the obtained status information of the tested nodes can be taken into consideration. For example, the process termination status of at least one type of node can be combined with the status information of multiple tested nodes to jointly determine whether the distributed test has ended. This priority design ensures a rapid response when the test process exits normally, while improving the accuracy of the judgment by leveraging log stability when the process has not terminated.

[0153] Alternatively, the monitoring device for distributed testing also includes a test initiation unit (not shown), which is configured to determine the end of the distributed test in any given period in response to the accumulated waiting time since the start of monitoring the distributed test reaching or exceeding a maximum waiting time.

[0154] Specifically, in the monitoring process of distributed testing, to prevent the monitoring process from continuing indefinitely due to anomalies such as deadlock, infinite loop, or communication interruption, this implementation introduces a timeout protection mechanism. This mechanism sets a "maximum waiting time" as the upper limit of the monitoring process time, ensuring that the monitoring process can be completed within a limited time under any circumstances, thereby avoiding long-term resource occupation.

[0155] The maximum wait time is a pre-set time threshold before starting distributed testing, used to limit the maximum duration of the entire monitoring process. This parameter can be flexibly configured according to the needs of the actual testing scenario. The specific value of the maximum wait time can be configured in various ways, such as command-line parameters, environment variables, configuration files, etc., to adapt to the needs of different testing scenarios.

[0156] The timeout protection check occurs before monitoring the running status of the test processes on multiple nodes under test. This means that at the start of each monitoring cycle, it is first checked whether the accumulated waiting time since the start of monitoring has reached the maximum waiting time. In response to the accumulated waiting time reaching or exceeding the maximum waiting time, regardless of the current state of the test process or whether the nodes under test have reached a stable state, the distributed test can be determined to have ended, and the monitoring process terminated.

[0157] This sequential design of "checking timeouts first, then performing other monitoring" gives the timeout protection mechanism the highest priority. It can ensure that the monitoring process can still terminate in a timely manner and avoid indefinite waiting when other judgment mechanisms, such as process status detection and log stability detection, fail to trigger the termination judgment normally due to abnormal conditions.

[0158] Therefore, according to at least one embodiment of this disclosure, by acquiring the amount of log data of the test process running on the node under test at different periods during the distributed testing process, and determining whether the node under test has reached a stable state based on whether the amount of log data acquired at different periods is equal, it is possible to effectively monitor the activity status of multiple nodes under test in distributed testing. Using the change in the amount of log data as an observation indicator provides an objective and quantifiable basis for determining whether the node under test still has testing activities. This does not require intruding into the internal testing process to obtain status information, nor does it require the test program to actively output an end signal, and it has the characteristics of simple monitoring method and wide applicability.

[0159] According to embodiments of this disclosure, this disclosure also provides an electronic device, a non-volatile computer-readable storage medium, and a computer program product.

[0160] Figure 9 A schematic block diagram of an example electronic device 3000 that can be used to implement embodiments of the present disclosure is shown. The electronic device 3000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Figure 9 As shown, the electronic device 3000 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 3000. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0162] Multiple components in electronic device 3000 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 3000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0163] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit, a graphics processing unit, various special-purpose artificial intelligence computing chips, various computing units running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various means and processes described above, such as a monitoring method for distributed testing. For example, in some embodiments, the monitoring method for distributed testing can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 3000 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the monitoring method for distributed testing described above can be performed. Alternatively, in other embodiments, the computing unit 301 can be configured to perform the monitoring method for distributed testing by any other suitable means (e.g., by means of firmware).

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These embodiments can be implemented as one or more computer programs that can execute on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] Program code for implementing the apparatus of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing method, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of this disclosure, a machine-readable medium can be a tangible medium that may include or store a program for use by or in conjunction with an instruction execution system, method, or apparatus. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, methods, or apparatuses, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display method for showing information to the user (e.g., a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display) monitor); and a keyboard and pointing method (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of methods can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0168] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0169] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0171] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0172] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0173] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A monitoring method for distributed testing, comprising: During the testing of the nodes under test in the distributed test, the amount of log data of the test processes running on the nodes under test is obtained periodically. as well as In response to the fact that the amount of log data acquired at a time point in the current period is equal to the amount of log data acquired at a time point in the previous period, it is determined that the tested node has reached a stable state in the current period.

2. The method according to claim 1, wherein, The method further includes: The distributed test is considered complete when at least one of the tested nodes in the distributed test reaches the stable state in the current period.

3. The method according to claim 2, wherein, The method further includes: In multiple cycles a predetermined number of times following the current cycle, the data volume of the log data of the tested node is obtained respectively; and In response to the fact that the amount of log data acquired in one of the plurality of periods is equal to the amount of log data acquired in the previous period of one of the plurality of periods, it is determined that the node under test has reached the stable state in one of the plurality of periods.

4. The method according to claim 3, wherein, The step of determining the end of the distributed test based on at least multiple tested nodes reaching the stable state in the current period includes: The distributed test is determined to be over if at least multiple nodes under test reach the stable state in the current period, and if multiple nodes under test reach the stable state in each of the multiple periods.

5. The method according to claim 4, wherein, The step of determining the end of the distributed test based on at least multiple tested nodes reaching the stable state in the current period, and multiple tested nodes reaching the stable state in each of the multiple periods, includes: Set the stability counter to an initial value of zero; For each of the plurality of cycles, in response to the plurality of tested nodes reaching the stable state, the count of the stability counter is incremented; and The distributed test is terminated when the stability counter reaches or exceeds the predetermined number of counts.

6. The method according to claim 4, wherein, The plurality of tested nodes include at least one type of node. The step of determining the end of the distributed test based on at least multiple tested nodes reaching the stable state in the current period, and multiple tested nodes reaching the stable state in each of the multiple periods, further includes: The distributed test is determined to be complete when multiple nodes under test reach the stable state in the current period, multiple nodes under test reach the stable state in each of the multiple periods, and the test process run by the at least one type of node has terminated.

7. The method according to claim 1, wherein, The method further includes: The running status of the test processes running on each of the multiple tested nodes is detected periodically; and In response to the detection that the test process being run by each of the multiple tested nodes has terminated in any period, the distributed test is determined to be over.

8. The method according to claim 1, wherein, The method further includes: The running status of the test processes running on each of the multiple tested nodes is detected periodically; and In response to the detection that at least one of the multiple test processes has not terminated in any period, the distributed test is determined to have ended, based at least on the fact that multiple tested nodes have reached the stable state in the current period.

9. The method according to claim 7 or 8, wherein, The method further includes: In any of the said cycles, the distributed test is determined to have ended in response to the accumulated waiting time since the start of monitoring the distributed test reaching or exceeding the maximum waiting time.

10. A monitoring device for distributed testing, comprising: The monitoring unit is configured to periodically acquire the log data of the test process running by the node under test during the testing process of the node under test in the distributed test. as well as The comparison unit is configured to determine that the node under test has reached a stable state in the current period if the amount of log data acquired at a time point in the current period is equal to the amount of log data acquired at a time point in the previous period.

11. An electronic device, comprising: A processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the monitoring method for distributed testing as described in any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the monitoring method for distributed testing as described in any one of claims 1 to 9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the monitoring method for distributed testing as described in any one of claims 1 to 9.

Images