A method for automatic test task scheduling and environment self-recovery of a board card cluster
By introducing out-of-band physical channels and a hierarchical self-healing strategy, the problems of hardware resource contention and monitoring blind spots in board server cluster testing were solved, achieving efficient fault location and automated recovery, and reducing operation and maintenance costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN XIAOSUAN TECH INFORMATION CO LTD
- Filing Date
- 2026-04-16
- Publication Date
- 2026-06-16
AI Technical Summary
Existing technologies for testing board-server clusters suffer from issues such as hardware resource contention, monitoring blind spots, and high maintenance costs, especially the inability to effectively recover when equipment crashes.
By introducing out-of-band physical channels, establishing a static topology table, performing resource pooling and mutual exclusion scheduling, constructing a zero-pollution sandbox environment, and implementing a hierarchical self-healing strategy, including L1-L3 self-healing measures, combined with heterogeneous dual-channel monitoring and temporal causal analysis.
It improves the accuracy of fault location in the event of an operating system crash, avoids hardware resource contention, reduces operation and maintenance costs, and ensures that each test is conducted in a standardized environment.
Smart Images
Figure CN122220167A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a method for scheduling automated test tasks and self-healing environment for board clusters. Background Technology
[0002] Currently, with the development of mobile cloud computing and embedded technologies, the demand for clustered testing based on board-based servers (such as ARM array servers and mobile phone farms) is increasing. Unlike traditional x86 server testing, board-based server environments have the following significant challenges: 1. Tight coupling of hardware resources: Test tasks often require exclusive use of specific physical interfaces (such as USB bandwidth, ADB channel, GPIO pins). High-density deployment can easily lead to bus bandwidth contention, resulting in unstable testing.
[0003] 2. "Monitoring blind spot" effect: Existing technologies mostly rely on the operating system of the device under test (such as Android's ADB protocol) for monitoring. However, during stress testing, if the device experiences a kernel crash or USB controller hangs, the system-level monitoring channel will also fail, making it impossible for the test system to obtain fault logs and implement effective recovery.
[0004] 3. High maintenance costs: Board devices crash frequently, and traditional recovery methods rely on manual plugging and unplugging or simple power-off, lacking refined and automated recovery strategies for different fault levels (application layer, link layer, hardware layer).
[0005] Therefore, there is still a need for an automated testing method that can sense the physical layer status of the board and has hardware-level resource scheduling and environmental self-healing capabilities.
[0006] It is evident that there is an urgent need for an automated test task scheduling and environment self-healing method for board clusters that can solve the problems of monitoring failure due to system crashes and high manual maintenance costs in existing technologies. Summary of the Invention
[0007] In view of this, the present disclosure provides a method for automated testing task scheduling and environment self-healing of board clusters, which at least partially solves some of the problems existing in the prior art.
[0008] This disclosure provides a method for automated testing task scheduling and environment self-healing for board clusters, including: Step 1, Physical Topology Mapping and Resource Pooling: Traverse the physical slots of the board server, establish a static topology relationship table, and classify each board into the available pool, maintenance pool or fault pool according to the real-time heartbeat status. The static topology relationship table includes physical board ID, USB port mapping path and out-of-band management interface. Step 2, Mutual Exclusion Scheduling Based on Profile: Receive test tasks and parse the required device feature profiles, match target boards that match the profiles in the available pool, and create an exclusive mutex lock for the physical ID of the target board in the distributed scheduling center. Step 3, out-of-band environment reset: Before test execution, the current state of the target board is forcibly verified through the out-of-band management interface, and standardized environment cleaning and dependency injection operations are performed to build a zero-pollution test sandbox environment; Step 4, Heterogeneous Dual-Channel Parallel Monitoring: During test execution, an in-band service channel and an out-of-band physical channel are established simultaneously. The in-band service channel collects the service logs of the test cases through the operating system debugging bridge, while the out-of-band physical channel independently collects the underlying kernel logs and hardware electrical parameters through the physical serial port or the baseboard management controller. Step 5, Anomaly Tracing and Hierarchical Self-Healing: When a test task interruption signal is detected, a time-series causal analysis is performed on the data collected from the in-band service channel and the out-of-band physical channel. If it is determined to be an environment-level blockage, a hierarchical self-healing strategy is triggered to restore the board state. If it is determined to be a service-level failure, a test report containing hardware snapshots is generated.
[0009] According to a specific implementation of an embodiment of this disclosure, the physical topology mapping further includes: Identify the bandwidth topology of each USB node within the board server to avoid scheduling high I / O throughput test tasks to adjacent board nodes sharing the same USB bus bandwidth during scheduling.
[0010] According to a specific implementation of this disclosure, the step of performing time-series causal analysis on the data collected from the in-band service channel and the out-of-band physical channel includes: Using the moment the test task was interrupted as the reference point, the data collected by the out-of-band physical channel within the preset time window was traced back. If a kernel crash keyword or serial port heartbeat loss is detected before the interruption time, it is determined to be an environment-level blockade; If the out-of-band physical channel data is normal but the service process exits abnormally, it is determined as a service-level failure.
[0011] According to one specific implementation of this disclosure, the hierarchical self-healing strategy includes the following levels executed sequentially: L1-level soft reset: When an application layer freeze is detected, process killing and cache clearing are performed through the in-band service channel; L2 port reset: When a USB communication link failure is detected but the system is still online, reset the power supply to the physical port of the USB hub to which the target board is connected; L3 cold boot: When a system-level crash or kernel failure is detected, the physical power supply to the target board is cut off and the board is powered on again via a relay or intelligent power distribution unit.
[0012] According to a specific implementation of an embodiment of this disclosure, step 5 further includes dynamic reputation management: A dynamically updated health reputation score is maintained for each board node; when the frequency or level of triggering the graded self-healing strategy of a certain board node exceeds a preset threshold, it is automatically migrated from the available pool to the fault pool, and a hardware maintenance notification is generated.
[0013] According to a specific implementation of an embodiment of this disclosure, the environmental cleaning step includes: Mount the board's file system and compare the current file system snapshot with the baseline image; Retain necessary files according to the preset whitelist, and delete temporary files and logs that are not on the whitelist; If the system partition is detected to have been tampered with, a command is sent through the out-of-band management interface to restore the system read-only partition to its factory settings.
[0014] The automated test task scheduling and environment self-healing scheme for board clusters in this embodiment includes: Step 1, physical topology mapping and resource pooling: traversing the physical slots of the board server, establishing a static topology relationship table, and assigning each board to an available pool, maintenance pool, or fault pool according to the real-time heartbeat status, wherein the static topology relationship table includes physical board ID, USB port mapping path, and out-of-band management interface; Step 2, profile-based mutual exclusion scheduling: receiving test tasks and parsing the required device feature profiles, matching target boards that match the profiles in the available pool, and creating an exclusive mutual exclusion lock for the physical ID of the target board in the distributed scheduling center; Step 3, out-of-band environment reset: before test execution, forcibly verifying the target board's current environment through the out-of-band management interface. The process begins with a pre-test state, followed by standardized environment cleaning and dependency injection to build a zero-pollution test sandbox environment. Step 4 involves heterogeneous dual-channel parallel monitoring: During test execution, an in-band service channel and an out-of-band physical channel are simultaneously established. The in-band service channel collects test case service logs via the operating system debug bridge, while the out-of-band physical channel independently collects underlying kernel logs and hardware electrical parameters via a physical serial port or baseboard management controller. Step 5 involves anomaly tracing and tiered self-healing: When a test task interruption signal is detected, a time-series causal analysis is performed on the data collected by the in-band service channel and the out-of-band physical channel. If the interruption is determined to be environmental-level, a tiered self-healing strategy is triggered to restore the board state. If the interruption is determined to be service-level, a test report containing hardware snapshots is generated.
[0015] The beneficial effects of the embodiments disclosed herein are as follows: 1. Breaking through monitoring blind spots: By introducing an "out-of-band physical channel," kernel logs can still be obtained via serial port or BMC even when the operating system crashes, greatly improving the accuracy of fault location; 2. Improve resource utilization: Mutual exclusion scheduling based on physical topology and USB bandwidth effectively avoids hardware resource contention between high-density boards; 3. Achieve unattended operation: Through the L1-L3 hierarchical self-healing strategy, it can automatically handle more than 95% of equipment failures, significantly reducing manual operation and maintenance costs; 4. High environmental purity: Forced environment reset based on out-of-band management ensures that each test runs in a standardized "sandbox", avoiding dirty data residue. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 A flowchart illustrating an automated test task scheduling and environment self-healing method for board clusters provided in this embodiment of the disclosure; Figure 2 This is a schematic diagram illustrating the specific implementation process of a method for scheduling automated testing tasks and self-healing environment for board clusters provided in this embodiment of the disclosure; Figure 3 This is a schematic diagram of a heterogeneous dual-channel monitoring architecture provided in an embodiment of the present disclosure; Figure 4 A logic flowchart of an L1-L3 hierarchical self-healing strategy provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of a collision prevention scheduling process based on USB bandwidth topology provided in an embodiment of this disclosure. Detailed Implementation
[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0019] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0020] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0021] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0022] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0023] This disclosure provides a method for automated testing task scheduling and environment self-healing for board clusters. The method can be applied to the board server cluster testing process in scenarios such as mobile cloud computing or embedded technology.
[0024] See Figure 1 This is a flowchart illustrating an automated test task scheduling and environment self-healing method for board clusters provided in this embodiment of the disclosure. Figure 1 and Figure 2 As shown, the method mainly includes the following steps: Step 1, Physical Topology Mapping and Resource Pooling: Traverse the physical slots of the board server, establish a static topology relationship table, and classify each board into the available pool, maintenance pool or fault pool according to the real-time heartbeat status. The static topology relationship table includes physical board ID, USB port mapping path and out-of-band management interface. Step 2, Mutual Exclusion Scheduling Based on Profile: Receive test tasks and parse the required device feature profiles, match target boards that match the profiles in the available pool, and create an exclusive mutex lock for the physical ID of the target board in the distributed scheduling center. Step 3, out-of-band environment reset: Before test execution, the current state of the target board is forcibly verified through the out-of-band management interface, and standardized environment cleaning and dependency injection operations are performed to build a zero-pollution test sandbox environment; Step 4, Heterogeneous Dual-Channel Parallel Monitoring: During test execution, an in-band service channel and an out-of-band physical channel are established simultaneously. The in-band service channel collects the service logs of the test cases through the operating system debugging bridge, while the out-of-band physical channel independently collects the underlying kernel logs and hardware electrical parameters through the physical serial port or the baseboard management controller. Step 5, Anomaly Tracing and Hierarchical Self-Healing: When a test task interruption signal is detected, a time-series causal analysis is performed on the data collected from the in-band service channel and the out-of-band physical channel. If it is determined to be an environment-level blockage, a hierarchical self-healing strategy is triggered to restore the board state. If it is determined to be a service-level failure, a test report containing hardware snapshots is generated.
[0025] The specific implementation includes the following steps: S1. Physical topology mapping and resource pooling: Traverse the physical slots of the board server, establish a static topology relationship table of physical board ID, USB port mapping path and out-of-band management interface (serial port / power control port), and divide the boards into available pool, maintenance pool and fault pool according to real-time heartbeat. S2. Profile-based Mutual Exclusion Scheduling: Receive test tasks, parse the device characteristics (profiles) required by the tasks, match the target board in the available pool, and create an exclusive mutex lock for the physical ID of the board in the distributed scheduling center. S3, Out-of-band Environment Reset: Before executing the test, the board status is forcibly verified through the out-of-band management interface, and standardized environment cleaning and dependency injection are performed to build a zero-pollution test sandbox. S4, Heterogeneous Dual-Channel Parallel Monitoring: The specific process is as follows Figure 3 As shown, during test execution, in-band service channels and out-of-band physical channels are established: In-band business channel: Collect business logs of test cases through the operating system debugging bridge (ADB / SSH); Out-of-band physical channel: Independently collect low-level kernel logs and hardware electrical parameters via physical serial port (UART) or baseboard management controller (BMC); S5. Anomaly Tracing and Hierarchical Self-Healing: When an interruption signal is detected, time-series causal analysis is performed on in-band and out-of-band data; if it is determined to be an environment-level blockage, a hierarchical self-healing strategy is triggered to restore the board state; if it is determined to be a service-level failure, a test report containing hardware snapshots is generated. Based on the above methods, further measures include: Regarding the timing causal analysis: Taking the interruption time of the test task as the benchmark, backtrack the out-of-band physical channel data within the preset time window; if the kernel panic keyword or serial port heartbeat loss is detected before the interruption time, it is determined to be an environment-level blockage; if the out-of-band channel is normal but the business process exits, it is determined to be a business-level failure.
[0026] Regarding the tiered self-healing strategy: it includes L1 level (soft reset): for application-level freezes, it executes process killing; L2 level (port reset): for USB packet loss, it resets the power supply to the USB hub port; L3 level (cold boot): for system crashes, it cuts off the physical power supply through a relay and then re-energizes the system.
[0027] Regarding bandwidth topology: Identify the bandwidth topology of each USB node in the board server to avoid scheduling high I / O throughput test tasks to adjacent board nodes that share the same USB bus bandwidth at the same time.
[0028] The automated test task scheduling and environment self-healing method for board clusters provided in this embodiment introduces an "out-of-band physical channel," which can still obtain kernel logs via serial port or BMC even when the operating system crashes, greatly improving the accuracy of fault location. Mutual exclusion scheduling based on physical topology and USB bandwidth effectively avoids hardware resource contention between high-density boards. Through the L1-L3 hierarchical self-healing strategy, it can automatically handle more than 95% of device faults, significantly reducing manual maintenance costs. Forced environment reset based on out-of-band management ensures that each test runs in a standardized "sandbox," avoiding dirty data residue.
[0029] The method of this disclosure will be further described below with reference to several specific embodiments.
[0030] Example 1: Implementation of Heterogeneous Dual-Channel Monitoring In this embodiment, the test platform maintains two independent connection handles for each board node.
[0031] Channel A (in-band): Uses ADB Client to connect to the board's TCP / IP port, responsible for sending am instrument commands and sending back Logcat logs.
[0032] Channel B (out-of-band): Use Serial Client to connect to the physical serial port ( / dev / ttyUSBx) of the board.
[0033] During testing, if channel A suddenly disconnects (throws a Connection Refused exception), the system immediately reads data from channel B for 10 seconds before and after the disconnection. If channel B prints "Out of memory: Kill process" or "Kernel panic," the system automatically determines the failure as an "environmental failure" and triggers a self-healing process, rather than falsely reporting it as a code bug.
[0034] Example 2: L1-L3 graded self-healing strategy like Figure 4 As shown in the figure, this embodiment illustrates the hierarchical processing logic in step S5: 1. Monitoring: The watchdog polls the board status once per second.
[0035] 2. L1 Trigger: If the test app is detected to be unresponsive (ANR), but the system UI is still responsive, send the adb shellam force-stop command, clear the cache, and retry the task.
[0036] 3. L2 Trigger: If ADB completely disconnects but the serial port still outputs a heartbeat, it indicates that the USB link is dead. The system sends a command to the USB Hub to cut off the VBUS power supply to that port for 500ms and then restore it, forcing the USB bus to re-enumerate.
[0037] 4. L3 Trigger: If there is no heartbeat output from the serial port (complete system crash), the system sends a command to the rack PDU to cut off the AC / DC power supply corresponding to the board, waits 5 seconds and then powers on again, and listens to the U-Boot startup log of the serial port. The system releases the resource lock only after confirming that it has fully started.
[0038] Example 3: Collision-Averse Scheduling Based on USB Bandwidth Topology like Figure 5 As shown, in step S1, the system constructs a USB tree topology. Assume that board A and board B are physically connected to the same USB 3.0 hub chip. When board A is performing a high I / O task (using >80% bandwidth) such as "full image flash," the scheduler automatically locks board B, preventing it from being assigned high-bandwidth tasks such as "automatic testing of high-definition video streams," until board A releases its bandwidth resources. This effectively avoids data packet loss and test timeouts caused by bandwidth constraints.
[0039] Example 4: Dynamic Management of Reputation The system maintains an initial reputation score (e.g., 100 points) for each board node.
[0040] Each successful task completion earns +1 reputation point.
[0041] Each time L2 level self-healing (USB reset) is triggered, reputation score is reduced by 5.
[0042] Each time L3 level self-healing (cold start) is triggered, the reputation score is reduced by 20.
[0043] When the credit score of a certain board falls below a preset threshold (e.g., 60 points), the system automatically migrates the board from the "available pool" to the "fault pool", stops assigning new tasks to it, and sends a maintenance work order containing serial port logs to the hardware maintenance personnel via email or DingTalk, indicating that there may be hardware aging or poor contact.
[0044] Example 5: Deep Environmental Cleaning In step S3, in order to construct a zero-pollution sandbox, the system performs the following operations: 1. File comparison: Mount the board's file system and compare the differences between the current / data partition and the baseline image.
[0045] 2. Whitelist Filtering: Based on the preset whitelist (such as system configuration files, preset account data), retain necessary files and delete all other non-whitelisted temporary files and generated logs.
[0046] 3. Low-level reset: If the system partition is detected to have been tampered with, a Bootloader command is sent via serial port to forcibly restore the system read-only partition to factory settings, ensuring the consistency of the test environment.
[0047] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.
[0048] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.
Claims
1. A method for automated test task scheduling and environment self-healing for board clusters, characterized in that, include: Step 1, Physical Topology Mapping and Resource Pooling: Traverse the physical slots of the board server, establish a static topology relationship table, and classify each board into the available pool, maintenance pool or fault pool according to the real-time heartbeat status. The static topology relationship table includes physical board ID, USB port mapping path and out-of-band management interface. Step 2, Mutual Exclusion Scheduling Based on Profile: Receive test tasks and parse the required device feature profiles, match target boards that match the profiles in the available pool, and create an exclusive mutex lock for the physical ID of the target board in the distributed scheduling center. Step 3, out-of-band environment reset: Before test execution, the current state of the target board is forcibly verified through the out-of-band management interface, and standardized environment cleaning and dependency injection operations are performed to build a zero-pollution test sandbox environment; Step 4, Heterogeneous Dual-Channel Parallel Monitoring: During test execution, an in-band service channel and an out-of-band physical channel are established simultaneously. The in-band service channel collects the service logs of the test cases through the operating system debugging bridge, while the out-of-band physical channel independently collects the underlying kernel logs and hardware electrical parameters through the physical serial port or the baseboard management controller. Step 5, Anomaly Tracing and Hierarchical Self-Healing: When a test task interruption signal is detected, a time-series causal analysis is performed on the data collected from the in-band service channel and the out-of-band physical channel. If it is determined to be an environment-level blockage, a hierarchical self-healing strategy is triggered to restore the board state. If it is determined to be a service-level failure, a test report containing hardware snapshots is generated.
2. The method according to claim 1, characterized in that, The physical topology mapping also includes: Identify the bandwidth topology of each USB node within the board server to avoid scheduling high I / O throughput test tasks to adjacent board nodes sharing the same USB bus bandwidth during scheduling.
3. The method according to claim 1, characterized in that, The steps for performing time-series causal analysis on data collected from in-band service channels and out-of-band physical channels include: Using the moment the test task was interrupted as the reference point, the data collected by the out-of-band physical channel within the preset time window was traced back. If a kernel crash keyword or serial port heartbeat loss is detected before the interruption time, it is determined to be an environment-level blockade; If the out-of-band physical channel data is normal but the service process exits abnormally, it is determined as a service-level failure.
4. The method according to claim 1, characterized in that, The hierarchical self-healing strategy includes the following levels executed sequentially: L1-level soft reset: When an application layer freeze is detected, process killing and cache clearing are performed through the in-band service channel; L2 port reset: When a USB communication link failure is detected but the system is still online, reset the power supply to the physical port of the USB hub to which the target board is connected; L3 cold boot: When a system-level crash or kernel failure is detected, the physical power supply to the target board is cut off and the board is powered on again via a relay or intelligent power distribution unit.
5. The method according to claim 1, characterized in that, Step 5 also includes dynamic credit management: A dynamically updated health reputation score is maintained for each board node; when the frequency or level of triggering the graded self-healing strategy of a certain board node exceeds a preset threshold, it is automatically migrated from the available pool to the fault pool, and a hardware maintenance notification is generated.
6. The method according to claim 1, characterized in that, The environmental cleaning steps include: Mount the board's file system and compare the current file system snapshot with the baseline image; Retain necessary files according to the preset whitelist, and delete temporary files and logs that are not on the whitelist; If the system partition is detected to have been tampered with, a command is sent through the out-of-band management interface to restore the system read-only partition to its factory settings.