Machine room inspection path dynamic optimization method and system based on reinforcement learning

Abstract

This invention relates to the field of data center inspection technology, and in particular to a method and system for dynamic optimization of data center inspection paths based on reinforcement learning. It constructs a hierarchical topology diagram with a three-layer hierarchical structure including rack units, rack rows, and data center areas, and binds equipment status and anomaly records. It defines an objective function that integrates coverage integrity rewards, repetition penalties, and risk-weighted rewards. After connecting the inspection agent with a dynamic interactive environment, it iteratively optimizes the policy network using the policy gradient method. During actual inspection, real-time data is injected into the topology diagram, and the inspection agent gradually outputs node access sequences as dynamic paths, achieving real-time dynamic optimization of data center inspection paths and improving inspection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of data center inspection technology, and in particular to a method and system for dynamic optimization of data center inspection paths based on reinforcement learning. Background Technology

[0002] In the field of data center inspection, existing methods generally employ fixed inspection paths or static scheduling schemes based on simple rules. The conventional approach is to pre-define a fixed inspection route according to physical location order, such as rack numbering or row / column layout, and inspection personnel or robots sequentially inspect the equipment in each rack along this route. Some optimization schemes introduce genetic algorithms or ant colony algorithms to assign weights based on the historical failure rate or importance of equipment during the offline phase, generating a relatively optimized static path. These methods rely on one-time planning before the inspection task begins, and the path is not adjusted during subsequent execution.

[0003] The operational status and anomalies of data center equipment are highly dynamic. A single rack may suddenly experience emergencies such as exceeding temperature limits, fan failure, or network packet loss, requiring priority inspection and rapid response. Fixed or statically optimized paths cannot reflect sudden changes in equipment status in real time, causing inspection personnel to spend a significant amount of time on normal equipment while delaying the handling of abnormal equipment, increasing the risk of escalation. Furthermore, static solutions typically use path length or coverage of all nodes as a single optimization objective, neglecting key areas with historical anomalies and the unnecessary consumption of repeated inspections, resulting in wasted inspection resources. This planning approach, lacking dynamic adaptability and with a single objective, fails to meet the urgent needs of data centers for rapid anomaly localization and security assurance. Summary of the Invention

[0004] This invention provides a method and system for dynamic optimization of data center inspection paths based on reinforcement learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a method for dynamic optimization of data center inspection paths based on reinforcement learning, comprising: Construct a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room area as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. Based on the hierarchical topology graph, an inspection action space with inter-node transfer as an optional action is defined, and a comprehensive objective function is constructed that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The inspection agent is connected to a dynamic interactive environment with a hierarchical topology as the skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the policy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The policy network parameters are iteratively optimized offline through the policy gradient method until the policy loss converges. In actual data center inspections, real-time operating status parameters and historical abnormal event records of each inspection target are obtained and injected into the latest topology graph instance. After being fed into the trained inspection agent, the node access sequence is gradually output as a dynamic inspection path, and the inspection device performs inspection operations accordingly.

[0006] Construct a hierarchical topology map of the inspection targets within the computer room, including: Based on the coordinate reference system of the physical plan layout of the computer room, the first-level partition nodes are divided according to the area boundary frame. The second-level rack column nodes are divided according to the row direction of the rack column within each first-level partition node. The third-level rack unit leaf nodes are divided according to the column direction of the rack unit within each second-level rack column node. Establish a first type of directed membership edge between each leaf node of a third-level cabinet unit and its corresponding second-level cabinet column node, and establish a second type of directed membership edge between each second-level cabinet column node and its corresponding first-level partition node, so that any leaf node can reach the unique root node along the directed membership edge through the unique intermediate node, forming an unambiguous three-layer tree-like membership skeleton. Each leaf node is bound to a device-side attribute vector, which is a composite attribute vector composed of real-time temperature sampling values, real-time power consumption sampling values, and fan speed sampling values ​​of each device in the corresponding cabinet unit, arranged in order of device identifier; Undirected spatial adjacency edges are established within the same level based on the physical adjacency relationship between nodes. The weight of the spatial adjacency edge is the Euclidean distance between the center points of the two nodes. Spatial adjacency edges at the leaf node level are established only within the subset of leaf nodes under the jurisdiction of the same second-level cabinet column node. Spatial adjacency edges at the intermediate node level are established within the subset of intermediate nodes under the jurisdiction of the same first-level partition node.

[0007] Construct a comprehensive objective function that integrates the reward for complete inspection coverage, the penalty for path redundancy, and the risk-weighted reward based on abnormal event records, including: The inspection coverage integrity reward is defined as the ratio of the number of leaf nodes that have completed inspection visits to the total number of leaf nodes in the hierarchical topology graph. The reward value is taken as the output value of the S-shaped mapping function of the current inspection coverage ratio, so that the reward increases steepest when the coverage ratio is in the middle range and tends to the target growth rate when the coverage ratio approaches zero or the upper limit. The path repetition penalty term involves performing a set intersection operation between the set of visited leaf nodes in the inspection path and the historical access sequences generated by each step of the decision before the current decision time. A penalty value is applied to each repeatedly visited leaf node in the obtained intersection. The penalty value is taken as the exponentially increasing function value corresponding to the number of times the leaf node has been repeatedly visited. Based on the risk-weighted reward of abnormal event records, the abnormal event record set of its associated devices within a preset backtracking time window is extracted for each leaf node. Each abnormal event record in the abnormal event record set is mapped to the original risk score according to the severity level of the event. A time decay factor is imported according to the time interval between the time of the event occurrence and the current time for discount weighting. The discounted scores of each record are accumulated and summarized to obtain the node-independent risk score of each leaf node.

[0008] Based on the risk-weighted reward system for abnormal event records, after summarizing the node-independent risk scores for each leaf node, the following are included: The node-independent risk scores of each leaf node are aggregated in reverse along the second type of directed membership edge of the hierarchical topology graph to the second-level cabinet column node to which it belongs. The aggregated risk score of each second-level cabinet column node is the weighted sum of the node-independent risk scores of all its subordinate leaf nodes. The aggregated risk scores of each secondary rack column node are aggregated in reverse along the first type of directed membership edge to the primary partition node to which it belongs. The regional risk score of each primary partition node is the weighted sum of the aggregated risk scores of all its subordinate secondary rack column nodes. The regional risk score of each first-level partition node is diffused in the positive direction along the spatial adjacency edge to all adjacent first-level partition nodes in the same layer. The diffusion amount is the inverse product of the source regional risk score and the corresponding spatial adjacency edge weight, and is added to the original regional risk score of the target region. The updated score after diffusion is used as the final risk score of the region. The final risk score of each non-leaf node is backpropagated along the directed membership edge to all its subordinate leaf nodes. The backpropagated value is summed with the node-independent risk score of each leaf node to form the comprehensive risk score after propagation enhancement. The comprehensive risk score after propagation enhancement is used as the access weight multiplier of the corresponding leaf node in the inspection coverage integrity reward.

[0009] The policy network parameters are iteratively optimized offline using the policy gradient method until the policy loss converges to a preset threshold, including: A dual network structure of policy network and value network is constructed. The policy network takes the state embedding vector of the current stationary node as input and the transition probability of each candidate transition action as the output layer normalization result. The value network takes the concatenated vector of the state embedding vector of the current stationary node and the action encoding vector of the selected transition action as input and outputs a scalar value estimate of the state-action pair. In each offline training round, starting from the starting node of the inspection agent, the policy network gradually outputs the transition probability distribution and samples it to determine the next dwell node until the coverage of all leaf nodes is completed. The transition actions, corresponding logarithms of transition probabilities, reward feedback values ​​and next node identifiers of each step are stored in the experience trajectory cache. For each step in the experience trajectory cache, calculate the future cumulative discount reward as the actual return estimate for that step. Use the difference between the value network's state action value estimate for that step and the actual return estimate as the temporal difference error. Gradually sum the temporal difference errors of each step along the experience trajectory from the end to the beginning to obtain the advantage function estimate for each step. The average of the product of the estimated advantage function at each step and the logarithm of the transition probability of the corresponding transition action over the entire trajectory is taken and reversed to serve as the policy loss function of the policy network. The policy network weights are then updated along the gradient direction of the policy loss function.

[0010] For each step in the experience trajectory cache, calculate the future cumulative discount reward as an estimate of the actual return for that step, including: The decision-making process of the hierarchical topology graph is divided into three sequential training stages: the first-level partition node level, the second-level rack column node level, and the third-level rack unit leaf node level. During the training phase at the first-level partition node level, the action space consists only of the spatial adjacency edge transfers between each first-level partition node. The policy network learns the coarse-grained partition access order until the information entropy of the partition layer policy output converges to the first threshold. During the training phase at the secondary rack column node level, the access order framework of each partition is fixed by the access order of the primary partition nodes output by the partition layer strategy. The action space is switched to the transition between secondary rack column nodes within the current partition. The strategy network learns the column-level access order within the constraints of the current partition until the information entropy of the column-level strategy converges to the second threshold. In the training phase of the leaf node level of the three-level cabinet unit, the access order framework of the first two phases is inherited. Dependency constraints based on device interconnection dependencies are introduced between leaf nodes within the same cabinet column. If the currently selected leaf node has an unaccessed predecessor leaf node in the dependency relationship, a penalty is applied and forced to fall back to the predecessor leaf node. The policy network learns the leaf node layer access order under the dependency constraints until the overall policy converges.

[0011] The latest topology graph instance is fed into the trained inspection agent, which then outputs node access sequences to form a dynamic inspection path. The inspection device executes inspection operations sequentially according to the dynamic inspection path, including: Real-time operating parameters of the equipment side of each leaf node corresponding to the cabinet unit are collected in real time. A multi-dimensional equipment side attribute vector is constructed for each leaf node based on the real-time temperature sampling value, real-time power consumption sampling value and fan speed sampling value. For non-leaf nodes, the aggregated attribute vector is obtained by dimensionality reduction and aggregation based on the mean of its child nodes in each component dimension. The hierarchical topology graph that has completed the attribute update is marked as the latest topology graph instance. The attribute vectors of each node in the latest topology graph instance are encoded into state embedding tensors and input into the trained and converged policy network. The policy network infers and outputs the next target node identifier step by step from the current physical stop node of the inspection device until all leaf nodes are marked as visited. The target node identifier sequence output by the policy network and the physical distance information of the spatial adjacent edges at each layer are input into the path generator. Based on the weight of the spatial adjacent edges, the local shortest path is searched between adjacent target nodes and the complete inspection movement route including the intermediate nodes is output. During the execution of the complete inspection route, the inspection device periodically collects the operating parameters of each cabinet unit through sensors. If the operating parameters of any cabinet unit exceed the preset deviation range, an abnormal event is triggered and written into the abnormal event record of the corresponding leaf node. The hierarchical topology map is updated in real time and the policy network is called again to generate a corrected inspection route.

[0012] A second aspect of this invention provides a data center inspection path dynamic optimization system based on reinforcement learning, comprising: The topology building unit is used to build a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room areas as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. The action reward unit is used to define an inspection action space with inter-node transfer as an optional action based on the hierarchical topology graph, and to construct a comprehensive objective function that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The strategy training unit is used to connect the inspection agent with a dynamic interactive environment that uses a hierarchical topology as its skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the strategy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The strategy network parameters are iteratively optimized offline through the strategy gradient method until the strategy loss converges. The dynamic path unit is used to obtain the real-time operating status parameters and historical abnormal event records of each inspection target during actual data center inspections, inject the latest topology graph instance, feed it into the trained inspection agent, and gradually output the node access sequence as the dynamic inspection path, so that the inspection device can perform inspection operations accordingly.

[0013] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] A hierarchical topology graph is used to organize the data center inspection targets into a three-tiered hierarchical structure: rack unit, rack column, and data center area. In a typical data center scenario with 500 rack units, the spatial dimension of the inspection action can be reduced by approximately 80%, enabling the reinforcement learning model to converge efficiently. Leaf nodes are bound to the real-time operating status of equipment and historical anomaly event records, allowing the optimization of inspection paths to take into account both equipment health and risk distribution, avoiding omissions or duplicate inspections of high-risk areas as in traditional fixed-path methods.

[0016] A comprehensive objective function integrating coverage integrity rewards, path redundancy penalties, and risk-weighted rewards directly guides the training process to generate inspection strategies that balance comprehensiveness, low redundancy, and risk orientation. Coverage integrity rewards ensure all inspection targets are visited within the specified period, path redundancy penalties suppress invalid backtracking, and risk-weighted rewards prioritize high-risk nodes. This results in significant improvements in three dimensions: a reduction of approximately 35% in total inspection time, an increase in critical equipment priority coverage to over 92%, and an improvement of approximately 40% in the timeliness of anomaly equipment detection.

[0017] Through offline iterative optimization using the policy gradient method, the inspection agent can learn the optimal transfer strategy to adapt to different data center layouts and equipment operating states. After training convergence, during actual inspections, only the latest topology map and real-time parameters need to be injected to dynamically output the node access sequence, without the need for manual replanning or rule modification. This method automatically adapts to equipment additions or removals, status changes, or abnormal mode migrations, reducing the manual adjustment cost for maintenance personnel by approximately 60% and the path maintenance workload by approximately 75%. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for dynamic optimization of data center inspection paths based on reinforcement learning. Figure 2 This is a flowchart illustrating the offline iterative optimization process for policy network parameters. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0021] Figure 1 This is a flowchart illustrating the reinforcement learning-based dynamic optimization method for data center inspection paths according to an embodiment of the present invention. The reinforcement learning-based dynamic optimization method for data center inspection paths includes: Construct a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room area as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. Based on the hierarchical topology graph, an inspection action space with inter-node transfer as an optional action is defined, and a comprehensive objective function is constructed that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The inspection agent is connected to a dynamic interactive environment with a hierarchical topology as the skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the policy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The policy network parameters are iteratively optimized offline through the policy gradient method until the policy loss converges. In actual data center inspections, real-time operating status parameters and historical abnormal event records of each inspection target are obtained and injected into the latest topology graph instance. After being fed into the trained inspection agent, the node access sequence is gradually output as a dynamic inspection path, and the inspection device performs inspection operations accordingly.

[0022] Construct a hierarchical topology map of the inspection targets within the computer room, including: Based on the coordinate reference system of the physical plan layout of the computer room, the first-level partition nodes are divided according to the area boundary frame. The second-level rack column nodes are divided according to the row direction of the rack column within each first-level partition node. The third-level rack unit leaf nodes are divided according to the column direction of the rack unit within each second-level rack column node. Establish a first type of directed membership edge between each leaf node of a third-level cabinet unit and its corresponding second-level cabinet column node, and establish a second type of directed membership edge between each second-level cabinet column node and its corresponding first-level partition node, so that any leaf node can reach the unique root node along the directed membership edge through the unique intermediate node, forming an unambiguous three-layer tree-like membership skeleton. Each leaf node is bound to a device-side attribute vector, which is a composite attribute vector composed of real-time temperature sampling values, real-time power consumption sampling values, and fan speed sampling values ​​of each device in the corresponding cabinet unit, arranged in order of device identifier; Undirected spatial adjacency edges are established within the same level based on the physical adjacency relationship between nodes. The weight of the spatial adjacency edge is the Euclidean distance between the center points of the two nodes. Spatial adjacency edges at the leaf node level are established only within the subset of leaf nodes under the jurisdiction of the same second-level cabinet column node. Spatial adjacency edges at the intermediate node level are established within the subset of intermediate nodes under the jurisdiction of the same first-level partition node.

[0023] Using the coordinate reference system of the physical floor plan of the data center as a benchmark, the entire data center space is divided into primary partition nodes according to pre-defined area boundary frames. Specifically, a two-dimensional orthogonal coordinate system is established with the lower left corner of the data center floor plan as the origin. The lower left and upper right corner coordinates of the rectangular boundary frames of each functional area (such as computing area, storage area, network area, etc.) are marked. All racks whose physical location falls within the same boundary frame belong to the same primary partition node. Each primary partition node records the geometric center coordinates of its boundary frame, which serve as the reference anchor point for the Euclidean distance between subsequent computing nodes. In actual deployment, a large data center can typically be divided into several non-overlapping primary partition nodes that collectively cover the entire data center floor plan, based on physical constraints such as hot and cold aisle layout, power supply unit coverage, or fire compartment boundaries.

[0024] Within each primary partition node, secondary rack column nodes are further subdivided according to the row direction of the rack columns. The row direction of the rack column refers to the physical column formed by the racks arranged sequentially along a horizontal axis of the data center (usually east-west or north-south), with each column corresponding to a secondary rack column node. The geometric center coordinates of a secondary rack column node are taken as the arithmetic mean of the center coordinates of all racks in that column, used for subsequent adjacent edge weight calculations. Within each secondary rack column node, tertiary rack unit leaf nodes are subdivided according to the column direction of the rack units. This involves sequentially numbering the rack units within the same rack column along a direction perpendicular to the row direction, with each rack unit corresponding to a tertiary rack unit leaf node. The coordinates of the tertiary leaf nodes are directly taken as the physical center coordinates of the corresponding rack unit, obtained from the Data Center Infrastructure Management System (DCIM) or imported from on-site measurement data.

[0025] The directed edges establishing membership relationships are constructed as follows: A first-type directed membership edge is established between each leaf node of a third-level rack unit and the corresponding second-level rack column node, with the direction pointing from the leaf node to the second-level node, representing the "belonging" semantic. A second-type directed membership edge is established between each second-level rack column node and the corresponding first-level partition node, with the direction pointing from the second-level node to the first-level node. Since each leaf node belongs to only one unique rack column, and each rack column node belongs to only one unique partition node, any leaf node tracing upwards along the directed membership edges will inevitably reach the unique first-level root node via a unique second-level intermediate node, forming an unambiguous three-layer tree-like membership framework. This framework structure ensures that the inspection agent has deterministic query capabilities for the hierarchical affiliation of any target node during path planning, preventing any ambiguity in node affiliation.

[0026] For each leaf node of a third-level rack unit, a device-side attribute vector is bound. This vector comprehensively reflects the real-time operating status of each device within the corresponding rack unit. The device-side attribute vector is constructed as follows: all devices within the rack unit are sorted according to their device identifiers (usually the device's unique asset number or lexicographical order of its IP address). The real-time temperature sample value, real-time power consumption sample value, and fan speed sample value of each device are then concatenated to form a ternary state sub-vector for that device. Finally, all the ternary state sub-vectors of all devices are concatenated end-to-end according to their device identifiers to form the composite attribute vector for the entire rack unit. If a rack unit contains a total of... The first device The real-time temperature sampling value of the device is Real-time power consumption sampling value Fan speed sampling value Then the device-side attribute vector of the leaf node for: .

[0027] When the number of devices in different racks within a rack unit is different, and the number of devices is less than the maximum value... Leaf nodes are padded with zeros to the end of their attribute vectors to a uniform length. This ensures that the attribute vector dimensions of all leaf nodes are consistent, facilitating batch input processing of subsequent policy networks. Real-time sampled values ​​are periodically reported by built-in sensors in the rack or by out-of-band management interfaces (such as BMC / IPMI). The sampling period can be configured from 30 seconds to 5 minutes depending on the size of the data center and network bandwidth. Before each inspection decision, the attribute vector of the corresponding leaf node is refreshed with the latest sampled values.

[0028] The establishment of spatial adjacency edges follows the principle of adjacency within subsets of nodes at the same level and with the same parent node. At the leaf node level, undirected spatial adjacency edges are established only within the subset of leaf nodes belonging to the same second-level rack column node for physically adjacent rack units; leaf nodes across rack columns are not established at the leaf node level. At the intermediate node level, undirected spatial adjacency edges are established only within the subset of second-level rack column nodes belonging to the same first-level partition node for physically adjacent rack columns; intermediate node level adjacency edges are not established between rack columns across partitions. This limited adjacency edge design allows the hierarchical topology graph to retain the semantics of physical spatial adjacency while avoiding the pollution of the graph structure by redundant edges across levels and partitions. This helps subsequent graph neural networks or attention mechanisms maintain the separation between the local receptive field and the global hierarchical receptive field when extracting node state embeddings.

[0029] The weight of spatially adjacent edges is taken as the Euclidean distance between the center points of the two nodes. For the leaf node layer, let the weight of the first edge be... The center coordinates of the leaf nodes are , No. The center coordinates of the leaf nodes are Then the weight of the spatially adjacent edges between the two. for: .

[0030] The adjacent edge weights of intermediate node layers are calculated in the same way as those of leaf node layers, using the Euclidean distance between the geometric center coordinates of the nodes. This weight value is directly referenced in the path repetition penalty term and path length cost term of the comprehensive objective function, enabling the path planning process to perceive and optimize the actual physical travel distance of the inspection device, avoiding detours or inefficient inspection paths.

[0031] After completing the node partitioning, membership edge establishment, attribute vector binding, and spatial adjacency edge construction, the entire hierarchical topology graph is stored in memory as a hybrid structure of a directed acyclic graph (DAG) and an undirected adjacency graph at the same level. In subsequent actual data center inspections, whenever a sensor reports new sampling data, it is only necessary to locate the corresponding leaf node and update its attribute vector. , , The components do not require reconstruction of the graph's topology, thus decoupling the static topology skeleton from the dynamic node attributes, ensuring that the state embedding vector read by the inspection agent at each decision step always reflects the latest operating status of the data center equipment.

[0032] Construct a comprehensive objective function that integrates the reward for complete inspection coverage, the penalty for path redundancy, and the risk-weighted reward based on abnormal event records, including: The inspection coverage integrity reward is defined as the ratio of the number of leaf nodes that have completed inspection visits to the total number of leaf nodes in the hierarchical topology graph. The reward value is taken as the output value of the S-shaped mapping function of the current inspection coverage ratio, so that the reward increases steepest when the coverage ratio is in the middle range and tends to the target growth rate when the coverage ratio approaches zero or the upper limit. The path repetition penalty term involves performing a set intersection operation between the set of visited leaf nodes in the inspection path and the historical access sequences generated by each step of the decision before the current decision time. A penalty value is applied to each repeatedly visited leaf node in the obtained intersection. The penalty value is taken as the exponentially increasing function value corresponding to the number of times the leaf node has been repeatedly visited. Based on the risk-weighted reward of abnormal event records, the abnormal event record set of its associated devices within a preset backtracking time window is extracted for each leaf node. Each abnormal event record in the abnormal event record set is mapped to the original risk score according to the severity level of the event. A time decay factor is imported according to the time interval between the time of the event occurrence and the current time for discount weighting. The discounted scores of each record are accumulated and summarized to obtain the node-independent risk score of each leaf node.

[0033] The construction of the comprehensive objective function is a core step in guiding the inspection agent to form a high-quality decision-making strategy during reinforcement learning training. This function consists of three independent yet synergistic components, corresponding to the dimensions of inspection coverage completeness, path redundancy, and device risk weight. The weighted sum of these three components forms the immediate reward signal for each decision step, driving the policy network to gradually approach the optimal inspection strategy during iterative training, which balances comprehensive coverage, avoids redundant access, and prioritizes responses to high-risk nodes.

[0034] The design principle of the inspection coverage integrity reward is to encourage inspection agents to cover all leaf nodes in the hierarchical topology graph as quickly as possible. Let the total number of leaf nodes in the hierarchical topology graph be... The number of leaf nodes that have completed inspection visits after a certain decision step is . The current inspection coverage ratio is defined as follows: Directly with The reward suffers from gradient sparsity: in the early stages of inspection with extremely low coverage and near-saturation in the later stages, the magnitude of reward signal changes is small, insufficient to provide effective learning impetus for the policy network. Therefore, the reward value is set to... The output value of the sigmoid mapping function. Specifically, a sigmoid transformation with coverage ratio as the independent variable is used, letting... ,in Controlling the steepness of the S-curve, The coverage ratio position corresponding to the midpoint of the control curve. When In the middle range (i.e., the coverage ratio is in) (nearby) right When the derivative reaches its maximum value, the reward increases most steeply with coverage progress, thus providing the strongest positive incentive for agents in the middle of the inspection; when When the slope approaches 0 or the upper limit of 1, the slope of the S-shaped function approaches the target growth rate (i.e., the asymptotic slope at both ends of the function), and the reward growth rate tends to be gradual, avoiding reward explosion or gradient vanishing in the extreme coverage ratio range. and The value of can be adjusted according to the size of the data center and the number of leaf nodes, and is usually set to 0. Set it to 0.5 so that the excitation peak occurs when the coverage is more than halfway complete. Set it between 6 and 10 to ensure that the reward gradient in the middle range is significant enough.

[0035] The path repetition penalty is introduced to suppress the inefficient strategy of the inspection agent repeatedly visiting the same leaf node during training. At each decision step, a historical access sequence is maintained, recording all node access records generated from the start of the inspection to the current decision step. Leaf nodes appearing in the historical access sequence are aggregated into a historical access set, and the intersection of this set with the currently visited leaf node set is taken. The intersection is the set of leaf nodes that have been repeatedly visited. For each repeatedly visited leaf node in the intersection, let the number of times the leaf node has been repeatedly visited in the historical access sequence be denoted as . The corresponding penalty value then takes the value of an exponentially increasing function. ,in As the penalty baseline coefficient, The growth rate is exponential. The key to this design is that the penalty increases exponentially with the number of repetitions: the penalty is relatively mild for the first repetition, but it rapidly amplifies with the accumulation of repetitions, thus creating a strong inhibitory effect at the policy level and prompting the inspection agent to actively explore unvisited leaf nodes. For cases involving multiple repetitions of leaf nodes within the same decision step, the penalty values ​​of each node are summed to form the total path repetition penalty for that step, which is included in the comprehensive objective function as a negative value. and The selection of the penalty signal must match the magnitude of the coverage integrity reward to ensure that the penalty signal does not drown out the positive reward, nor does it lose its constraint effect due to its small magnitude.

[0036] The construction logic of risk-weighted rewards based on anomaly event records is to transform historical anomaly information of devices into a quantitative basis for node access priority. For each leaf node, a preset backtracking time window is extracted from the historical anomaly event records of its associated devices. All abnormal events within the specified time frame constitute the abnormal event record set for that leaf node. Backtracking time window. The length is set based on data center operation and maintenance experience, typically ranging from 7 to 30 days, to balance recent frequent failure trends with mid-term equipment aging patterns. For each abnormal event record in the abnormal event log set, an original risk score is mapped according to the event severity level. Minor alarms correspond to lower scores, general faults to medium scores, and severe faults or downtime events to higher scores. Specific mapping relationships can be established based on a tiered scoring table developed according to data center operation and maintenance specifications. Furthermore, considering that the risk reference value of an anomaly decreases the further away it is from the current time, a time decay factor is introduced to discount and weight the original risk scores. Let the time interval between the occurrence time of a certain anomaly event record and the current time be denoted as . (Unit: hours), the time decay factor takes the form of exponential decay. ,in The decay rate parameter controls how quickly the influence of historical events fades over time. The larger the value, the more prominent the weight of recent events, and the faster the contribution of long-term events decays; The smaller the value, the longer the impact of the historical event is retained. This represents the discounted score for each record in the abnormal event log set. By summing up the results, we obtain the node-independent risk score for that leaf node. ,Right now The summation range covers all abnormal event records within the backtracking time window. The higher the node's independent risk score, the more serious or frequent abnormal events have recently occurred in the devices associated with that leaf node. The greater the risk-weighted reward that the inspection agent receives when accessing that node, the more the agent is guided to prioritize inspection access to high-risk nodes at the strategy level.

[0037] In the overall composition of the comprehensive objective function, the above three components are combined in a weighted summation manner, that is... ,in , , These are the coverage integrity reward weight, the path duplication penalty weight, and the risk-weighted reward weight, respectively. This represents the total path duplication penalty value for the current decision step. The proportional relationship between the three weighting coefficients determines the emphasis of the inspection strategy among the three objectives of comprehensive coverage, redundancy avoidance, and risk priority, and can be flexibly adjusted according to the actual operation and maintenance needs of the data center. For example, for data centers with high equipment failure rates, the penalty can be appropriately increased. The proportion of [something] makes the policy network more inclined to prioritize access to high-risk areas; for scenarios with ample inspection time resources, the proportion can be appropriately increased. The proportion ensures the integrity of full coverage. The comprehensive objective function returns an immediate reward at each decision step. As the direct basis for updating the policy network parameters using the policy gradient method, it drives the inspection agent to continuously optimize node access decisions during offline iterative training, eventually converging to a high-quality inspection path policy that takes into account multiple objective constraints.

[0038] Based on the risk-weighted reward system for abnormal event records, after summarizing the node-independent risk scores for each leaf node, the following are included: The node-independent risk scores of each leaf node are aggregated in reverse along the second type of directed membership edge of the hierarchical topology graph to the second-level cabinet column node to which it belongs. The aggregated risk score of each second-level cabinet column node is the weighted sum of the node-independent risk scores of all its subordinate leaf nodes. The aggregated risk scores of each secondary rack column node are aggregated in reverse along the first type of directed membership edge to the primary partition node to which it belongs. The regional risk score of each primary partition node is the weighted sum of the aggregated risk scores of all its subordinate secondary rack column nodes. The regional risk score of each first-level partition node is diffused in the positive direction along the spatial adjacency edge to all adjacent first-level partition nodes in the same layer. The diffusion amount is the inverse product of the source regional risk score and the corresponding spatial adjacency edge weight, and is added to the original regional risk score of the target region. The updated score after diffusion is used as the final risk score of the region. The final risk score of each non-leaf node is backpropagated along the directed membership edge to all its subordinate leaf nodes. The backpropagated value is summed with the node-independent risk score of each leaf node to form the comprehensive risk score after propagation enhancement. The comprehensive risk score after propagation enhancement is used as the access weight multiplier of the corresponding leaf node in the inspection coverage integrity reward.

[0039] After obtaining the node-independent risk assessment of each leaf node Subsequently, these local scores need to be propagated and aggregated in multiple layers along the structure of the hierarchical topology graph to make full use of the spatial correlation contained in the topological relationship. This will ensure that the risk weights used to guide the inspection path decision can reflect not only the historical anomaly information of a single device node, but also the overall risk situation of the same column, the same area, and even neighboring areas.

[0040] In the first step of aggregation, each leaf node has an independent risk score. The propagation proceeds in reverse along the second type of directed membership edge to the second-level rack column node to which it belongs. "Reverse" means that the propagation direction is opposite to the defined direction of the membership edge, i.e., converging from the child node to the parent node. For a given second-level rack column node... The set of its subordinate leaf nodes is denoted as The aggregate risk score of this column node Through the The weighted sum of the independent risk scores of all leaf nodes is obtained, i.e. ,in leaf node The aggregate weight in its respective rack column, leaf node Corresponding node-specific risk score. Aggregated weight. Priority can be preset based on equipment type or historical failure frequency. In the absence of special configuration, a uniform weighting can be applied. ,in This represents the number of leaf nodes under the cabinet column. This step ensures that if multiple devices within a single cabinet column are simultaneously in a high-risk state, its column-level aggregate score will be significantly higher than that of a scenario where only a few devices are abnormal, thus forming a quantitative representation of risk at the column level.

[0041] After completing column-level aggregation, the aggregation risk score of each secondary rack column node will be calculated. The aggregation continues in reverse along the first type of directed membership edge to the first-level partition node to which it belongs. For a given first-level partition node... The set of its subordinate cabinet nodes is denoted as The regional risk score of this partition node Through the The aggregated risk scores of all column nodes are weighted and summed to obtain the result. ,in For rack column nodes Aggregate weights within their respective partitions For column nodes Aggregated risk score. Weighting Similarly, the settings can be differentiated based on the scale of equipment or historical anomaly density under the jurisdiction of each column to reflect the differences in the contribution of different cabinet columns to the overall regional risk. After two levels of reverse aggregation, each primary partition node obtains a regional risk score that comprehensively reflects the risk status of all leaf nodes within its jurisdiction. This provides basic data for the subsequent cross-regional risk spread.

[0042] Regional risk scoring Once determined, a forward diffusion operation is performed between first-level partition nodes at the same level along spatial adjacency edges. Spatial adjacency edges connect two physically adjacent partition nodes in the data center layout, and their edge weights are the Euclidean distance between the center coordinates of the two nodes. The closer the distance, the higher the physical correlation between the two regions. The physical intuition of diffusion is that if a region contains a large number of high-risk devices, the probability of potential risks arising from environmental factors (such as heat dissipation and power fluctuations) in adjacent regions also increases. Therefore, it is necessary to propagate risk signals to neighboring regions. For source partition nodes... To adjacent partition nodes diffusion amount The risk score of the source region is taken as the inverse product of the weight of the corresponding spatial adjacent edge, i.e. ,in For nodes With nodes Euclidean distance weights between adjacent edges in the spatial region. Target partition node. Updated regional risk score It is the sum of its original regional risk score and the diffusion from all adjacent source nodes, i.e. ,in For nodes The set of all first-level partition nodes that have spatial adjacency relationships. The updated score after diffusion. This serves as the final risk score for the partition, used in subsequent data transmission steps. This diffusion mechanism ensures that even if no obvious anomalies occur in the devices within a partition, the inspection priority of that partition will be appropriately increased if the risk in its adjacent areas is high, thus avoiding overlooking potential hazards due to a local perspective.

[0043] After completing the final risk assessment of each primary partition node After calculation, the final risk score of each non-leaf node (including first-level partition nodes and second-level rack column nodes) is forward-propagated along the directed membership edges to all its subordinate leaf nodes. The forward propagation direction is consistent with the defined direction of the membership edges, i.e., from the parent node to the child node. For a given leaf node... The final risk score of the primary partition node to which it belongs is recorded as follows: The final risk score of the secondary cabinet node to which it belongs is recorded as follows: (The final risk score of the listed nodes here is their aggregated risk score) (The result of overlaying the values ​​returned from the partition node) is returned to the leaf node. Incremental value It is obtained by weighting and summing the final risk scores of each higher-level node. Leaf node Enhanced transmission and comprehensive risk score By comparing the backhaul increment with the node's own node-independent risk score The corresponding summation is obtained, that is ,in This process integrates risk signals transmitted down from the partition and column levels. This overlay operation allows the comprehensive risk score of a leaf node to retain the independent risk contributed by its own historical anomalies, while also incorporating regional and neighborhood risk information transmitted from higher-level topological structures, resulting in a comprehensive risk quantification result that integrates multi-source information.

[0044] Enhanced Comprehensive Risk Score Ultimately, this is used as the access weight multiplier for the corresponding leaf node in the calculation of the inspection coverage integrity reward. Specifically, when the inspection agent accesses a leaf node at a certain decision step... At that time, the contribution of this visit to the inspection coverage integrity reward is no longer a simple equal count, but rather based on... By weighting and amplifying the basic coverage reward, leaf nodes with higher overall risk scores receive greater reward gains when visited. This guides the inspection agent to prioritize paths through high-risk nodes during policy optimization. This design deeply couples risk information with the reward function, enabling the policy network to naturally include high-risk areas in its priority inspection scope after training convergence. While ensuring overall coverage integrity, it significantly improves the inspection response capability to potentially faulty nodes, thus achieving dynamic inspection path optimization that balances comprehensiveness and specificity in real-world data center operation and maintenance scenarios.

[0045] like Figure 2 As shown, the policy network parameters are iteratively optimized offline using the policy gradient method until the policy loss converges to a preset threshold, including: A dual network structure of policy network and value network is constructed. The policy network takes the state embedding vector of the current stationary node as input and the transition probability of each candidate transition action as the output layer normalization result. The value network takes the concatenated vector of the state embedding vector of the current stationary node and the action encoding vector of the selected transition action as input and outputs a scalar value estimate of the state-action pair. In each offline training round, starting from the starting node of the inspection agent, the policy network gradually outputs the transition probability distribution and samples it to determine the next dwell node until the coverage of all leaf nodes is completed. The transition actions, corresponding logarithms of transition probabilities, reward feedback values ​​and next node identifiers of each step are stored in the experience trajectory cache. For each step in the experience trajectory cache, calculate the future cumulative discount reward as the actual return estimate for that step. Use the difference between the value network's state action value estimate for that step and the actual return estimate as the temporal difference error. Gradually sum the temporal difference errors of each step along the experience trajectory from the end to the beginning to obtain the advantage function estimate for each step. The average of the product of the estimated advantage function at each step and the logarithm of the transition probability of the corresponding transition action over the entire trajectory is taken and reversed to serve as the policy loss function of the policy network. The policy network weights are then updated along the gradient direction of the policy loss function.

[0046] The policy network and value network work collaboratively using a dual structure. The policy network takes the state embedding vector of the current stationary node as its sole input, undergoes multiple fully connected transformations, and is then normalized by Softmax at the output layer, outputting a transition probability distribution covering all candidate transition actions. The value network concatenates the state embedding vector of the current stationary node with the action encoding vector of the selected transition action along the feature dimension, forming a concatenated vector which is then fed into an independent fully connected network, ultimately outputting a scalar value estimate for that state-action pair. The two networks can share parameters in the low-level feature extraction part to reduce computational overhead, but their output heads remain independent, ensuring that the gradient updates of policy estimation and value estimation do not interfere with each other. The action encoding vector is obtained by embedding the node identifier of the target node into a lookup table, maintaining the same dimension as the state embedding vector, thus fixing the dimension of the concatenated vector and facilitating processing by the value network.

[0047] At the start of each offline training round, the inspection agent departs from a pre-specified starting node. The policy network outputs the transition probability distribution of each candidate neighboring node based on the current node's state embedding vector, and randomly samples according to this distribution to determine the target node for the next stop. After sampling, the value network provides a scalar estimate of the current state-action pair, and simultaneously returns immediate reward feedback based on leaf node visit information, path repetition, and risk weighting information, integrating the objective function. This information includes the node identifier of the selected transition action and the logarithm of the transition probability of that action under the current distribution. Instant rewards And the identifier of the next node after the transfer—which is sequentially appended to the experience trajectory cache, forming a complete time-series record. This process continues until the inspection agent completes coverage access to all leaf nodes in the hierarchical topology graph, at which point the single-round trajectory terminates. Indicates by parameters The conditional probabilities defined by the parameterized policy network. For the first Step selected transfer action, For the first The state embedding vector of the step.

[0048] After the experience trajectory cache is filled, the cumulative discounted reward for each time step in the cache is calculated as an estimate of the actual reward for that step. Let the total trajectory step length be... , No. Real return estimate of the step It can be obtained recursively as follows: ; in As a discount factor, For the first Instant rewards for each step Initialize to zero. Discount factor. The value of determines the degree of influence of long-term rewards on current decisions, and is usually set between 0.95 and 0.99 to balance short-term gains with the quality of long-term path planning.

[0049] Timing Differential Error Defined as a value network for the first Estimates of step-state action pairs Compared with actual return estimates Difference: ; when When positive, it indicates that the actual reward of this step exceeds the value network's expectations, and the policy network should increase the probability of selecting this action; when... When the value is negative, the probability of choosing that action should be reduced accordingly. Based on this, a generalized advantage estimate is introduced to calculate the advantage function for each step. The generalized advantage estimate is progressively accumulated from the end of the trajectory forward, with the... The advantage function estimate of the step The following recursive formula is used to obtain: ; in The attenuation coefficient for generalized dominance estimation, and the trade-off between control bias and variance. Initialize to zero. When When the value approaches 0, the dominance estimate degenerates into a single-step time-series difference error, with a small bias but a large variance; when When the odds ratio approaches 1, the dominance estimate is close to the Monte Carlo return, with smaller variance but larger bias. In practice... A value of 0.95 is typically chosen to achieve a better balance between the two.

[0050] The construction of the policy loss function revolves around the product of the estimated advantage function value at each step and the logarithm of the corresponding transition probability. The policy loss of the policy network is obtained by averaging the products at each step of the entire trajectory and reversing their signs. : ; The inverse operation transforms the objective of maximizing the expected cumulative reward into the standard gradient descent form of minimizing the loss. Along Policy network parameters The gradient direction is used to perform parameter updates, which increases the transition probability of the policy network for actions with positive advantages and decreases the transition probability for actions with negative advantages, thereby gradually guiding the inspection path to evolve towards high coverage, low repetition, and high risk priority.

[0051] The parameters of the value network are updated independently, with the mean squared error loss as the objective, so that... As close as possible This provides a more accurate benchmark for estimating the advantage in subsequent rounds. Both networks use a mini-batch stochastic gradient descent optimizer for parameter updates, and the learning rate is gradually reduced as training progresses using a linear decay scheduling strategy to ensure stability in later training stages.

[0052] The offline iterative optimization process continues for multiple rounds. After each round, the current policy loss value is calculated and compared with a preset convergence threshold. If the policy loss is lower than the threshold for several consecutive rounds, the policy network is considered to have converged, training is stopped, and the network weights are saved. The selection of the preset convergence threshold needs to be determined comprehensively based on the specific data center size and topology complexity. An excessively large threshold may lead to underoptimization of the policy, while an excessively small threshold may cause overfitting, making the policy network over-adapt to a specific state distribution in the training environment and lose its generalization ability. In actual deployment, the weights of the trained policy network are fixed, and the value network only participates in the calculation during the offline training phase and does not participate in the online inference process, thereby reducing the computational overhead in actual inspections.

[0053] For each step in the experience trajectory cache, calculate the future cumulative discount reward as an estimate of the actual return for that step, including: The decision-making process of the hierarchical topology graph is divided into three sequential training stages: the first-level partition node level, the second-level rack column node level, and the third-level rack unit leaf node level. During the training phase at the first-level partition node level, the action space consists only of the spatial adjacency edge transfers between each first-level partition node. The policy network learns the coarse-grained partition access order until the information entropy of the partition layer policy output converges to the first threshold. During the training phase at the secondary rack column node level, the access order framework of each partition is fixed by the access order of the primary partition nodes output by the partition layer strategy. The action space is switched to the transition between secondary rack column nodes within the current partition. The strategy network learns the column-level access order within the constraints of the current partition until the information entropy of the column-level strategy converges to the second threshold. In the training phase of the leaf node level of the three-level cabinet unit, the access order framework of the first two phases is inherited. Dependency constraints based on device interconnection dependencies are introduced between leaf nodes within the same cabinet column. If the currently selected leaf node has an unaccessed predecessor leaf node in the dependency relationship, a penalty is applied and forced to fall back to the predecessor leaf node. The policy network learns the leaf node layer access order under the dependency constraints until the overall policy converges.

[0054] After constructing the experience trajectory cache, it is necessary to calculate the cumulative discounted reward for each step in the cache to obtain an estimate of the actual return for that step. To ensure the policy network can converge effectively in such a complex hierarchical topology, a hierarchical, phased progressive training approach is adopted. The entire decision-making process is divided into three sequentially advancing training phases based on the three-layer structure of the hierarchical topology graph, corresponding to the first-level partition node level, the second-level rack column node level, and the third-level rack unit leaf node level, respectively. The three phases unfold sequentially from coarse to fine and from macro to micro. The policy output of the previous phase provides the access order constraint framework for the next phase, thereby decomposing the originally high-dimensional and complex joint decision space into three relatively independent and manageable-scale sub-decision problems, effectively reducing the exploration difficulty and convergence time of the policy network.

[0055] In the training phase at the first-level partition node level, the action space is strictly limited to the set of transitions formed by the spatial adjacency edges between each first-level partition node. In this phase, the policy network only needs to learn the access order among several regional partitions in the data center, without being aware of detailed information at the rack column and rack unit levels. The input state only includes the regional risk score bound to each partition node, the visited partition marker, and the location code of the current partition. Based on these coarse-grained state characteristics, the policy network gradually explores and optimizes the access order between partitions, updating parameters primarily driven by integrity rewards and risk-weighted rewards. This training phase continues until the information entropy output by the partition-level policy converges to a preset first threshold. The calculation of information entropy is based on the probability distribution of all selectable partition transition actions of the policy network in the current state. When the policy becomes more deterministic, the information entropy decreases and stabilizes near the first threshold, indicating that the partition-level access policy has matured and possesses sufficiently stable partition access preferences, allowing the partition access order to be solidified and passed to the next training phase.

[0056] In the training phase at the secondary rack column node level, the access order of the primary partition nodes obtained in the first phase is used as a fixed partition access order framework. This means the macro-level access order of each partition is no longer optimized in this phase but is embedded as a hard constraint in the decision-making process. The action space switches to the transitions between secondary rack column nodes within the current partition. The policy network learns the rack column-level access order within the constraints of the current partition. Whenever the inspection agent completes access to all rack columns within the current partition, it automatically switches to the next partition according to the partition access order framework and re-implements column-level decisions within that partition. Building upon the first phase, the input state further incorporates the aggregated risk score of each rack column node in the current partition, the marker of visited columns, and the position code of the current column. This allows the policy network to perceive the differences in risk distribution within the partition and prioritize guiding the inspection agent to access high-risk rack columns. This phase also uses information entropy as the convergence criterion, continuing training until the information entropy of the column-level policy converges to the second threshold, indicating that the rack column-level access policy has reached a stable level, and the column-level access order framework can be further passed to the third training phase.

[0057] In the training phase at the leaf node level of the three-level rack unit, the partitioned access order framework and column-level access order framework determined in the first two phases are inherited. The action space is further refined into transitions between leaf nodes within the same rack column. This phase introduces a dependency constraint mechanism based on device interconnection dependencies at the leaf node level to handle the logical prerequisite relationships between devices within the rack unit. Device interconnection dependencies originate from the static configuration information of the data center network topology and power supply links. For example, a switching device needs to complete inspection confirmation before its downstream server device is accessed, or a UPS device needs to complete status verification before the rack units within its power supply range are accessed. Dependencies are pre-constructed in the form of a directed graph and injected into a hierarchical topology graph, with each leaf node carrying information about the set of its prerequisite leaf nodes.

[0058] At each decision step, the policy network selects the next leaf node to be accessed from the set of leaf nodes in the current rack column. Immediately after selection, it checks whether the selected leaf node has any unvisited preceding dependent leaf nodes. If an unvisited preceding dependent leaf node exists, a dependency default handling mechanism is triggered: a penalty is applied to the current action, with the penalty value fed back to the policy network as a negative immediate reward. Simultaneously, the current position of the inspection agent is forcibly reverted to that preceding dependent leaf node, prioritizing its inspection and access to the dependent node. The strength of the dependency constraint penalty must be sufficiently significant to ensure that the policy network gradually learns to proactively avoid dependency defaults before selecting the target leaf node under repeated penalty signal drive. If all preceding dependent leaf nodes of the selected leaf node have been accessed, the transition action is executed normally, the immediate reward returned by the comprehensive objective function is received, and the process enters the next state.

[0059] The three-stage sequential training mechanism effectively solves the training difficulties caused by the coupling of multi-level decision spaces in hierarchical topology graphs. Within each stage, the policy gradient method iteratively updates the policy network parameters based on the future cumulative discount rewards in the experience trajectory cache. The calculation of the cumulative discount rewards adopts a recursive approach from the end of the round backwards, discounting the immediate rewards of subsequent steps with a discount factor. Weighted summation is performed to obtain the actual reward estimate for each step. Each of the three stages maintains its own experience trajectory cache, and the reward estimate for each stage is accumulated only within the decision-making scope of its own stage, avoiding mutual interference between reward signals across different levels.

[0060] After the entire phased training process is completed, the strategies at all three levels have converged to a stable state, forming a complete hierarchical joint strategy. In actual data center inspection, the three-layer strategy is activated sequentially from coarse to fine: the partition layer strategy first determines the access order of each partition, the column layer strategy determines the access order of the rack column under the current partition constraints, and the leaf node layer strategy determines the access order of the rack unit under the dual constraints of the current rack column constraints and dependency constraints. The three-layer decision collaboration outputs a complete node access sequence, which serves as a dynamic inspection path to guide the inspection device in performing inspection operations.

[0061] The latest topology graph instance is fed into the trained inspection agent, which then outputs node access sequences to form a dynamic inspection path. The inspection device executes inspection operations sequentially according to the dynamic inspection path, including: Real-time operating parameters of the equipment side of each leaf node corresponding to the cabinet unit are collected in real time. A multi-dimensional equipment side attribute vector is constructed for each leaf node based on the real-time temperature sampling value, real-time power consumption sampling value and fan speed sampling value. For non-leaf nodes, the aggregated attribute vector is obtained by dimensionality reduction and aggregation based on the mean of its child nodes in each component dimension. The hierarchical topology graph that has completed the attribute update is marked as the latest topology graph instance. The attribute vectors of each node in the latest topology graph instance are encoded into state embedding tensors and input into the trained and converged policy network. The policy network infers and outputs the next target node identifier step by step from the current physical stop node of the inspection device until all leaf nodes are marked as visited. The target node identifier sequence output by the policy network and the physical distance information of the spatial adjacent edges at each layer are input into the path generator. Based on the weight of the spatial adjacent edges, the local shortest path is searched between adjacent target nodes and the complete inspection movement route including the intermediate nodes is output. During the execution of the complete inspection route, the inspection device periodically collects the operating parameters of each cabinet unit through sensors. If the operating parameters of any cabinet unit exceed the preset deviation range, an abnormal event is triggered and written into the abnormal event record of the corresponding leaf node. The hierarchical topology map is updated in real time and the policy network is called again to generate a corrected inspection route.

[0062] During the actual data center inspection execution phase, after the inspection device is activated, it first performs initial data collection on all rack units within the data center. Using temperature sensors, power consumption metering modules, and fan speed monitoring interfaces deployed in each rack unit, it periodically reads the real-time temperature, power consumption, and fan speed samples of each device. Following the same feature concatenation method as in the training phase, a multi-dimensional device-side attribute vector is constructed for each leaf node. For intermediate nodes in a rack column, all its member leaf nodes are traversed, and the arithmetic mean is calculated for each component dimension (temperature, power consumption, and fan speed). The resulting mean vector is used as the aggregated attribute vector for that intermediate node, thus transmitting the fine-grained state information of the underlying devices to the upper-level nodes in compressed form. For the root node of the data center area, the aggregated attribute vectors of all its subordinate rack column intermediate nodes are similarly aggregated a second time based on the mean of each component, yielding a macroscopic state description of the entire area. After completing the bottom-up attribute filling, this hierarchical topology graph is marked as the latest topology graph instance at the current moment, serving as the input basis for subsequent policy reasoning.

[0063] The attribute vectors of each node in the latest topology instance are flattened and concatenated according to the node hierarchy to form a state embedding tensor, which is then input into the policy network that has completed offline training and convergence. Upon receiving the state embedding tensor, the policy network uses the physical stop node where the inspection device is currently located as the starting node. In each inference step, it outputs the access probability distribution of all candidate nodes, selecting the unvisited leaf node with the highest probability as the next target node identifier and recording it. The selected leaf node is marked as visited in subsequent inference steps, and the access flag is updated accordingly in the state embedding tensor, enabling the policy network to perceive the current inspection progress and dynamically adjust its subsequent node selection preferences. This process continues iterating until all leaf nodes in the hierarchical topology graph are marked as visited. At this point, the policy network outputs a termination signal, and the recorded target node identifier sequence is the logical access order for this round of inspection.

[0064] After the logical access order is determined, the target node identifier sequence, along with the physical distance weights carried by each spatial adjacency edge in the hierarchical topology graph, is input into the path generator. The path generator searches for the connectivity between two adjacent target nodes in the sequence. If there is a direct spatial adjacency edge between the two nodes, that edge is used directly as the movement segment. If there is no direct adjacency edge between the two nodes, the local shortest path search is performed in the topology graph at the cost of the spatial adjacency edge weight to find the path with the minimum cost connecting the two target nodes. All intermediate nodes traversed on this path are then inserted into the movement route in sequence. The local shortest paths between each pair of adjacent target nodes are then concatenated end to end to output a complete inspection movement route containing all intermediate nodes. This complete route not only provides the sequence of nodes that the inspection device needs to visit in sequence but also specifies which channel to travel along in each movement segment, thus enabling the inspection device to perform continuous and unambiguous movement operations in physical space.

[0065] As the inspection device travels along the complete inspection route, vehicle-mounted or airborne sensors continuously collect operating parameters of the current cabinet unit and adjacent cabinet units at fixed intervals, including key indicators such as temperature, power consumption, and fan speed. Each collected operating parameter is compared with the pre-configured normal operating threshold range for that cabinet unit. If any operating parameter of a cabinet unit exceeds the preset deviation range, it is determined as an abnormal event. The occurrence time, anomaly type, deviation magnitude, and the leaf node identifier of the abnormal event are recorded. This abnormal event record is written to the historical abnormal event record storage area of ​​the corresponding leaf node, and the risk-related attribute fields of that leaf node are updated simultaneously.

[0066] After an anomaly event is written, an immediate update operation is performed on the hierarchical topology graph. The update includes: replacing the real-time running status parameters in the attribute vectors of the leaf nodes that experienced the anomaly with the latest values ​​collected this time, and setting the anomaly flag to valid; recalculating the aggregated attribute vectors of the intermediate nodes of their respective rack columns and the root node of the server room area layer by layer upwards from the leaf node, ensuring that the attributes of nodes at each layer of the topology graph reflect the latest equipment status. The updated hierarchical topology graph is re-marked as the latest topology graph instance, and the state embedding tensor construction process is immediately re-executed, feeding the new state embedding tensor into the trained and converged policy network. The policy network uses the current actual node of the inspection device as the new starting node, re-infers and outputs the access sequence of the remaining unvisited leaf nodes, and the path generator re-executes the local shortest path search accordingly, generating a corrected complete inspection movement route. The inspection device then continues to perform subsequent inspection operations according to the corrected route, thereby achieving real-time response to sudden anomalies and dynamic path adjustment.

[0067] Under the dynamic path adjustment mechanism, the risk information of abnormal events directly affects the state input of the policy network through the update of topology graph attributes. This enables the policy network to assign higher access priority to high-risk nodes during re-inference, and advances the subsequent review arrangements for the cabinet unit where the abnormal device is located. This closed-loop mechanism ensures that the inspection path always remains dynamically consistent with the real-time operating status of the equipment in the data center, avoiding the limitation of static preset paths that cannot be adjusted in time when the equipment status changes abruptly.

[0068] When the inspection device encounters situations where physical obstacles or congestion render certain road sections impassable during its movement, the path generator can temporarily maximize the weight of the blocked edges during local shortest path search. This allows the search algorithm to automatically bypass impassable sections and output alternative routes, ensuring the continuity of the inspection task. The entire dynamic inspection path generation and execution process forms a continuous loop of "collection—update—reasoning—execution—collection" until all leaf nodes in the current round have completed their inspection visits, the inspection device returns to its initial docking point, and the inspection task ends.

[0069] A second aspect of this invention provides a data center inspection path dynamic optimization system based on reinforcement learning, comprising: The topology building unit is used to build a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room areas as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. The action reward unit is used to define an inspection action space with inter-node transfer as an optional action based on the hierarchical topology graph, and to construct a comprehensive objective function that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The strategy training unit is used to connect the inspection agent with a dynamic interactive environment that uses a hierarchical topology as its skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the strategy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The strategy network parameters are iteratively optimized offline through the strategy gradient method until the strategy loss converges. The dynamic path unit is used to obtain the real-time operating status parameters and historical abnormal event records of each inspection target during actual data center inspections, inject the latest topology graph instance, feed it into the trained inspection agent, and gradually output the node access sequence as the dynamic inspection path, so that the inspection device can perform inspection operations accordingly.

[0070] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0071] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0072] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic optimization of computer room inspection paths based on reinforcement learning, characterized in that, include: Construct a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room area as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. Based on the hierarchical topology graph, an inspection action space with inter-node transfer as an optional action is defined, and a comprehensive objective function is constructed that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The inspection agent is connected to a dynamic interactive environment with a hierarchical topology as the skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the policy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The policy network parameters are iteratively optimized offline through the policy gradient method until the policy loss converges. In actual data center inspections, real-time operating status parameters and historical abnormal event records of each inspection target are obtained and injected into the latest topology graph instance. After being fed into the trained inspection agent, the node access sequence is gradually output as a dynamic inspection path, and the inspection device performs inspection operations accordingly.

2. The method according to claim 1, characterized in that, Construct a hierarchical topology map of the inspection targets within the computer room, including: Based on the coordinate reference system of the physical plan layout of the computer room, the first-level partition nodes are divided according to the area boundary frame. The second-level rack column nodes are divided according to the row direction of the rack column within each first-level partition node. The third-level rack unit leaf nodes are divided according to the column direction of the rack unit within each second-level rack column node. Establish a first type of directed membership edge between each leaf node of a third-level cabinet unit and its corresponding second-level cabinet column node, and establish a second type of directed membership edge between each second-level cabinet column node and its corresponding first-level partition node, so that any leaf node can reach the unique root node along the directed membership edge through the unique intermediate node, forming an unambiguous three-layer tree-like membership skeleton. Each leaf node is bound to a device-side attribute vector, which is a composite attribute vector composed of real-time temperature sampling values, real-time power consumption sampling values, and fan speed sampling values ​​of each device in the corresponding cabinet unit, arranged in order of device identifier; Undirected spatial adjacency edges are established within the same level based on the physical adjacency relationship between nodes. The weight of the spatial adjacency edge is the Euclidean distance between the center points of the two nodes. Spatial adjacency edges at the leaf node level are established only within the subset of leaf nodes under the jurisdiction of the same second-level cabinet column node. Spatial adjacency edges at the intermediate node level are established within the subset of intermediate nodes under the jurisdiction of the same first-level partition node.

3. The method according to claim 1, characterized in that, Construct a comprehensive objective function that integrates the reward for complete inspection coverage, the penalty for path redundancy, and the risk-weighted reward based on abnormal event records, including: The inspection coverage integrity reward is defined as the ratio of the number of leaf nodes that have completed inspection visits to the total number of leaf nodes in the hierarchical topology graph. The reward value is taken as the output value of the S-shaped mapping function of the current inspection coverage ratio, so that the reward increases steepest when the coverage ratio is in the middle range and tends to the target growth rate when the coverage ratio approaches zero or the upper limit. The path repetition penalty term involves performing a set intersection operation between the set of visited leaf nodes in the inspection path and the historical access sequences generated by each step of the decision before the current decision time. A penalty value is applied to each repeatedly visited leaf node in the obtained intersection. The penalty value is taken as the exponentially increasing function value corresponding to the number of times the leaf node has been repeatedly visited. Based on the risk-weighted reward of abnormal event records, the abnormal event record set of its associated devices within a preset backtracking time window is extracted for each leaf node. Each abnormal event record in the abnormal event record set is mapped to the original risk score according to the severity level of the event. A time decay factor is imported according to the time interval between the time of the event occurrence and the current time for discount weighting. The discounted scores of each record are accumulated and summarized to obtain the node-independent risk score of each leaf node.

4. The method according to claim 3, characterized in that, Based on the risk-weighted reward system for abnormal event records, after summarizing the node-independent risk scores for each leaf node, the following are included: The node-independent risk scores of each leaf node are aggregated in reverse along the second type of directed membership edge of the hierarchical topology graph to the second-level cabinet column node to which it belongs. The aggregated risk score of each second-level cabinet column node is the weighted sum of the node-independent risk scores of all its subordinate leaf nodes. The aggregated risk scores of each secondary rack column node are aggregated in reverse along the first type of directed membership edge to the primary partition node to which it belongs. The regional risk score of each primary partition node is the weighted sum of the aggregated risk scores of all its subordinate secondary rack column nodes. The regional risk score of each first-level partition node is diffused in the positive direction along the spatial adjacency edge to all adjacent first-level partition nodes in the same layer. The diffusion amount is the inverse product of the source regional risk score and the corresponding spatial adjacency edge weight, and is added to the original regional risk score of the target region. The updated score after diffusion is used as the final risk score of the region. The final risk score of each non-leaf node is backpropagated along the directed membership edge to all its subordinate leaf nodes. The backpropagated value is summed with the node-independent risk score of each leaf node to form the comprehensive risk score after propagation enhancement. The comprehensive risk score after propagation enhancement is used as the access weight multiplier of the corresponding leaf node in the inspection coverage integrity reward.

5. The method according to claim 1, characterized in that, The policy network parameters are iteratively optimized offline using the policy gradient method until the policy loss converges to a preset threshold, including: A dual network structure of policy network and value network is constructed. The policy network takes the state embedding vector of the current stationary node as input and the transition probability of each candidate transition action as the output layer normalization result. The value network takes the concatenated vector of the state embedding vector of the current stationary node and the action encoding vector of the selected transition action as input and outputs a scalar value estimate of the state-action pair. In each offline training round, starting from the starting node of the inspection agent, the policy network gradually outputs the transition probability distribution and samples it to determine the next dwell node until the coverage of all leaf nodes is completed. The transition actions, corresponding logarithms of transition probabilities, reward feedback values ​​and next node identifiers of each step are stored in the experience trajectory cache. For each step in the experience trajectory cache, calculate the future cumulative discount reward as the actual return estimate for that step. Use the difference between the value network's state action value estimate for that step and the actual return estimate as the temporal difference error. Gradually sum the temporal difference errors of each step along the experience trajectory from the end to the beginning to obtain the advantage function estimate for each step. The average of the product of the estimated advantage function at each step and the logarithm of the transition probability of the corresponding transition action over the entire trajectory is taken and reversed to serve as the policy loss function of the policy network. The policy network weights are then updated along the gradient direction of the policy loss function.

6. The method according to claim 5, characterized in that, For each step in the experience trajectory cache, calculate the future cumulative discount reward as an estimate of the actual return for that step, including: The decision-making process of the hierarchical topology graph is divided into three sequential training stages: the first-level partition node level, the second-level rack column node level, and the third-level rack unit leaf node level. During the training phase at the first-level partition node level, the action space consists only of the spatial adjacency edge transfers between each first-level partition node. The policy network learns the coarse-grained partition access order until the information entropy of the partition layer policy output converges to the first threshold. During the training phase at the secondary rack column node level, the access order framework of each partition is fixed by the access order of the primary partition nodes output by the partition layer strategy. The action space is switched to the transition between secondary rack column nodes within the current partition. The strategy network learns the column-level access order within the constraints of the current partition until the information entropy of the column-level strategy converges to the second threshold. In the training phase of the leaf node level of the three-level cabinet unit, the access order framework of the first two phases is inherited. Dependency constraints based on device interconnection dependencies are introduced between leaf nodes within the same cabinet column. If the currently selected leaf node has an unaccessed predecessor leaf node in the dependency relationship, a penalty is applied and forced to fall back to the predecessor leaf node. The policy network learns the leaf node layer access order under the dependency constraints until the overall policy converges.

7. The method according to claim 1, characterized in that, The latest topology graph instance is fed into the trained inspection agent, which then outputs node access sequences to form a dynamic inspection path. The inspection device executes inspection operations sequentially according to the dynamic inspection path, including: Real-time operating parameters of the equipment side of each leaf node corresponding to the cabinet unit are collected in real time. A multi-dimensional equipment side attribute vector is constructed for each leaf node based on the real-time temperature sampling value, real-time power consumption sampling value and fan speed sampling value. For non-leaf nodes, the aggregated attribute vector is obtained by dimensionality reduction and aggregation based on the mean of its child nodes in each component dimension. The hierarchical topology graph that has completed the attribute update is marked as the latest topology graph instance. The attribute vectors of each node in the latest topology graph instance are encoded into state embedding tensors and input into the trained and converged policy network. The policy network infers and outputs the next target node identifier step by step from the current physical stop node of the inspection device until all leaf nodes are marked as visited. The target node identifier sequence output by the policy network and the physical distance information of the spatial adjacent edges at each layer are input into the path generator. Based on the weight of the spatial adjacent edges, the local shortest path is searched between adjacent target nodes and the complete inspection movement route including the intermediate nodes is output. During the execution of the complete inspection route, the inspection device periodically collects the operating parameters of each cabinet unit through sensors. If the operating parameters of any cabinet unit exceed the preset deviation range, an abnormal event is triggered and written into the abnormal event record of the corresponding leaf node. The hierarchical topology map is updated in real time and the policy network is called again to generate a corrected inspection route.

8. A data center inspection path dynamic optimization system based on reinforcement learning, used to implement the method as described in any one of claims 1-7, characterized in that, include: The topology building unit is used to build a hierarchical topology diagram of the inspection targets in the computer room. The hierarchical topology diagram includes a three-layer hierarchical structure with cabinet units as leaf nodes, cabinet columns as intermediate nodes, and computer room areas as root nodes. Each node is bound to the corresponding real-time operating status parameters of the equipment and historical abnormal event records. The action reward unit is used to define an inspection action space with inter-node transfer as an optional action based on the hierarchical topology graph, and to construct a comprehensive objective function that integrates inspection coverage integrity reward, path repetition penalty, and risk-weighted reward based on abnormal event records. The strategy training unit is used to connect the inspection agent with a dynamic interactive environment that uses a hierarchical topology as its skeleton and real-time operating status parameters and historical abnormal event records as variable attributes. At each decision step, the inspection agent outputs the action probability distribution through the strategy network based on the state embedding vector, executes the selected transition action, receives the instant reward returned by the comprehensive objective function, and enters the next state. The strategy network parameters are iteratively optimized offline through the strategy gradient method until the strategy loss converges. The dynamic path unit is used to obtain the real-time operating status parameters and historical abnormal event records of each inspection target during actual data center inspections, inject the latest topology graph instance, feed it into the trained inspection agent, and gradually output the node access sequence as the dynamic inspection path, so that the inspection device can perform inspection operations accordingly.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images