Data center inspection robot monitoring and analysis method and system based on machine vision

Abstract

This invention provides a machine vision-based data center inspection robot monitoring and analysis method and system. This system involves acquiring video streams and environmental parameter data collected by the inspection robot, using deep learning target detection to identify abnormal conditions, constructing a multimodal data fusion analysis model to form a knowledge graph to generate root cause analysis results, and planning inspection routes based on priority scores. This invention enables intelligent and precise data center monitoring, improving inspection efficiency and fault diagnosis accuracy.

Description

Technical Field

[0001] The present invention relates to inspection robot technology, and in particular to a data center inspection robot monitoring and analysis method and system based on machine vision. Background Art

[0002] As data centers continue to expand in size and their infrastructure becomes more complex, traditional manual inspection methods are no longer able to meet the efficient and accurate monitoring needs of modern data centers. As critical infrastructure in the information age, the stable operation of data centers is directly related to the reliability of various online services. Currently, robotic inspection technology has been gradually applied to daily data center operations and maintenance. Equipped with various sensors and cameras, it enables automated monitoring of the operating status of cabinets and equipment. Machine vision-based data center inspection systems can identify and analyze key information such as device indicator light status, cable connectivity, and cabinet temperature, providing real-time monitoring support for data center operations and maintenance.

[0003] Existing inspection technology has obvious shortcomings in the fusion of multi-source heterogeneous data. Visual information and environmental parameter data are usually analyzed as independent monitoring dimensions, making it difficult to establish the correlation between the visual status of the equipment and the operating parameters. This leads to low accuracy in abnormal root cause analysis and the inability to effectively identify the internal connections of equipment failures.

[0004] Traditional inspection systems lack intelligent inspection strategy optimization mechanisms. Inspection route planning is often based on preset fixed routes and cannot dynamically adjust inspection priorities according to the real-time status of data center equipment. This makes it difficult to achieve efficient responses when equipment abnormalities occur, reducing inspection efficiency and the timeliness of fault handling.

[0005] Existing technologies mainly rely on single-modal feature extraction to identify abnormal equipment conditions. In particular, the recognition accuracy of subtle changes such as equipment indicator light status and cable connection abnormalities in complex environments is low, making it difficult to adapt to diverse scenarios with different lighting conditions and equipment types, limiting the scope of application and reliability of the inspection system. Summary of the Invention

[0006] The embodiments of the present invention provide a data center inspection robot monitoring and analysis method and system based on machine vision, which can solve the problems in the prior art.

[0007] A first aspect of an embodiment of the present invention provides a data center inspection robot monitoring and analysis method based on machine vision, comprising:

[0008] Acquire monitoring video stream data collected by a data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data;

[0009] Analyzing the real-time image using a deep learning target detection model to identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features;

[0010] Construct a multimodal data fusion analysis model, perform spatiotemporal registration of the visual abnormality characteristics with the environmental parameter data, and construct a device abnormality knowledge graph using an improved graph reasoning algorithm. The improved graph reasoning algorithm enables collaborative analysis of multi-source heterogeneous data by introducing a dynamic weight adaptation mechanism to generate root cause analysis results.

[0011] The root cause analysis results are input into the strategy network to generate inspection task priority scores. The inspection paths are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots.

[0012] The real-time image is analyzed using a deep learning target detection model to identify the device indicator light status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal status features including:

[0013] Use deep learning target detection models to extract device indicator light status, cable connection status, and cabinet temperature display values, and construct a cause-and-effect graph of device status.

[0014] Based on the device state causal graph, an attention mechanism is used to calculate feature weights for the indicator light area, cable area, and temperature display area in the real-time image, and the feature weights are weightedly fused with the visual features extracted by the deep learning object detection model to construct a multi-scale visual feature representation;

[0015] For abnormal states detected in the multi-scale visual feature representation, a visual feature counterfactual verification function is constructed. The visual feature counterfactual verification function performs abnormality verification by calculating the difference between the state recognition probability after image enhancement processing and the original state recognition probability. Based on the result of the abnormality verification, the visual similarity between the candidate abnormal area and the historical abnormal case is calculated to generate the visual abnormality attribution confidence.

[0016] An abnormality evolution path is constructed according to the visual abnormality attribution confidence, and an early warning score calculation function is designed based on the abnormality evolution path. The early warning score calculation function comprehensively evaluates the abnormality degree of the device state and the visual feature deviation to generate a visual abnormality state feature.

[0017] Constructing an abnormality evolution path according to the visual abnormality attribution confidence, and designing an early warning score calculation function based on the abnormality evolution path includes:

[0018] Acquire a device abnormal state sequence, the device abnormal state sequence including an abnormal visual feature vector, a device state feature vector, and a state transition probability, calculate the device abnormal state sequence according to the visual abnormality attribution confidence, and generate a state transition weight;

[0019] Constructing a multidimensional feature fusion function based on the abnormal state sequence of the device, and performing weighted summation of the state transition weight and the multidimensional feature fusion function to obtain a fusion feature representation;

[0020] A hierarchical early warning score calculation model is constructed using the fused feature representation, the feature distance between the abnormal visual feature vector and the normal state benchmark is calculated, the feature distance is multiplied by the device state feature vector to obtain an abnormality degree score, the rate of change of the state transition probability is calculated and combined with the time attenuation factor to generate a trend score, and at the comprehensive evaluation layer, the abnormality degree score, the trend score, and the confidence score are weightedly fused to generate a warning score calculation function.

[0021] Constructing a multimodal data fusion analysis model, performing spatiotemporal registration of the visual abnormal state features with the environmental parameter data, and using an improved graph inference algorithm to construct a device abnormality knowledge graph includes:

[0022] Constructing a multimodal data fusion analysis model, inputting the visual abnormal state feature and the environmental parameter data into the multimodal data fusion analysis model for spatiotemporal registration, and extracting the timestamp identifier and spatial position identifier of the visual abnormal state feature and the environmental parameter data;

[0023] Constructing a temporal correlation matrix based on the timestamp identifier, constructing a spatial adjacency matrix based on the spatial position identifier, and performing weighted fusion of the temporal correlation matrix and the spatial adjacency matrix to generate a spatiotemporal registration matrix;

[0024] An improved graph reasoning algorithm is used to construct a device anomaly knowledge graph, which includes an entity node layer, an attribute node layer, and a relationship edge set. Entity nodes representing different devices are constructed in the entity node layer, and attribute nodes representing device abnormal states are constructed in the attribute node layer. Entity relationship edges of the entity nodes are constructed based on the spatiotemporal registration matrix, and attribute relationship edges of the attribute nodes are constructed based on device state transition rules. The improved graph reasoning algorithm is used to iteratively update the entity relationship edges and the attribute relationship edges to generate a device anomaly knowledge graph with a hierarchical structure.

[0025] The improved graph inference algorithm introduces a dynamic weight adaptation mechanism to achieve collaborative analysis of multi-source heterogeneous data, generating root cause analysis results including:

[0026] Constructing a knowledge graph structure of the improved graph reasoning algorithm, introducing a dynamic weight adaptation mechanism into the knowledge graph structure, wherein the dynamic weight adaptation mechanism calculates the association strength between nodes to obtain the importance weight of the node, and constructs an edge weight matrix based on the importance weight of the node, wherein the edge weight matrix represents the strength of the propagation relationship between nodes in the knowledge graph;

[0027] Extracting corresponding feature vectors from the visual anomaly data and the device status data based on the edge weight matrix, combining the feature vectors with the importance weights of the nodes to calculate a correlation matrix between features, generating a unified feature representation based on the correlation matrix, and updating the unified feature representation as a node attribute to the corresponding node in the knowledge graph structure;

[0028] A root cause inference path is constructed based on the updated node attributes and the edge weight matrix. Taking the detected abnormal node as the starting point, the edge weight matrix is ​​used to calculate the multi-hop propagation probability from the abnormal node to each candidate root cause node. The multi-hop propagation probability is combined with the attribute similarity between nodes to calculate the confidence score of each root cause inference path. The node corresponding to the root cause inference path with the highest confidence score is selected as the root cause analysis result.

[0029] The root cause analysis results are input into the policy network to generate inspection task priority scores. Inspection routes are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots, including:

[0030] Inputting the root cause analysis results into a policy network, calculating the impact scope and time urgency of the equipment failure based on the root cause analysis results, and generating an inspection task priority score based on the impact scope and time urgency, wherein the inspection task priority score represents the execution priority order of different inspection tasks;

[0031] Planning an inspection path according to the inspection task priority score, obtaining device distribution information and channel constraint information in the real-time layout structure of the data center, constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path;

[0032] Construct a two-layer interactive learning mechanism to continuously optimize the inspection strategy. In the task evaluation layer, actual inspection results are collected to update the weight matrix of the strategy network. In the path planning layer, the information transfer probability matrix is ​​optimized based on the inspection execution efficiency.

[0033] Based on the optimized inspection strategy, multi-robot collaborative scheduling is realized. The priority ranking of multiple inspection tasks is calculated according to the weight matrix of the updated strategy network. Inspection paths are allocated to different inspection robots based on the optimized information transfer probability matrix. The adaptive collaborative scheduling of inspection robots is realized through the joint decision-making of the priority ranking result and the result of the allocated inspection path.

[0034] Constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path includes:

[0035] Generate an adjacency matrix of node connections based on the device distribution information and the channel constraint information, input the query matrix, the key matrix, and the value matrix into a temporal attention calculation unit to obtain temporal correlation strength between nodes, and calculate the causal relationship strength between nodes based on the adjacency matrix and the temporal correlation strength;

[0036] A multi-head attention scoring network is constructed based on the causal relationship strength, and task status information is input into multiple parallel attention calculation units to obtain attention weights. The attention weights are used to calculate the initial priority score of the inspection task, and the propagation impact of the task priority is calculated in combination with the status information of adjacent nodes. The propagation impact and the initial priority score are weightedly combined to generate the inspection task priority score;

[0037] An initial information transfer probability matrix is ​​constructed based on the equipment distribution information and the channel constraint information. The initial information transfer probability matrix is ​​dynamically optimized based on the inspection task priority score to obtain the information transfer probability considering the task priority. The optimal inspection path is solved by a dynamic programming method using the information transfer probability and the inspection task priority score.

[0038] A second aspect of an embodiment of the present invention provides a data center inspection robot monitoring and analysis system based on machine vision, comprising:

[0039] The first unit is used to obtain monitoring video stream data collected by the data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data;

[0040] The second unit is configured to analyze the real-time image using a deep learning target detection model, identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features;

[0041] The third unit is used to build a multimodal data fusion analysis model, perform spatiotemporal registration of the visual abnormal state characteristics with the environmental parameter data, and use an improved graph reasoning algorithm to build a device abnormality knowledge graph. The improved graph reasoning algorithm uses a dynamic weight adaptation mechanism to achieve collaborative analysis of multi-source heterogeneous data and generate root cause analysis results;

[0042] The fourth unit is used to input the root cause analysis results into the strategy network to generate an inspection task priority score, plan the inspection path based on the inspection task priority score combined with the real-time layout structure of the data center, continuously optimize the inspection strategy through a two-layer interactive learning mechanism, and realize adaptive collaborative scheduling of inspection robots.

[0043] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0047] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0048] The beneficial effects of this application are as follows:

[0049] The machine vision-based data center inspection robot monitoring and analysis method provided by the present invention uses a deep learning target detection model to identify the display status of equipment indicator lights, cable connection status, and cabinet temperature display values, and combines environmental parameter data to achieve comprehensive monitoring of the status of data center equipment, thereby improving the accuracy and completeness of anomaly detection.

[0050] By constructing a multimodal data fusion analysis model and introducing an improved graph inference algorithm, we achieve spatiotemporal alignment and collaborative analysis of visual abnormality characteristics and environmental parameter data, improving the efficiency and accuracy of root cause analysis of equipment abnormalities and effectively reducing fault diagnosis time and the need for manual intervention.

[0051] The inspection task priority scoring mechanism and two-layer interactive learning mechanism based on root cause analysis results realize the intelligent planning of inspection paths and continuous optimization of inspection strategies, enabling inspection robots to perform adaptive collaborative scheduling according to the real-time status of the data center, greatly improving the efficiency and reliability of data center operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1This is a flow chart of a data center inspection robot monitoring and analysis method based on machine vision according to an embodiment of the present invention;

[0053] Figure 2 This is a bar chart comparing the performance of the data center equipment status detection model according to an embodiment of the present invention;

[0054] Figure 3 This is a flow chart of a method for constructing a device anomaly knowledge graph based on multimodal data fusion according to an embodiment of the present invention;

[0055] Figure 4 This is a logic block diagram of the improved graph reasoning algorithm according to an embodiment of the present invention;

[0056] Figure 5 This is a bar chart comparing the performance of the inspection path optimization method based on temporal attention and dynamic programming in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0058] The technical solution of the present invention is described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Figure 1 FIG. 1 is a flow chart of a data center inspection robot monitoring and analysis method based on machine vision according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Acquire monitoring video stream data collected by a data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data;

[0061] Analyzing the real-time image using a deep learning target detection model to identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features;

[0062] Construct a multimodal data fusion analysis model, perform spatiotemporal registration of the visual abnormality characteristics with the environmental parameter data, and construct a device abnormality knowledge graph using an improved graph reasoning algorithm. The improved graph reasoning algorithm enables collaborative analysis of multi-source heterogeneous data by introducing a dynamic weight adaptation mechanism to generate root cause analysis results.

[0063] The root cause analysis results are input into the strategy network to generate inspection task priority scores. The inspection paths are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots.

[0064] In an optional embodiment, the real-time image is analyzed using a deep learning target detection model to identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features including:

[0065] Use deep learning target detection models to extract device indicator light status, cable connection status, and cabinet temperature display values, and construct a cause-and-effect graph of device status.

[0066] Based on the device state causal graph, an attention mechanism is used to calculate feature weights for the indicator light area, cable area, and temperature display area in the real-time image, and the feature weights are weightedly fused with the visual features extracted by the deep learning object detection model to construct a multi-scale visual feature representation;

[0067] For abnormal states detected in the multi-scale visual feature representation, a visual feature counterfactual verification function is constructed. The visual feature counterfactual verification function performs abnormality verification by calculating the difference between the state recognition probability after image enhancement processing and the original state recognition probability. Based on the result of the abnormality verification, the visual similarity between the candidate abnormal area and the historical abnormal case is calculated to generate the visual abnormality attribution confidence.

[0068] An abnormality evolution path is constructed according to the visual abnormality attribution confidence, and an early warning score calculation function is designed based on the abnormality evolution path. The early warning score calculation function comprehensively evaluates the abnormality degree of the device state and the visual feature deviation to generate a visual abnormality state feature.

[0069] Constructing a causal graph of device status. The system uses the YOLOv5 object detection model to analyze real-time images collected from data center cabinets. The YOLOv5 model was trained on 5,000 annotated images, covering three object categories: indicator light status (normal green, warning yellow, fault red, off), cable connection status (loose, normal, disconnected), and temperature display values ​​(normal range 18-27°C, warning range 28-35°C, critical range >35°C). The model achieved a mean average precision (mAP@0.5) of 93.7% on the test set. Detection results include object category, location coordinates, confidence level, and status attributes. For example, for indicator light detection, the system not only identifies the indicator light's location but also analyzes its color status (green: normal operation; yellow: warning; red: fault; gray: power off). Based on the detection results, the system constructs a causal graph of device status, recording the dependencies between device components, such as the causal chain from "host power indicator off to network connection indicator off."

[0070] Based on the device state causal graph, the system uses an attention mechanism to calculate feature weights. For each detected area, the system calculates spatial and channel attention weights. The spatial attention mechanism generates an attention map through convolution operations, highlighting key areas such as indicator lights, cable connectors, and temperature displays. For example, the system assigns a weight of 0.3 to the green indicator light indicating normal operation, 0.7 to the yellow indicator light indicating a warning, and 0.9 to the red indicator light indicating a fault.

[0071] The channel attention mechanism calculates weights for the RGB channels separately. For the red indicator light, the R channel weight is 0.8, the G channel weight is 0.1, and the B channel weight is 0.1. Finally, the system combines the attention weights with the visual features extracted by YOLOv5 (a 256×256×256 feature map). Using the Feature Pyramid Network (FPN), the system constructs a multi-scale visual feature representation consisting of feature maps at 256×256, 128×128, and 64×64 scales to accommodate the detection needs of device components of varying sizes.

[0072] A counterfactual verification function for visual features is constructed for detected anomalies. When a suspected anomaly is detected, the system verifies it through image enhancement. For example, for loose cable detection, the system performs contrast enhancement (increase by 20%), brightness adjustment (increase by 15%), and sharpening (sharpening factor 1.5) on the original image before re-detecting the target. If the enhanced probability of identifying a loose cable increases from the original 0.75 to 0.92, a difference of 0.17, exceeding the preset threshold of 0.15, the image is confirmed to be a true anomaly.

[0073] The system also calculates the visual similarity between the current anomaly area and samples in the historical anomaly case library. This similarity is calculated using the cosine distance of the feature vectors. For example, the similarity between the currently detected red fault indicator light and a typical fault indicator light in a historical case is calculated to be 0.85. Based on the anomaly verification results and historical similarity, the system generates a confidence level for attributing the visual anomaly. For example, for the detected indicator light anomaly, the system assigns an attribution confidence level of 0.92.

[0074] The system constructs an anomaly evolution path based on the confidence level of visual anomaly attribution. By recording abnormal state changes in 10 consecutive image frames, such as a temperature increase from 26°C to 32°C accompanied by a change in the cooling fan indicator from green to yellow, the system identifies the potential evolution path of a cooling system failure. Based on the anomaly evolution path, the system designs a warning score calculation function to comprehensively evaluate the degree of device state abnormality and visual feature deviation.

[0075] The warning score function considers the following factors: anomaly duration (weight 0.3), anomaly severity (weight 0.4), similarity to historical cases (weight 0.2), and anomaly diffusion trend (weight 0.1). For example, for a red indicator light failure lasting 5 minutes, the warning score is calculated as 0.3 × (5 / 10) + 0.4 × 0.9 + 0.2 × 0.85 + 0.1 × 0.7 = 0.77. This exceeds the preset threshold of 0.7, and the system generates a high-priority visual anomaly feature alarm. The generated visual anomaly feature includes anomaly type, location information, anomaly confidence, warning score, and recommended handling measures, providing decision support for data center operations and maintenance personnel.

[0076] Figure 2 This is a bar chart comparing the performance of the data center equipment status detection model according to an embodiment of the present invention:

[0077] This figure compares the recognition accuracy of three model approaches across four different inspection scenarios. These include indicator light status recognition, cable connection status, temperature display value recognition, and comprehensive anomaly detection. For each scenario, the performance of the basic model (white bars), the attention-enhanced model (shaded bars), and the counterfactual verification-enhanced model (grid bars) are shown. The counterfactual verification-enhanced model achieved the best results in all scenarios, with accuracy rates of 92.5%, 91.5%, 90.0%, and 90.5%, respectively. The attention-enhanced model performed second, with accuracy rates of 89.0%, 87.5%, 85.0%, and 86.0%, respectively. The basic model performed the worst, with accuracy rates of 82.5%, 80.1%, 77.5%, and 79.0%, respectively. The data shows that the introduction of the attention mechanism and counterfactual verification significantly improves the model's recognition accuracy, with an average improvement of over 10 percentage points, validating the effectiveness of these two enhancement methods. In particular, the enhanced model exhibits a clear advantage over the basic model in the complex comprehensive anomaly detection task.

[0078] In an optional embodiment, constructing an abnormality evolution path according to the visual abnormality attribution confidence, and designing a warning score calculation function based on the abnormality evolution path includes:

[0079] Acquire a device abnormal state sequence, the device abnormal state sequence including an abnormal visual feature vector, a device state feature vector, and a state transition probability, calculate the device abnormal state sequence according to the visual abnormality attribution confidence, and generate a state transition weight;

[0080] Constructing a multidimensional feature fusion function based on the abnormal state sequence of the device, and performing weighted summation of the state transition weight and the multidimensional feature fusion function to obtain a fusion feature representation;

[0081] A hierarchical early warning score calculation model is constructed using the fused feature representation, the feature distance between the abnormal visual feature vector and the normal state benchmark is calculated, the feature distance is multiplied by the device state feature vector to obtain an abnormality degree score, the rate of change of the state transition probability is calculated and combined with the time attenuation factor to generate a trend score, and at the comprehensive evaluation layer, the abnormality degree score, the trend score, and the confidence score are weightedly fused to generate a warning score calculation function.

[0082] In industrial production environments, equipment operation generates a large amount of visual data. Analyzing this data can help identify potential anomalies early. This embodiment provides a method for constructing anomaly evolution paths and designing a warning score calculation function based on visual anomaly attribution confidence.

[0083] Obtaining a sequence of device abnormal states is fundamental to constructing an abnormality evolution path. This sequence consists of three key components: an abnormal visual feature vector, a device state feature vector, and a state transition probability. The abnormal visual feature vector is extracted from device monitoring images using a deep learning model and can be represented as a 128-dimensional vector. The device state feature vector contains sensor data such as temperature, vibration, and sound, and is typically a 32-dimensional vector. The state transition probability describes the probability of the device transitioning from one state to another.

[0084] Calculating the sequence of abnormal equipment states based on visual anomaly attribution confidence requires the use of a pre-trained visual attribution model. This model, based on an attention mechanism, automatically locates abnormal regions in an image and assigns an attribution confidence score. For example, when a bearing on a production line is worn, the model identifies the worn area and assigns a confidence score of 0.85. This confidence score is then associated with each state in the current equipment state sequence to generate a state transition weight. Specifically, anomalies with a confidence score above 0.7 are assigned a higher state transition weight (e.g., 0.8-1.0), anomalies with a confidence score between 0.3-0.7 are assigned a medium weight (0.4-0.7), and anomalies with a confidence score below 0.3 are assigned a lower weight (0.1-0.3).

[0085] Constructing a multidimensional feature fusion function based on the equipment abnormal state sequence is a key step in achieving effective early warning. This fusion function uses an adaptive weighting mechanism to dynamically adjust the weights of different features during the fusion process based on their importance. Specifically, the abnormal visual feature vector and the equipment state feature vector are first converted to the same dimensional space, such as 64-dimensional space, through a nonlinear mapping. An attention network is then used to calculate the importance score of each feature. Finally, a weighted sum of the features is performed based on these scores. For example, in a bearing abnormality detection case, the visual feature was assigned a weight of 0.75, the vibration feature a weight of 0.15, and the temperature feature a weight of 0.10. The state transition weight (e.g., 0.82) is then weighted and summed with the result of the multidimensional feature fusion function to produce a fused feature representation that more comprehensively reflects the current abnormal state of the equipment and its evolving trends.

[0086] A hierarchical warning score calculation model is constructed using fused feature representation. This model consists of three evaluation levels: anomaly level assessment, trend assessment, and comprehensive assessment. In the anomaly level assessment, the feature distance between the abnormal visual feature vector and the normal state baseline is calculated. Specifically, cosine similarity is used to calculate the difference between the current feature and the normal baseline feature. The normal baseline feature is obtained through statistical analysis of a large amount of normal operating data. For example, the cosine similarity between the current feature of a bearing and the normal baseline is 0.65, indicating a certain difference, and the feature distance is 0.35. This feature distance is multiplied by the device status feature vector to obtain the anomaly level score. If the temperature anomaly value in the device status feature vector is 0.8, the anomaly level score is 0.28.

[0087] At the trend assessment layer, the rate of change of the state transition probability is calculated and combined with a time decay factor to generate a trend score. The rate of change of the state transition probability is calculated by comparing the state transition probabilities at multiple consecutive time points. For example, the state transition probabilities of a device in the last five detection cycles were 0.1, 0.15, 0.25, 0.4, and 0.55, respectively, showing a clear upward trend with a rate of change of 0.45. The time decay factor uses an exponential decay function to give more weight to recent state changes. If the decay rate is set to 0.9, the most recent state change has a weight of 1, the previous one has a weight of 0.9, the previous one has a weight of 0.81, and so on. Combining the rate of change and the decay factor, the calculated trend score is 0.38.

[0088] The confidence score directly uses the confidence level of visual anomaly attribution, such as 0.85 in the aforementioned example. At the comprehensive assessment layer, the anomaly degree score, trend score, and confidence score are weighted and fused to generate an early warning score. The weights of the three scores can be dynamically adjusted according to the needs of different production scenarios. In the field of high-precision manufacturing, more emphasis is placed on the degree of anomaly, which is given a weight of 0.5, and the trend score and confidence score each account for 0.25; while in the continuous production process, more attention is paid to trend changes, and the trend score is given a weight of 0.5, and the other two each account for 0.25. In the aforementioned bearing case, if an equal weight of 0.33 is used, the final early warning score is 0.33×0.28+0.33×0.38+0.33×0.85≈0.50, indicating that the equipment is in a medium-risk state and requires key monitoring.

[0089] The warning score calculation function constructed through the above steps can comprehensively consider the current abnormal status of the equipment, historical evolution trends, and the credibility of abnormal judgments, providing a scientific basis for equipment maintenance decisions and effectively preventing the occurrence of equipment failures.

[0090] In an optional embodiment, constructing a multimodal data fusion analysis model, performing spatiotemporal registration of the visual abnormal state features with the environmental parameter data, and constructing a device abnormality knowledge graph using an improved graph inference algorithm includes:

[0091] Constructing a multimodal data fusion analysis model, inputting the visual abnormal state feature and the environmental parameter data into the multimodal data fusion analysis model for spatiotemporal registration, and extracting the timestamp identifier and spatial position identifier of the visual abnormal state feature and the environmental parameter data;

[0092] Constructing a temporal correlation matrix based on the timestamp identifier, constructing a spatial adjacency matrix based on the spatial position identifier, and performing weighted fusion of the temporal correlation matrix and the spatial adjacency matrix to generate a spatiotemporal registration matrix;

[0093] An improved graph reasoning algorithm is used to construct a device anomaly knowledge graph, which includes an entity node layer, an attribute node layer, and a relationship edge set. Entity nodes representing different devices are constructed in the entity node layer, and attribute nodes representing device abnormal states are constructed in the attribute node layer. Entity relationship edges of the entity nodes are constructed based on the spatiotemporal registration matrix, and attribute relationship edges of the attribute nodes are constructed based on device state transition rules. The improved graph reasoning algorithm is used to iteratively update the entity relationship edges and the attribute relationship edges to generate a device anomaly knowledge graph with a hierarchical structure.

[0094] like Figure 3 As shown, the method includes:

[0095] The multimodal data fusion analysis model is constructed to spatially and temporally align visual anomaly characteristics with environmental parameter data, and utilizes an improved graph inference algorithm to construct a knowledge graph of device anomalies. This method achieves comprehensive perception and analysis of device anomalies through multiple key steps.

[0096] When building a multimodal data fusion analysis model, the first step is to collect visual anomaly characteristics and environmental parameter data. Visual anomaly characteristics can be captured by high-definition cameras deployed at industrial sites, capturing images of equipment exteriors. For example, visual anomaly characteristics such as peeling paint on transformers and loose joints within substations can be detected. Environmental parameter data, including temperature, humidity, and vibration, is acquired through various sensors. This collected data is then fed into the multimodal data fusion analysis model for spatiotemporal registration.

[0097] The first step in spatiotemporal registration is to extract timestamps and spatial locations. For visual data, each image frame includes the acquisition timestamp and camera position coordinates. For environmental parameter data, each record includes the measurement time and sensor installation location. For example, image data for a transformer includes a timestamp of "2023-04-15 14:30:25" and spatial coordinates of "X=105.36, Y=78.92, Z=2.35," while temperature sensor data for the same area includes a timestamp of "2023-04-15 14:30:20" and spatial coordinates of "X=105.40, Y=78.95, Z=2.38."

[0098] Construct a time series correlation matrix. This matrix describes the degree of correlation between different data points along the time dimension. Time series correlation can be determined by calculating the difference in timestamps. The smaller the difference, the stronger the correlation. For example, two data points with a time difference of 5 seconds can be assigned a correlation weight of 0.95; for data points with a time difference of 60 seconds, the correlation weight drops to 0.6. In practice, the time correlation decay function can be adjusted based on the characteristics of different devices.

[0099] Construct a spatial adjacency matrix, which represents the degree of association between different data points in the spatial dimension. Adjacency can be calculated using Euclidean distance or other spatial distance metrics. For example, two data points within a spatial distance of 0.5 meters can be assigned a spatial association weight of 0.9; for data points within a distance of 5 meters, the spatial association weight drops to 0.3. In real industrial environments, spatial association also needs to take into account the physical layout of the equipment and obstructions.

[0100] The temporal correlation matrix and the spatial adjacency matrix are weighted and fused to generate a spatiotemporal registration matrix. This weighting can be achieved using a linear combination, for example, setting a weight of 0.6 for the time dimension and 0.4 for the spatial dimension to calculate a comprehensive correlation score. In practice, these weights can be optimized based on specific scenarios. For example, for devices with rapidly changing temperatures, the weight of the time dimension may need to be increased.

[0101] Using an improved graph inference algorithm to construct a knowledge graph for equipment anomalies is a key step. The knowledge graph consists of three main components: an entity node layer, an attribute node layer, and a relationship edge set. In the entity node layer, entity nodes are constructed to represent different devices. Each device corresponds to an entity node, which records basic information such as the device ID, type, and installation location. For example, transformer A can be represented as an entity node with attributes such as "ID=TR001," "Type=Oil-immersed Transformer," and "Location=Distribution Room Area B."

[0102] At the attribute node layer, construct attribute nodes to represent device abnormalities. Attribute nodes contain information such as the abnormality type, severity, and duration. For example, "Oil temperature abnormality" can be represented as an attribute node with attributes such as "Abnormality Type = Temperature Exceeded," "Severity = Medium," and "Duration = 30 Minutes."

[0103] Entity relationship edges are constructed between entity nodes based on the spatiotemporal registration matrix. Entity relationship edges describe relationships between different devices, such as physical connections and functional dependencies. For example, an "electrically connected" relationship can be established between transformer A and circuit breaker B, with a weight of 0.85, indicating that the two are electrically closely connected.

[0104] Attribute relationship edges are constructed for attribute nodes based on device state transition rules. Attribute relationship edges describe the causal and transformational relationships between different abnormal states. For example, a "cause" relationship can be established between "abnormal oil temperature" and "increased pressure" with a probability of 0.75, indicating that abnormal oil temperature causes increased pressure.

[0105] An improved graph inference algorithm is used to iteratively update entity and attribute relationship edges. This algorithm combines an attention mechanism with propagation rules to iteratively compute node features and edge weights. Specifically, in each iteration, the algorithm first calculates weighted features for neighboring nodes, then updates the representation of the central node, adjusting edge weights based on changes in node states. For example, if an abnormal oil temperature in transformer A is found to be highly correlated with a tripping event in circuit breaker B, the association weight between these two nodes is increased.

[0106] After multiple iterations, a hierarchical knowledge graph of device anomalies is generated. This graph not only displays the physical connections between devices but also illustrates the propagation paths and causal chains of anomalies. By analyzing features such as node centrality and community structure within the knowledge graph, key devices and core fault sources can be identified. For example, if multiple device anomalies are linked to a specific node, that node is the root cause and should be prioritized for inspection and repair.

[0107] This method of combining multimodal data fusion with knowledge graphs can effectively capture the spatiotemporal characteristics and association patterns of abnormal equipment states, providing strong technical support for predictive maintenance and fault diagnosis of industrial equipment.

[0108] In an optional embodiment, the improved graph reasoning algorithm implements collaborative analysis of multi-source heterogeneous data by introducing a dynamic weight adaptation mechanism, and generates root cause analysis results including:

[0109] Constructing a knowledge graph structure of the improved graph reasoning algorithm, introducing a dynamic weight adaptation mechanism into the knowledge graph structure, wherein the dynamic weight adaptation mechanism calculates the association strength between nodes to obtain the importance weight of the node, and constructs an edge weight matrix based on the importance weight of the node, wherein the edge weight matrix represents the strength of the propagation relationship between nodes in the knowledge graph;

[0110] Extracting corresponding feature vectors from the visual anomaly data and the device status data based on the edge weight matrix, combining the feature vectors with the importance weights of the nodes to calculate a correlation matrix between features, generating a unified feature representation based on the correlation matrix, and updating the unified feature representation as a node attribute to the corresponding node in the knowledge graph structure;

[0111] A root cause inference path is constructed based on the updated node attributes and the edge weight matrix. Taking the detected abnormal node as the starting point, the edge weight matrix is ​​used to calculate the multi-hop propagation probability from the abnormal node to each candidate root cause node. The multi-hop propagation probability is combined with the attribute similarity between nodes to calculate the confidence score of each root cause inference path. The node corresponding to the root cause inference path with the highest confidence score is selected as the root cause analysis result.

[0112] like Figure 4 As shown, the method includes:

[0113] Build a knowledge graph structure consisting of device nodes, status nodes, fault nodes, and relationship edges. For example, in an industrial production line, each device such as a pump, valve, or sensor can be considered a device node, temperature anomalies and pressure fluctuations can be considered status nodes, and pipeline leaks and motor overheating can be considered fault nodes. Nodes are connected through relationship edges such as "cause," "associated with," and "belongs to," forming the initial knowledge graph.

[0114] A dynamic weight adaptation mechanism is introduced into the constructed knowledge graph structure. This mechanism obtains the importance weight of the node by calculating the association strength between nodes. In specific implementation, for a certain node A, the connection frequency with its adjacent nodes is calculated. For example, if node A and node B appear 50 times together in the historical data, and appear 10 times together with node C, then the association strength of node B with node A is 0.83 (50 / (50+10)). Repeating this process for all nodes in the entire graph can obtain the importance weight value of each node. For example, in an actual industrial scenario, the pump node has an importance weight of 0.75, while an auxiliary sensor node has an importance weight of only 0.3.

[0115] An edge weight matrix is ​​constructed based on the node importance weights. This matrix represents the strength of the communication relationships between nodes in the knowledge graph. For an edge connecting node i and node j, its weight calculation depends on the importance weights of the two nodes at both ends and their historical co-occurrence frequency. For example, for a piece of production equipment, if the pressure sensor node (importance weight 0.8) and the valve fault node (importance weight 0.7) have a historical co-occurrence frequency of 0.6, the edge weight between them can be calculated as 0.8 × 0.7 × 0.6 = 0.336. Entering all edge weights into the matrix yields a complete edge weight matrix.

[0116] Corresponding feature vectors are extracted from visual anomaly data and equipment status data based on the edge weight matrix. For visual anomaly data, image processing technology is used to extract features such as edge features, texture features, and color distribution features. For example, for image data on the surface of a device, the extracted features may include the temperature distribution features of the device surface (hot spot area accounts for 25%, temperature gradient value is 0.4), surface crack features (crack length 5mm, width 0.8mm), etc. For equipment status data, features such as equipment vibration frequency, current fluctuation, and temperature change rate are extracted. For example, for a certain pump equipment, the extracted status features may include vibration frequency (23Hz), current fluctuation (±2A), bearing temperature change rate (1.5℃ / hour), etc.

[0117] The extracted feature vectors are combined with the node importance weights to calculate the feature correlation matrix. For each pair of features, their correlation is calculated and multiplied by the corresponding node importance weight. For example, if the original correlation coefficient between the equipment vibration frequency feature and the bearing temperature change rate feature is 0.75, and the corresponding node importance weights are 0.8 and 0.7, respectively, the correlation value of this pair of features in the correlation matrix is ​​0.75 × 0.8 × 0.7 = 0.42.

[0118] Generate a unified feature representation based on the correlation matrix. All features are weighted and fused according to their correlation to obtain a feature representation with unified dimensions. For example, for a device node, its unified feature representation can be a 128-dimensional vector containing device state parameters and visual features, where the first 64 dimensions represent device state parameters such as temperature, pressure, and vibration, and the last 64 dimensions represent visual features such as color, texture, and shape. The generated unified feature representation is updated as a node attribute to the corresponding node in the knowledge graph structure.

[0119] The root cause inference path is constructed based on the updated node attributes and edge weight matrix. Starting from a detected anomaly node, such as a device's temperature anomaly node, the edge weight matrix is ​​used to calculate the multi-hop propagation probability from the anomaly node to each candidate root cause node. Multi-hop propagation probabilities are calculated by accumulating edge weights. For example, the propagation probability of a three-hop path from the temperature anomaly node to the bearing wear node is 0.336 × 0.285 × 0.412 = 0.0395.

[0120] The confidence score for each root cause inference path is calculated by combining the multi-hop propagation probability with the attribute similarity between nodes. Attribute similarity between nodes is calculated by comparing their unified feature representations. For example, if the attribute similarity between the temperature anomaly node and the bearing wear node is 0.78, the final confidence score for the path is 0.0395 × 0.78 = 0.03081.

[0121] The node corresponding to the root cause reasoning path with the highest confidence score is selected as the root cause analysis result. In this case, the system discovered that the root cause of the temperature anomaly was bearing wear (confidence 0.03081), not motor overheating (confidence 0.02755) or cooling system failure (confidence 0.02103). The system ultimately outputs bearing wear as the root cause of the temperature anomaly and provides the corresponding reasoning path and confidence score.

[0122] In an optional embodiment, the root cause analysis results are input into a policy network to generate inspection task priority scores, and inspection routes are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots, including:

[0123] Inputting the root cause analysis results into a policy network, calculating the impact scope and time urgency of the equipment failure based on the root cause analysis results, and generating an inspection task priority score based on the impact scope and time urgency, wherein the inspection task priority score represents the execution priority order of different inspection tasks;

[0124] Planning an inspection path according to the inspection task priority score, obtaining device distribution information and channel constraint information in the real-time layout structure of the data center, constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path;

[0125] Construct a two-layer interactive learning mechanism to continuously optimize the inspection strategy. In the task evaluation layer, actual inspection results are collected to update the weight matrix of the strategy network. In the path planning layer, the information transfer probability matrix is ​​optimized based on the inspection execution efficiency.

[0126] Based on the optimized inspection strategy, multi-robot collaborative scheduling is realized. The priority ranking of multiple inspection tasks is calculated according to the weight matrix of the updated strategy network. Inspection paths are allocated to different inspection robots based on the optimized information transfer probability matrix. The adaptive collaborative scheduling of inspection robots is realized through the joint decision-making of the priority ranking result and the result of the allocated inspection path.

[0127] The root cause analysis results are input into the policy network to generate inspection task priority scores. Inspection routes are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism.

[0128] The root cause analysis results are input into the policy network to generate inspection task priority scores. Specifically, a deep neural network is constructed as the policy network, consisting of an input layer, multiple hidden layers, and an output layer. The input layer receives the multidimensional feature vector from the root cause analysis, including features such as the fault type code (e.g., 0 for abnormal temperature, 1 for abnormal humidity, 2 for airflow obstruction, 3 for abnormal power supply), the fault level (e.g., a numerical value from 0 to 5, with higher values ​​indicating higher fault levels), and historical fault frequency (e.g., the number of failures of a particular device within 30 days). The hidden layer uses the ReLU activation function for nonlinear mapping, and the output layer generates a priority score between 0 and 100.

[0129] When assessing the impact of a device failure, the system detects the number of other devices that are dependent on the failed device. For example, if a core switch failure affects 50 servers, its impact range is 50; a typical server failure affects only itself, with an impact range of 1. Time urgency is calculated based on the fault's development trend and the predicted time window for fault expansion. For example, a server with a rapidly rising temperature has an urgency of 90, while a stable, non-critical alarm has an urgency of 30. Based on these input features, the policy network outputs a comprehensive priority score, such as a core network device overheating fault with a score of 95 and a non-critical path server fan alarm with a score of 45.

[0130] The inspection route is planned based on the generated inspection task priority score. The system first obtains the real-time layout structure information of the data center, including the three-dimensional coordinate position of the equipment, cabinet arrangement, channel direction, etc. For the equipment distribution information, the system records the location coordinates (x, y, z) of each device to be inspected, such as device A is located at (10, 20, 15) and device B is located at (10, 25, 15). For channel constraint information, the system marks the passable areas and prohibited areas, such as the cold channel is marked as 1, the hot channel is marked as 0, and the special area with directional restrictions is marked as 2. Based on this information, an information transfer probability matrix is ​​constructed, which represents the possibility of transfer from one device location to another.

[0131] The transition probability between adjacent cabinet devices is 0.9, the transition probability between devices spanning multiple channels is 0.3, and the transition probability within a restricted area is 0. By combining the transition probability matrix with the inspection task priority scores, an improved ant colony algorithm is used to generate the optimal inspection path. For example, for three devices with priority scores of 95, 75, and 60, respectively, the system calculates the total utility of various access sequences. For example, the utility of the path [Device 1 → Device 3 → Device 2] is 950, and the utility of the path [Device 1 → Device 2 → Device 3] is 920. The access sequence with the highest utility value is ultimately selected as the optimal inspection path.

[0132] A two-tier interactive learning mechanism is built to continuously optimize inspection strategies. At the task evaluation layer, the system records the actual results of each inspection, including indicators such as fault confirmation rate (e.g., 85% of devices predicted as high-risk faults actually have serious problems), false alarm rate (e.g., 15% of alarms are false alarms), and average processing time (e.g., the average inspection time per fault point is 3.5 minutes).

[0133] Based on this real-world data, the policy network's weight matrix is ​​updated using gradient descent. For example, if a certain type of fault is found to be over-prioritized, the corresponding feature's weight is reduced from 0.75 to 0.65. At the path planning layer, the system calculates the execution efficiency of inspection routes, including path completion time (e.g., 105 minutes compared to an expected 90 minutes) and path adherence rate (e.g., 80% of the planned path is accurately executed). Based on these statistical results, the system updates the information transition probability matrix. If traversing a certain channel area is found to be taking significantly longer than expected, the transition probability for that area is reduced from 0.7 to 0.5. Through this two-layer learning mechanism, the system continuously optimizes inspection strategies based on real-world operational experience.

[0134] Multi-robot collaborative scheduling is achieved based on the optimized inspection strategy. The system recalculates the priority ranking of all currently pending inspection tasks based on the updated policy network weight matrix. For example, if there are 20 inspection points in a data center, the calculated priority ranking from high to low is [Device 7, Device 15, Device 3, ...]. Based on the optimized information transfer probability matrix, the system uses a clustering algorithm to group geographically close inspection points and then assigns these groups to different inspection robots.

[0135] For example, if a data center has three inspection robots, the system divides 20 inspection points into three zones: Robot A covers seven points in the northeast zone, Robot B covers six points in the southwest zone, and Robot C covers seven points in the central zone. During task assignment, the system considers both priority sorting and spatial distribution to ensure that high-priority tasks are processed promptly while avoiding path intersections and conflicts between robots. Ultimately, the system achieves adaptive collaborative scheduling of inspection robots by combining priority sorting with inspection path assignment results, improving overall inspection efficiency.

[0136] In an optional embodiment, constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path includes:

[0137] Generate an adjacency matrix of node connections based on the device distribution information and the channel constraint information, input the query matrix, the key matrix, and the value matrix into a temporal attention calculation unit to obtain temporal correlation strength between nodes, and calculate the causal relationship strength between nodes based on the adjacency matrix and the temporal correlation strength;

[0138] A multi-head attention scoring network is constructed based on the causal relationship strength, and task status information is input into multiple parallel attention calculation units to obtain attention weights. The attention weights are used to calculate the initial priority score of the inspection task, and the propagation impact of the task priority is calculated in combination with the status information of adjacent nodes. The propagation impact and the initial priority score are weightedly combined to generate the inspection task priority score;

[0139] An initial information transfer probability matrix is ​​constructed based on the equipment distribution information and the channel constraint information. The initial information transfer probability matrix is ​​dynamically optimized based on the inspection task priority score to obtain the information transfer probability considering the task priority. The optimal inspection path is solved by a dynamic programming method using the information transfer probability and the inspection task priority score.

[0140] When generating a node connection adjacency matrix based on device distribution information and channel constraint information, the system first obtains the location coordinates of all devices within the factory and the connection channel constraints. For example, in a chemical plant with 10 devices, the system records the two-dimensional coordinate position of each device, such as device 1 at (10, 15) and device 2 at (25, 30). The system also obtains channel constraint information, including which devices have direct channel connections and which channels are temporarily closed for maintenance or safety reasons. Based on this information, the system constructs a 10×10 adjacency matrix. The element aij in the matrix indicates whether there is a direct channel connection between device i and device j. If there is a direct channel connection, the value is 1; otherwise, it is 0.

[0141] When performing temporal attention calculations, the system first extracts the query matrix, key matrix, and value matrix for each device from historical inspection data. The query matrix represents the features of the current node, while the key matrix and value matrix represent the features of the target node. Specifically, the system extracts features such as inspection frequency, failure rate, and last inspection time for each device from historical inspection records to construct a feature matrix with dimensions of 10×6 (assuming 6 features are extracted for each device).

[0142] This feature matrix is ​​linearly transformed to generate a query matrix, a key matrix, and a value matrix, each with a 10×6 dimension. The system then inputs these matrices into the temporal attention calculation unit to calculate the temporal correlation strength between nodes. The temporal correlation strength is represented as a 10×10 matrix, where the element tij represents the degree of temporal correlation between device i and device j, with a value ranging from 0 to 1.

[0143] The causal strength between nodes is calculated based on the adjacency matrix and the temporal correlation strength. This involves element-wise multiplication of the adjacency matrix and the temporal correlation strength matrix to produce a new 10×10 matrix representing the combined causal strength, taking into account both physical connections and temporal correlations. For example, if a direct channel exists between device 1 and device 2 (adjacency matrix value is 1) and the temporal correlation strength is 0.8, then their causal strength is 0.8. If there is no direct channel connection, then the causal strength is 0.

[0144] When building a multi-head attention scoring network based on causal strength, the system creates four parallel attention computation units (using four-head attention as an example). The system takes as input the task status information of each device, including features such as device operating time, fault risk level, and maintenance urgency, to form a 10×8 task status matrix (assuming each device has eight status features).

[0145] This task state matrix is ​​fed into four attention computation units, each of which independently computes attention weights. Within each attention head, the system maps the task state matrix into a query matrix, a key matrix, and a value matrix, each with dimensions of 10×2 (8-dimensional features divided into four heads, each with 2 dimensions). The system uses causal strength to constrain the attention computation process, ensuring that only nodes with causal relationships generate valid attention weights.

[0146] Attention weights are used to calculate the initial priority scores for inspection tasks. Specifically, the system concatenates the outputs from the four attention heads and applies linear mapping to obtain the initial priority scores for the ten devices, for example, device 1 scores 85, device 2 scores 63, and so on. The system combines the status information of neighboring nodes to calculate the propagation effect of task priorities, taking into account the impact of the status of each device's neighbors on its priority.

[0147] For example, if one of device 1's neighbors is a high-risk device, device 1's priority is increased. The system calculates the propagation impact value using a graph propagation mechanism. For example, the propagation impact value for device 1 is +5, while the propagation impact value for device 2 is -2. The system weights the propagation impact and the initial priority score to generate the final inspection task priority score, with a weighting ratio of 0.8:0.2. For example, the final priority score for device 1 is 85 × 0.8 + 5 × 0.2 = 69, and the final priority score for device 2 is 63 × 0.8 + (-2) × 0.2 = 50.

[0148] Based on device distribution and channel constraint information, an initial information transfer probability matrix is ​​constructed. This matrix represents the probability of a transfer from one device to another. The system calculates transfer probabilities based on the distance between devices and channel constraints. For example, transfer probability is higher between devices that are close and connected by a channel. The system generates a 10×10 initial transfer probability matrix, where element pij represents the probability of a transfer from device i to device j, and the sum of all probabilities is 1.

[0149] The initial information transition probability matrix is ​​dynamically optimized based on the inspection task priority scores. Specifically, the system normalizes the task priority scores to obtain priority weights. The system uses the priority weights to adjust the initial transition probabilities, increasing the transition probability for high-priority devices and decreasing the transition probability for low-priority devices.

[0150] For example, if device j has a priority score of 50 and a normalized weight of 0.06, the adjusted probability of a transfer from device i to device j is pij × (1 + 0.06). After the system adjusts the transfer probabilities between all devices, it normalizes each row to ensure that the sum of the probabilities of a transfer from any device to any other device is 1.

[0151] The system uses information transition probabilities and inspection task priority scores to solve the optimal inspection path through dynamic programming. The system defines a state as the currently accessed device and the set of previously visited devices, and a transition as moving from the current device to the next unvisited device. The system uses the optimized transition probability matrix as the state transition probability, with maximizing the total benefit of accessing high-priority devices as the objective function. The system solves the optimal path using a dynamic programming algorithm, obtaining a device access sequence of length 10, such as [1, 3, 5, 7, 9, 10, 8, 6, 4, 2]. This indicates the order in which inspectors should visit each device to optimize inspection efficiency.

[0152] Figure 5 This is a bar chart comparing the performance of the inspection path optimization method based on temporal attention and dynamic programming according to an embodiment of the present invention:

[0153] This figure compares the performance of three different inspection path planning methods across four evaluation dimensions: path length optimization, time efficiency improvement, task prioritization, and overall performance score. The white bars in the figure represent the performance of the traditional path planning method, the slashed bars represent the temporal attention enhancement method, and the grid bars represent the performance of the dynamic programming optimization method. Data analysis shows that the dynamic programming optimization method achieved the best performance across all evaluation dimensions, achieving 92.5% in path length optimization, 91.5% in time efficiency improvement, 94.0% in task prioritization, and 95.0% in overall performance. The temporal attention enhancement method performed second best, with performance scores of 87.5%, 86.5%, 86.0%, and 88.5% in the four dimensions, respectively. The traditional path planning method performed the worst, with performance scores of 79.0%, 81.0%, 77.0%, and 80.1% in each dimension, respectively. The data shows that by introducing the temporal attention mechanism and dynamic programming optimization strategy, the overall performance of inspection path planning has been significantly improved, with an average improvement of approximately 15 percentage points. Especially in terms of comprehensive performance scoring, the optimized method has more obvious advantages than the traditional method, which fully demonstrates the effectiveness of the proposed optimization method.

[0154] A second aspect of an embodiment of the present invention provides a data center inspection robot monitoring and analysis system based on machine vision, comprising:

[0155] The first unit is used to obtain monitoring video stream data collected by the data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data;

[0156] The second unit is configured to analyze the real-time image using a deep learning target detection model, identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features;

[0157] The third unit is used to build a multimodal data fusion analysis model, perform spatiotemporal registration of the visual abnormal state characteristics with the environmental parameter data, and use an improved graph reasoning algorithm to build a device abnormality knowledge graph. The improved graph reasoning algorithm uses a dynamic weight adaptation mechanism to achieve collaborative analysis of multi-source heterogeneous data and generate root cause analysis results;

[0158] The fourth unit is used to input the root cause analysis results into the strategy network to generate an inspection task priority score, plan the inspection path based on the inspection task priority score combined with the real-time layout structure of the data center, continuously optimize the inspection strategy through a two-layer interactive learning mechanism, and realize adaptive collaborative scheduling of inspection robots.

[0159] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including:

[0160] processor;

[0161] a memory for storing processor-executable instructions;

[0162] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0163] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0164] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data center inspection robot monitoring and analysis method based on machine vision, characterized in that: include: Acquire monitoring video stream data collected by a data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data; Analyzing the real-time image using a deep learning target detection model to identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features; A multimodal data fusion analysis model is constructed to perform spatiotemporal registration of the visual abnormality characteristics with the environmental parameter data. An improved graph reasoning algorithm is used to construct a device abnormality knowledge graph. The improved graph reasoning algorithm enables collaborative analysis of multi-source heterogeneous data by introducing a dynamic weight adaptation mechanism to generate root cause analysis results, including: Constructing a multimodal data fusion analysis model, inputting the visual abnormal state feature and the environmental parameter data into the multimodal data fusion analysis model for spatiotemporal registration, and extracting the timestamp identifier and spatial position identifier of the visual abnormal state feature and the environmental parameter data; Constructing a temporal correlation matrix based on the timestamp identifier, constructing a spatial adjacency matrix based on the spatial position identifier, and performing weighted fusion of the temporal correlation matrix and the spatial adjacency matrix to generate a spatiotemporal registration matrix; An improved graph reasoning algorithm is used to construct a device anomaly knowledge graph, the device anomaly knowledge graph comprising an entity node layer, an attribute node layer, and a relationship edge set. Entity nodes representing different devices are constructed in the entity node layer, and attribute nodes representing device abnormal states are constructed in the attribute node layer. Entity relationship edges of the entity nodes are constructed based on the spatiotemporal registration matrix, and attribute relationship edges of the attribute nodes are constructed based on device state transition rules. The improved graph reasoning algorithm is used to iteratively update the entity relationship edges and the attribute relationship edges to generate a device anomaly knowledge graph with a hierarchical structure. The root cause analysis results are input into the strategy network to generate inspection task priority scores. The inspection paths are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots.

2. The method according to claim 1, characterized in that The real-time image is analyzed using a deep learning target detection model to identify the device indicator light status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal status features including: Use deep learning target detection models to extract device indicator light status, cable connection status, and cabinet temperature display values, and construct a cause-and-effect graph of device status. Based on the device state causal graph, an attention mechanism is used to calculate feature weights for the indicator light area, cable area, and temperature display area in the real-time image, and the feature weights are weightedly fused with the visual features extracted by the deep learning object detection model to construct a multi-scale visual feature representation; For abnormal states detected in the multi-scale visual feature representation, a visual feature counterfactual verification function is constructed. The visual feature counterfactual verification function performs abnormality verification by calculating the difference between the state recognition probability after image enhancement processing and the original state recognition probability. Based on the result of the abnormality verification, the visual similarity between the candidate abnormal area and the historical abnormal case is calculated to generate the visual abnormality attribution confidence. An abnormality evolution path is constructed according to the visual abnormality attribution confidence, and an early warning score calculation function is designed based on the abnormality evolution path. The early warning score calculation function comprehensively evaluates the abnormality degree of the device state and the visual feature deviation to generate a visual abnormality state feature.

3. The method according to claim 2, characterized in that Constructing an abnormality evolution path according to the visual abnormality attribution confidence, and designing an early warning score calculation function based on the abnormality evolution path includes: Acquire a device abnormal state sequence, the device abnormal state sequence including an abnormal visual feature vector, a device state feature vector, and a state transition probability, calculate the device abnormal state sequence according to the visual abnormality attribution confidence, and generate a state transition weight; Constructing a multidimensional feature fusion function based on the abnormal state sequence of the device, and performing weighted summation of the state transition weight and the multidimensional feature fusion function to obtain a fusion feature representation; A hierarchical early warning score calculation model is constructed using the fused feature representation, the feature distance between the abnormal visual feature vector and the normal state benchmark is calculated, the feature distance is multiplied by the device state feature vector to obtain an abnormality degree score, the rate of change of the state transition probability is calculated and combined with the time attenuation factor to generate a trend score, and at the comprehensive evaluation layer, the abnormality degree score, the trend score, and the confidence score are weightedly fused to generate a warning score calculation function.

4. The method according to claim 1, wherein The improved graph inference algorithm introduces a dynamic weight adaptation mechanism to achieve collaborative analysis of multi-source heterogeneous data, generating root cause analysis results including: Constructing a knowledge graph structure of the improved graph reasoning algorithm, introducing a dynamic weight adaptation mechanism into the knowledge graph structure, wherein the dynamic weight adaptation mechanism calculates the association strength between nodes to obtain the importance weight of the node, and constructs an edge weight matrix based on the importance weight of the node, wherein the edge weight matrix represents the strength of the propagation relationship between nodes in the knowledge graph; Extracting corresponding feature vectors from the visual anomaly data and the device status data based on the edge weight matrix, combining the feature vectors with the importance weights of the nodes to calculate a correlation matrix between features, generating a unified feature representation based on the correlation matrix, and updating the unified feature representation as a node attribute to the corresponding node in the knowledge graph structure; A root cause inference path is constructed based on the updated node attributes and the edge weight matrix. Taking the detected abnormal node as the starting point, the edge weight matrix is ​​used to calculate the multi-hop propagation probability from the abnormal node to each candidate root cause node. The multi-hop propagation probability is combined with the attribute similarity between nodes to calculate the confidence score of each root cause inference path. The node corresponding to the root cause inference path with the highest confidence score is selected as the root cause analysis result.

5. The method according to claim 1, characterized in that The root cause analysis results are input into the policy network to generate inspection task priority scores. Inspection routes are planned based on the inspection task priority scores and the real-time layout structure of the data center. The inspection strategy is continuously optimized through a two-layer interactive learning mechanism to achieve adaptive collaborative scheduling of inspection robots, including: Inputting the root cause analysis results into a policy network, calculating the impact scope and time urgency of the equipment failure based on the root cause analysis results, and generating an inspection task priority score based on the impact scope and time urgency, wherein the inspection task priority score represents the execution priority order of different inspection tasks; Planning an inspection path according to the inspection task priority score, obtaining device distribution information and channel constraint information in the real-time layout structure of the data center, constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path; Construct a two-layer interactive learning mechanism to continuously optimize the inspection strategy. In the task evaluation layer, actual inspection results are collected to update the weight matrix of the strategy network. In the path planning layer, the information transfer probability matrix is ​​optimized based on the inspection execution efficiency. Based on the optimized inspection strategy, multi-robot collaborative scheduling is realized. The priority ranking of multiple inspection tasks is calculated according to the weight matrix of the updated strategy network. Inspection paths are allocated to different inspection robots based on the optimized information transfer probability matrix. The adaptive collaborative scheduling of inspection robots is realized through the joint decision-making of the priority ranking result and the result of the allocated inspection path.

6. The method according to claim 5, characterized in that Constructing an information transfer probability matrix based on the device distribution information and the channel constraint information, and combining the information transfer probability matrix with the inspection task priority score to generate an optimal inspection path includes: Generate an adjacency matrix of node connections based on the device distribution information and the channel constraint information, input the query matrix, the key matrix, and the value matrix into a temporal attention calculation unit to obtain temporal correlation strength between nodes, and calculate the causal relationship strength between nodes based on the adjacency matrix and the temporal correlation strength; A multi-head attention scoring network is constructed based on the causal relationship strength, and task status information is input into multiple parallel attention calculation units to obtain attention weights. The attention weights are used to calculate the initial priority score of the inspection task, and the propagation impact of the task priority is calculated in combination with the status information of adjacent nodes. The propagation impact and the initial priority score are weightedly combined to generate the inspection task priority score; An initial information transfer probability matrix is ​​constructed based on the equipment distribution information and the channel constraint information. The initial information transfer probability matrix is ​​dynamically optimized based on the inspection task priority score to obtain the information transfer probability considering the task priority. The optimal inspection path is solved by a dynamic programming method using the information transfer probability and the inspection task priority score.

7. A data center inspection robot monitoring and analysis system based on machine vision, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to obtain monitoring video stream data collected by the data center inspection robot, extract real-time images of the data center cabinet area from the monitoring video stream data, and simultaneously collect temperature distribution data and equipment load data in the cabinet area as environmental parameter data; The second unit is configured to analyze the real-time image using a deep learning target detection model, identify the device indicator light display status, cable connection status, and cabinet temperature display value in the data center cabinet area, and generate visual abnormal state features; The third unit is used to build a multimodal data fusion analysis model, perform spatiotemporal registration of the visual abnormal state characteristics with the environmental parameter data, and use an improved graph reasoning algorithm to build a device abnormality knowledge graph. The improved graph reasoning algorithm uses a dynamic weight adaptation mechanism to achieve collaborative analysis of multi-source heterogeneous data and generate root cause analysis results; The fourth unit is used to input the root cause analysis results into the strategy network to generate an inspection task priority score, plan the inspection path based on the inspection task priority score combined with the real-time layout structure of the data center, continuously optimize the inspection strategy through a two-layer interactive learning mechanism, and realize adaptive collaborative scheduling of inspection robots.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images