Multi-modal information fusion intelligent inspection and fault positioning method for data center machine room

Abstract

The application provides a data center machine room multi-modal information fusion intelligent inspection and fault positioning method, belonging to the technical field of image analysis. In view of the problems that the existing machine room inspection is difficult to detect the color jump of indicator light, the loosening of lock buckle, the displacement of cable and other subtle changes, and the fault positioning precision is insufficient, the method constructs a three-dimensional semantic map by fusing laser radar SLAM, visual inertial odometry and depth camera data of a mobile inspection robot, projects and binds multi-modal inspection data such as visible light, infrared, ultrasonic and gas concentration to the equipment instance in the map; for the same equipment instance, align the historical and current inspection images to the same observation area for pixel-level difference comparison, detect subtle change events; bind the change event and the three-dimensional coordinates of the equipment instance, call the equipment knowledge graph to infer the fault reason, generate the maintenance work order with three-dimensional coordinates and fault reason, realize stereoscopic fault positioning, and can find hidden troubles in the early stage.

Description

Technical Field

[0001] This invention belongs to the field of image analysis technology, specifically relating to a method for intelligent inspection and fault location of data center computer rooms through multimodal information fusion. Background Technology

[0002] Data center equipment is densely packed and operates continuously for extended periods. Unplanned downtime often stems from subtle changes such as indicator light color changes, loose latches, cable displacement, and equipment surface deformation. These early signs are very faint and difficult to detect during manual inspections. Once they develop into apparent faults, they often result in irreversible business interruptions.

[0003] The existing inspection methods have two shortcomings in detecting the aforementioned subtle changes: First, the anomaly detection results are usually limited to two-dimensional image differences or single-modal threshold alarms, and cannot accurately correspond the change events to specific equipment instances and their spatial locations. Maintenance personnel can only know that there may be an anomaly in a certain area, and still need to check each unit on-site. Second, there is a lack of effective spatiotemporal alignment and difference comparison methods for subtle changes such as indicator light color changes, latch displacement, and cable displacement, making it difficult to reliably detect them at the micro-sign stage, and even more difficult to automatically generate maintenance work orders with precise spatial coordinates.

[0004] Therefore, how to achieve the detection and fault location of subtle changes with precise spatial coordinates at the equipment instance level during data center inspection is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the existing technology, a method for intelligent inspection and fault location of data center computer rooms through multimodal information fusion is provided, which is applied to a mobile inspection robot, including: A three-dimensional semantic map of the data center room is constructed. The three-dimensional semantic map is constructed by fusing LiDAR SLAM data, visual inertial odometry data and depth camera data, and includes device instances, the three-dimensional coordinates of the device instances and semantic labels. During the inspection process, multimodal inspection data is collected, including at least two of visible light images, infrared images, ultrasonic data, and gas concentration data. Based on the pose of the mobile inspection robot, the multimodal inspection data is projected onto the three-dimensional semantic map and bound to the corresponding device instance in the three-dimensional semantic map. For the same device instance in the three-dimensional semantic map, based on the three-dimensional coordinates of the device instance in the three-dimensional semantic map and the observation pose during historical and current inspections, the historical inspection images and the current inspection images are aligned to the same observation area of ​​the device instance, and pixel-level difference comparison is performed on the aligned images to detect subtle change events of the device instance. The subtle change events include at least one of the following: indicator light color change, latch displacement, cable displacement, and device surface deformation. The subtle change event is bound to the three-dimensional coordinates of the device instance in the three-dimensional semantic map to generate a fault location result with three-dimensional coordinates. The device knowledge graph is called based on the subtle change event to infer the cause of the fault and generate a maintenance work order with three-dimensional coordinates and fault cause.

[0006] According to the technical solution provided in this application, the step of performing pixel-level difference comparison on the aligned image to detect subtle change events in the device instance includes: Pixel-level difference comparison is performed on the aligned image to obtain the two-dimensional change region corresponding to the device instance; The depth data collected by the mobile inspection robot during the current inspection is obtained. Based on the camera parameters when the current inspection image is collected and the observation pose of the mobile inspection robot, the two-dimensional change area is back-projected onto the coordinate system of the three-dimensional semantic map to obtain the three-dimensional spatial position corresponding to the two-dimensional change area. Based on the three-dimensional coordinates and preset dimensions of the device instance in the three-dimensional semantic map, the spatial tolerance range corresponding to the device instance is determined; Determine whether the three-dimensional spatial position falls within the spatial tolerance range corresponding to the device instance. If it does, the two-dimensional change area is determined to be a valid change area, and the subtle change event of the device instance is determined based on the valid change area. If it does not fall within the range, the two-dimensional change area is determined to be an invalid change area and is removed.

[0007] According to the technical solution provided in this application, the multimodal inspection data includes visible light images, and at least one of infrared images, ultrasonic data, and gas concentration data; the method further includes: Based on the semantic tags of the device instance, the multimodal signal association weights of the device instance are determined, and the multimodal signal association weights represent the degree of physical association between the visible light change event of the device instance and the other modal signals; After detecting the subtle change event, the corresponding multimodal signal association weight is queried based on the event type of the subtle change event and the semantic tag of the device instance; Obtain the feature data of the other modal signals in the current inspection, and calculate the association confidence of the subtle change event based on the association weight of the multimodal signals and the feature data of the other modal signals; The alarm level of the subtle change event is determined based on the correlation confidence level, and a corresponding priority maintenance work order is generated.

[0008] According to the technical solution provided in this application, the subtle change event can be one or more; after binding the subtle change event to the three-dimensional coordinates of the device instance in the three-dimensional semantic map, the method further includes: When multiple minor change events are detected on the device instance, the three-dimensional spatial distance between the minor change events is calculated based on the three-dimensional coordinates bound to each minor change event. Determine whether there exists at least two subtle change events whose three-dimensional spatial distance is less than a preset aggregation threshold, wherein the preset aggregation threshold is set according to the device size corresponding to the semantic tag of the device instance; If they exist, the at least two minor change events are identified as a spatial clustered change event group, and a high-priority maintenance work order is generated based on the spatial clustered change event group. If none exist, the multiple minor change events are identified as spatially dispersed change events, and a regular priority maintenance work order is generated based on the spatially dispersed change events.

[0009] According to the technical solution provided in this application, the historical inspection image includes a local reference image corresponding to the device instance; after binding the subtle change event with the three-dimensional coordinates of the device instance in the three-dimensional semantic map, it further includes: Record the number of times the aforementioned subtle change event occurs across multiple consecutive inspection cycles; When the number of occurrences of the same type of subtle change event bound to the three-dimensional coordinates of the same device instance reaches a preset persistence threshold in a continuous inspection cycle, the local reference image of the device instance is determined to be outdated. Update the local reference image corresponding to the device instance based on the current inspection image, and use the updated local reference image as the historical inspection image for the next inspection cycle. Specifically, before determining that the local reference image of the equipment instance is outdated, the repeated occurrence of the same type of minor change event in the same equipment instance during continuous inspection cycles is marked as a continuous change event and merged into the same maintenance work order.

[0010] According to the technical solution provided in this application, the three-dimensional semantic map also includes a first mapping relationship between the semantic tags of the device instance and the set of change types, wherein the set of change types includes at least one change type that may occur on the device structure corresponding to the semantic tag; The step of performing pixel-level difference comparison on the aligned image includes: Based on the semantic tags of the device instance, query the first mapping relationship to determine the set of change types corresponding to the device instance; Based on the physical characteristics of each change type in the set of change types, the difference comparison parameters corresponding to each change type are determined respectively. The difference comparison parameters include at least one of color channel weights, edge detection operators, and texture difference measures. Based on the difference comparison parameters corresponding to each change type, pixel-level difference comparison is performed on the aligned image to obtain the difference response map corresponding to each change type. The subtle change events and their types for the device instance are determined based on each difference response map.

[0011] According to the technical solution provided in this application, after determining the subtle change events and their change types of the device instance based on each difference response map, the method further includes: Obtain the response quality index of each difference response map in multiple consecutive inspection cycles, wherein the response quality index includes at least one of signal-to-noise ratio and repeatability of change events; When the response quality index of the difference response map corresponding to a certain change type is lower than the preset quality threshold, the difference comparison parameter corresponding to the change type is adjusted by a preset adjustment step size until the response quality index of the difference response map corresponding to the change type is not lower than the preset quality threshold, or the number of adjustments reaches the preset maximum number of adjustments. Update the adjusted difference comparison parameters to the first mapping relationship.

[0012] According to the technical solution provided in this application, the three-dimensional semantic map also includes a second mapping relationship between the semantic tags of the device instance and the light-sensitive area; The step of performing pixel-level difference comparison on the aligned image includes: When the difference in lighting conditions between the historical inspection image and the current inspection image exceeds the preset lighting tolerance range, the second mapping relationship is queried according to the semantic tag of the device instance, and the image area of ​​the device instance is divided into a light-sensitive area and a light-insensitive area. Pixel-level difference comparison is performed directly on the light-insensitive area, and pixel-level difference comparison is performed on the light-sensitive area after light correction; When the difference in lighting conditions does not exceed the preset lighting tolerance range, pixel-level difference comparison is directly performed on the aligned image.

[0013] According to the technical solution provided in this application, the light-sensitive area is determined in the following way: Obtain statistical data on the pixel value changes of the image region corresponding to the semantic tag of the device instance under different lighting conditions during historical inspections; Based on the pixel value change statistics, the illumination sensitivity of the image region corresponding to the semantic tag is determined; When the light sensitivity is higher than a preset sensitivity threshold, the image region corresponding to the semantic tag is determined as a light-sensitive region; otherwise, it is determined as a light-insensitive region. The correspondence between the semantic tags and the determined light-sensitive or light-insensitive regions is stored in the second mapping relationship.

[0014] According to the technical solution provided in this application, the illumination correction of the light-sensitive area includes: Obtain the illumination response characteristics of the illumination-sensitive area during historical inspections; The change in illumination condition parameters is determined based on the illumination condition parameters corresponding to the historical inspection images and the illumination condition parameters corresponding to the current inspection images. The pixel values ​​of the light-sensitive areas in the current inspection image are corrected based on the light response characteristics and the changes in the light condition parameters.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: I. By constructing a 3D semantic map containing equipment instances, 3D coordinates, and semantic labels, and projecting multimodal inspection data onto the 3D semantic map and binding it with the corresponding equipment instances, a precise spatial association between multimodal inspection data and equipment instances is achieved, providing a unified semantic spatial benchmark for detecting subtle changes and locating faults.

[0016] Second, by aligning historical inspection images and current inspection images to the same observation area of ​​the same equipment instance for pixel-level difference comparison, it is possible to detect subtle changes that are difficult for humans to perceive, such as changes in indicator light color, latch displacement, cable displacement, and equipment surface deformation, thus giving the inspection robot an environmental subtle change perception capability that surpasses the human eye.

[0017] Third, by binding subtle change events with the three-dimensional coordinates of device instances in a three-dimensional semantic map, fault location results with three-dimensional coordinates are generated, realizing three-dimensional fault location at the device instance level with precise spatial coordinates, so that alarms directly correspond to the specific spatial location of specific devices.

[0018] Fourth, by inferring the cause of a fault based on subtle changes in the device knowledge graph, a maintenance work order with three-dimensional coordinates and the cause of the fault is generated. This allows maintenance personnel to know the location and possible cause of the fault without on-site inspection, eliminating the potential for unplanned downtime in the data center at the micro-symptom stage. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating the steps of the intelligent inspection and fault location method for multimodal information fusion in data center computer rooms provided in this application; Figure 2 A scene diagram of a data center server room and a mobile inspection robot provided in this application; Figure 3 The multimodal associated alarm block diagram provided for this application. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] As mentioned in the background section, this application proposes a multimodal information fusion intelligent inspection and fault location method for data center computer rooms, applied to mobile inspection robots, such as... Figure 1 As shown, it includes: S1. Construct a three-dimensional semantic map of the data center room. The three-dimensional semantic map is constructed by fusing LiDAR SLAM data, visual inertial odometry data, and depth camera data, and includes device instances, the three-dimensional coordinates of the device instances, and semantic labels. S2. Collect multimodal inspection data during the inspection process. The multimodal inspection data includes at least two of the following: visible light images, infrared images, ultrasonic data, and gas concentration data. Based on the pose of the mobile inspection robot, project the multimodal inspection data onto the three-dimensional semantic map and bind it to the corresponding device instance in the three-dimensional semantic map. S3. For the same device instance in the three-dimensional semantic map, based on the three-dimensional coordinates of the device instance in the three-dimensional semantic map and the observation pose during historical inspections and current inspections, align the historical inspection images and the current inspection images to the same observation area of ​​the device instance, and perform pixel-level difference comparison on the aligned images to detect subtle change events of the device instance. The subtle change events include at least one of the following: indicator light color change, latch displacement, cable displacement, and device surface deformation. S4. Bind the subtle change event to the three-dimensional coordinates of the device instance in the three-dimensional semantic map to generate a fault location result with three-dimensional coordinates, and infer the cause of the fault based on the device knowledge graph according to the subtle change event, and generate a maintenance work order with three-dimensional coordinates and fault cause.

[0023] Specifically, such as Figure 2 As shown, a mobile inspection robot is deployed within the data center server room. This robot is equipped with LiDAR, a visual inertial odometry system, a depth camera, a visible light camera, an infrared thermal imager, an ultrasonic sensor, and a gas concentration sensor. An equipment instance refers to a physical device unit within the data center server room that has independent functions and requires monitoring, including servers, switches, UPS power supplies, power distribution cabinets, and air conditioning units. Semantic tags refer to identification information describing the functional type and structural characteristics of an equipment instance, including "server panel," "power module," "indicator light area," "cable interface," and "fan unit." Minor change events refer to changes occurring on the equipment instance that appear as local pixel differences in the image but are small in physical scale, including at least one of the following: indicator light color change, latch displacement, cable displacement, and equipment surface deformation.

[0024] When constructing a 3D semantic map, the robot collects 3D point cloud data of the computer room environment using LiDAR, obtains robot pose estimation and sparse feature points using visual inertial odometry, and acquires depth information of the equipment surface using a depth camera. These three types of data are fused as follows: First, using the pose estimation provided by the visual inertial odometry as the initial value, the LiDAR point cloud and the depth map from the depth camera are registered using an iterative nearest-point algorithm to obtain a registered 3D point cloud. Then, semantic segmentation is performed on the registered 3D point cloud. In this embodiment, the PointNet++ network is used for point-by-point semantic classification of the 3D point cloud. The input is an N×3 point cloud coordinate matrix, where N is the number of points in the point cloud, and the output is an N×C semantic category probability matrix, where C is the number of semantic categories. The category with the highest probability for each point is taken as the semantic category of that point. Points belonging to different equipment instances in the point cloud are identified, and each equipment instance is assigned a semantic label. Finally, the semantic segmentation result is merged with the 3D point cloud registration result to generate a 3D semantic map. This map is marked with the 3D coordinates and semantic labels of each equipment instance. The 3D coordinates of a device instance are the coordinates of its geometric center. Semantic tags can also be obtained through manual annotation.

[0025] During the inspection, the robot moves along a preset path, collecting multimodal inspection data along the way. Visible light images are used to capture the appearance features of the equipment, infrared images are used to capture the temperature distribution on the equipment surface, ultrasonic data is used to capture the sound characteristics of the equipment during operation, and gas concentration data is used to capture the gas content in the machine room environment. The robot's pose is provided by a visual inertial odometry system. When projecting the multimodal inspection data onto a 3D semantic map, the visible light and infrared images are projected onto the 3D semantic map through a camera projection model. Specifically, for each pixel in the visible light image, based on the intrinsic and extrinsic parameter matrices of the visible light camera and the robot's pose, a spatial ray in the coordinate system of the 3D semantic map is determined for that pixel. This spatial ray intersects with the surface of the equipment instance in the 3D semantic map, and the equipment instance at the intersection point is the equipment instance corresponding to that pixel. Ultrasonic data and gas concentration data are associated with the equipment instances through a sensor spatial coverage model. Specifically, the ultrasonic sensor and the gas concentration sensor have preset spatial coverage areas. Based on the robot's current pose and the sensor's installation location, the corresponding area of ​​the sensor's spatial coverage area in the 3D semantic map is determined. The device instances within this area are the device instances corresponding to the ultrasonic data and gas concentration data. The bound data is stored using the device instance identifier as an index. Each device instance identifier is associated with the visible light image area, infrared image area, ultrasonic data segment, and gas concentration data segment corresponding to that device instance.

[0026] For the same device instance in the 3D semantic map, the robot acquires historical inspection images of that device instance in the historical inspection and current inspection images in the current inspection. Based on the 3D coordinates of the device instance in the 3D semantic map and the observation poses during the historical and current inspections, the historical and current inspection images are aligned to the same observation area of ​​the device instance. The alignment method is as follows: based on the observation poses during the historical and current inspections, the pixel coordinates corresponding to the 3D coordinates of the device instance in the historical and current inspection images are calculated respectively. Using these pixel coordinates as the center, a local image region corresponding to the device instance is extracted from both the historical and current inspection images. If the area of ​​the local image region of the device instance in the current inspection image is less than a preset minimum area threshold, the pixel-level difference comparison is skipped, and the image of the device instance is re-acquired in the next inspection cycle. Then, a pixel-level difference comparison is performed on the two local image regions. The specific method for pixel-level difference comparison is as follows: Calculate the sum of the absolute differences of the RGB three-channel pixel values ​​of corresponding pixels in two local image regions. If the sum of the absolute differences exceeds a preset difference threshold, the pixel is marked as a difference pixel. The preset difference threshold is set according to the semantic tag of the device instance. For example, for the "indicator light area," the preset difference threshold is set to 30; for the "device casing," the preset difference threshold is set to 50. Connected regions formed by difference pixels are considered as pixel regions with significant differences.

[0027] Subtle change events in device instances are identified based on significantly different pixel regions. Specifically, the classification method involves extracting color histogram features and edge features from the differing pixel regions. If the area of ​​the differing pixel region is less than a preset area threshold and the mean change of a certain color channel in the color histogram features exceeds a preset color change threshold, it is determined to be an indicator light color change. If the shape of the differing pixel region is elongated and the edge position in the edge features is offset, it is determined to be cable displacement. If the shape of the differing pixel region is rectangular and the edge position in the edge features is offset, it is determined to be latch displacement. If the area of ​​the differing pixel region is greater than a preset area threshold and the texture features change, it is determined to be device surface deformation.

[0028] By binding subtle change events with the 3D coordinates of device instances in a 3D semantic map, fault location results with 3D coordinates are generated. The cause of the fault is inferred from the device knowledge graph based on the subtle change events. The device knowledge graph is constructed as follows: a directed graph is built using device type, change event type, and fault cause as nodes, and "cause" and "manifestation" as edges. For example, the "power module abnormality" node is connected to the "indicator light color change" node via the "manifestation" edge, and the "indicator light color change" node is connected to the "power module abnormality" node via the "cause" edge. The relationships in the graph are manually compiled from equipment manuals and historical maintenance records. The initial set of nodes and edges is manually entered from equipment manuals and expert experience, and subsequently iteratively updated based on maintenance feedback. For example, if the equipment manual states that "the typical manifestation of power module overheating is abnormal indicator light color," then an "manifestation" edge is established from the "power module overheating" node to the "indicator light color change" node. During reasoning, the event type of the subtle change event is used as the query condition. The fault cause nodes connected to the event type through "represents as" or "cause" edges are retrieved from the knowledge graph, and the retrieved fault causes are used as the inference results. A maintenance work order is generated based on the inferred fault cause and the 3D coordinates.

[0029] This embodiment binds multimodal inspection data to equipment instances using a 3D semantic map, enabling subsequent change detection and fault localization to be performed at the equipment instance level, avoiding the problem of alarm information being disconnected from the actual equipment in traditional methods. By comparing pixel-level differences between historical and current inspection images, subtle changes imperceptible to the human eye can be detected. By binding subtle change events to 3D coordinates and inferring fault causes using a knowledge graph, automated maintenance work orders with precise spatial coordinates and fault causes are generated.

[0030] In a preferred embodiment, the step of performing pixel-level difference comparison on the aligned image to detect subtle change events in the device instance includes: Pixel-level difference comparison is performed on the aligned image to obtain the two-dimensional change region corresponding to the device instance; The depth data collected by the mobile inspection robot during the current inspection is obtained. Based on the camera parameters when the current inspection image is collected and the observation pose of the mobile inspection robot, the two-dimensional change area is back-projected onto the coordinate system of the three-dimensional semantic map to obtain the three-dimensional spatial position corresponding to the two-dimensional change area. Based on the three-dimensional coordinates and preset dimensions of the device instance in the three-dimensional semantic map, the spatial tolerance range corresponding to the device instance is determined; Determine whether the three-dimensional spatial position falls within the spatial tolerance range corresponding to the device instance. If it does, the two-dimensional change area is determined to be a valid change area, and the subtle change event of the device instance is determined based on the valid change area. If it does not fall within the range, the two-dimensional change area is determined to be an invalid change area and is removed.

[0031] Specifically, the two-dimensional change region refers to the pixel region where the pixel value difference exceeds a preset difference threshold, obtained through pixel-level difference comparison after aligning historical and current inspection images. This two-dimensional change region contains one or more connected components, each corresponding to a potential change event. After obtaining the two-dimensional change region, the robot acquires the depth data collected by the depth camera during the current inspection. The depth data provides the depth value corresponding to each pixel in the current scene, i.e., the distance from the camera to the surface of an object in the scene. Based on the camera parameters and the robot's observation pose when acquiring the current inspection image, each pixel in the two-dimensional change region is back-projected into the coordinate system of the three-dimensional semantic map. Camera parameters include focal length, principal point coordinates, and distortion coefficients. The observation pose includes the position and orientation of the robot's base in the coordinate system of the three-dimensional semantic map, as well as the camera's installation position and angle relative to the robot's base. The specific calculation method for back-projection is as follows: for each pixel in the two-dimensional change region, the direction vector of the pixel in the camera coordinate system is determined based on the camera parameters, and the depth value of the pixel in the camera coordinate system is determined based on the depth data, thereby obtaining the three-dimensional coordinates of the pixel in the camera coordinate system. Then, based on the camera's installation position and angle relative to the robot base, the three-dimensional coordinates are transformed from the camera coordinate system to the robot base coordinate system. Then, based on the robot's observed pose, the three-dimensional coordinates in the robot base coordinate system are transformed to the coordinate system of the three-dimensional semantic map to obtain the three-dimensional spatial position corresponding to the two-dimensional change area.

[0032] The spatial tolerance range refers to a cuboid region centered on the three-dimensional coordinates of the device instance and with preset dimensions as its sides. The preset dimensions are set based on the typical device dimensions corresponding to the semantic tag of the device instance. For example, for the semantic tag "server," the preset dimensions are set to the typical dimensions of a server chassis, i.e., 0.8 meters long, 0.05 meters wide, and 0.04 meters high; for the semantic tag "indicator area," the preset dimensions are set to the typical dimensions of an indicator panel, i.e., 0.1 meters long, 0.02 meters wide, and 0.02 meters high; and for the semantic tag "cable interface," the preset dimensions are set to the typical dimensions of a cable interface, i.e., 0.05 meters long, 0.05 meters wide, and 0.05 meters high. The system determines whether the three-dimensional spatial position obtained from the back projection falls within this spatial tolerance range. If it does, it indicates that the two-dimensional change area is indeed within the physical space of the device instance and is determined as a valid change area. Minor change events of the device instance are then identified based on the valid change area. If it does not fall within this range, it indicates that the two-dimensional change area may be a spurious change caused by residual registration errors or background interference and is determined as an invalid change area and discarded.

[0033] This embodiment verifies the spatial attribution of a changed region by backprojecting a two-dimensional changed region onto a three-dimensional semantic map and comparing it with the spatial tolerance range of the device instance. Since the backprojection uses depth data collected during the current inspection, the verification is based on the actual spatial location of the device instance in the current scene. Even if the device instance itself has shifted, it will not cause the valid changed region to be incorrectly removed. By removing invalid changed regions, the false alarm rate caused by registration residual errors or background interference is reduced, improving the accuracy of subsequent fault location. The technical principle is that a changed region in a two-dimensional image can only indicate that a certain pixel area has changed, but it cannot indicate whether the change belongs to the target device instance. By backprojecting to three-dimensional space and comparing it with the spatial range of the device instance, the spatial attribution of the two-dimensional changed region can be elevated from the image level to the three-dimensional physical level, thereby confirming whether the change actually occurred on the target device instance.

[0034] In a preferred embodiment, the multimodal inspection data includes visible light images, and at least one of infrared images, ultrasonic data, and gas concentration data; the method further includes: Based on the semantic tags of the device instance, the multimodal signal association weights of the device instance are determined, and the multimodal signal association weights represent the degree of physical association between the visible light change event of the device instance and the other modal signals; After detecting the subtle change event, the corresponding multimodal signal association weight is queried based on the event type of the subtle change event and the semantic tag of the device instance; Obtain the feature data of the other modal signals in the current inspection, and calculate the association confidence of the subtle change event based on the association weight of the multimodal signals and the feature data of the other modal signals; The alarm level of the subtle change event is determined based on the correlation confidence level, and a corresponding priority maintenance work order is generated.

[0035] Specifically, such as Figure 3 As shown, the multimodal inspection data includes visible light images, as well as at least one of infrared images, ultrasonic data, and gas concentration data. Visible light images are used to capture subtle changes in the appearance of the equipment, infrared images are used to capture the temperature distribution on the surface of the equipment, ultrasonic data is used to capture the sound characteristics of the equipment during operation, and gas concentration data is used to capture the gas content in the computer room environment.

[0036] Multimodal signal association weight refers to the degree of physical correlation between visible light change events and other modal signals of a device instance, and its value is greater than or equal to zero and less than or equal to one. This weight is determined based on the semantic tag of the device instance. The association weight is obtained by statistically analyzing the frequency of simultaneous occurrence of visible light change events and anomalies in all other modal signals on device instances with the same semantic tag in historical maintenance records. This frequency is then normalized to the range of zero to one and used as the association weight. For example, for the semantic tag "power module," in historical maintenance records, the frequency of simultaneous occurrence of indicator light color changes and temperature increases is 80%, so the association weight between visible light change events and infrared signals is set to 0.8; the frequency of simultaneous occurrence of indicator light color changes and sound anomalies is 30%, so the association weight between visible light change events and ultrasonic signals is set to 0.3. For the semantic tag "fan unit," in historical maintenance records, the frequency of simultaneous occurrence of surface deformation and sound anomalies is 90%, so the association weight between visible light change events and ultrasonic signals is set to 0.9.

[0037] After detecting a subtle change event, the corresponding multimodal signal association weight is queried based on the event type and semantic tag of the device instance. Feature data of the remaining modal signals in the current inspection is then obtained. Feature data is a normalized value obtained after assessing the anomaly level of the remaining modal signals, ranging from zero to one. The normalization method for infrared signal feature data is as follows: calculate the temperature deviation between the highest surface temperature of the device instance in the current inspection and the average normal surface temperature of the device instance in historical inspections, divide the temperature deviation by a preset upper limit of temperature deviation, and obtain the feature data of the infrared signal. The preset upper limit of temperature deviation is set according to the semantic tag of the device instance; for example, for "power module," the preset upper limit of temperature deviation is set to 15 degrees Celsius. If the average normal temperature of the device instance in historical inspections is missing, the average normal temperature of other device instances with the same semantic tag is used as a substitute. If the average normal temperature of other device instances with the same semantic tag is also missing, the average of the average normal temperatures of all device instances in the same cabinet is used as a substitute. The normalization method for the characteristic data of ultrasonic signals is as follows: calculate the spectral deviation between the ultrasonic spectrum of the equipment instance in the current inspection and the normal ultrasonic spectrum of the same equipment instance in historical inspections, and divide the spectral deviation by a preset upper limit of spectral deviation to obtain the characteristic data of the ultrasonic signal. The normalization method for the characteristic data of gas concentration signals is as follows: calculate the concentration deviation between the gas concentration in the area where the equipment instance is located in the current inspection and the average normal gas concentration in that area in historical inspections, and divide the concentration deviation by a preset upper limit of concentration deviation to obtain the characteristic data of the gas concentration signal.

[0038] The association confidence of a subtle change event is obtained by multiplying the association weight of the multimodal signal with the feature data of the other modal signals. When there are multiple other modal signals, the product corresponding to each modal signal is calculated separately, and the maximum value is taken as the association confidence. The higher the association confidence, the stronger the physical correlation between the subtle change event and the other modal signals, and the more likely the change event is to be a real fault.

[0039] The alarm level of a minor change event is determined based on the correlation confidence level, and a corresponding priority maintenance work order is generated. If the correlation confidence level is higher than a first threshold (set to 0.7), the minor change event is confirmed as a genuine fault event, and a high-priority maintenance work order is generated. If the correlation confidence level is lower than or equal to the first threshold but higher than a second threshold (set to 0.3), the minor change event is marked as a suspected change event, and a medium-priority maintenance work order is generated. If the correlation confidence level is lower than or equal to the second threshold, the minor change event is retained as an independent alarm event, a low-priority maintenance work order is generated, and the minor change event is marked as a single-modal change event in the work order.

[0040] In a preferred embodiment, the subtle change event is one or more; after binding the subtle change event to the three-dimensional coordinates of the device instance in the three-dimensional semantic map, the method further includes: When multiple minor change events are detected on the device instance, the three-dimensional spatial distance between the minor change events is calculated based on the three-dimensional coordinates bound to each minor change event. Determine whether there exists at least two subtle change events whose three-dimensional spatial distance is less than a preset aggregation threshold, wherein the preset aggregation threshold is set according to the device size corresponding to the semantic tag of the device instance; If they exist, the at least two minor change events are identified as a spatial clustered change event group, and a high-priority maintenance work order is generated based on the spatial clustered change event group. If none exist, the multiple minor change events are identified as spatially dispersed change events, and a regular priority maintenance work order is generated based on the spatially dispersed change events.

[0041] Specifically, when multiple minor change events are detected on the same device instance, it is necessary to determine whether these change events cluster in three-dimensional space to determine whether they may be caused by the same fault source. In this embodiment, the three-dimensional spatial distance refers to the Euclidean distance between the three-dimensional coordinates bound to two minor change events. The preset clustering threshold is a length value used to determine whether two change events are sufficiently close in space. This threshold is determined based on the short side length of the device bounding box corresponding to the semantic tag of the device instance, and is twice that short side length. For example, if the short side of the bounding box of a server chassis is 0.05 meters, the preset clustering threshold is 0.1 meters; if the short side of the bounding box of an indicator light panel is 0.02 meters, the preset clustering threshold is 0.04 meters; and if the short side of the bounding box of a cable interface is 0.03 meters, the preset clustering threshold is 0.06 meters. A spatially clustered change event group refers to two or more minor change events, where the three-dimensional spatial distance between each pair is less than the preset clustering threshold. Spatially dispersed change events refer to multiple minor change events in which no two change events have a three-dimensional spatial distance less than a preset clustering threshold, or only a few change events are close to each other but do not constitute a cluster as a whole.

[0042] In practice, after the robot detects multiple minor change events for a specific device instance during the current inspection cycle, it first acquires the 3D coordinates associated with each minor change event. Assume three change events are detected: event A, event B, and event C. The 3D coordinates of event A are (x1, y1, z1), event B is (x2, y2, z2), and event C is (x3, y3, z3). The 3D spatial distance dAB between event A and event B, the 3D spatial distance dAC between event A and event C, and the 3D spatial distance dBC between event B and event C are calculated. Then, each distance is compared with a preset clustering threshold. If the 3D spatial distance between at least one pair of change events is less than the preset clustering threshold (e.g., dAB is less than the preset clustering threshold), then event A and event B are identified as a spatially clustered change event group. If the distance between event C and either event A or event B is also less than the preset clustering threshold, then event C is also added to this clustered change event group. If the distance between all pairs of events is greater than or equal to a preset clustering threshold, these events are classified as spatially dispersed change events. Maintenance work orders of different priorities are generated based on the judgment results. If a group of spatially clustered change events exists, it indicates that multiple change events are highly concentrated in space and are likely caused by the same fault source. For example, overheating of the same power module may cause indicator light discoloration and surface deformation to occur simultaneously. In this case, a high-priority maintenance work order is generated to remind maintenance personnel to handle it first. If the change events are spatially dispersed, it indicates that these events may be caused by different reasons and have a relatively lower urgency. A regular-priority maintenance work order is generated.

[0043] This embodiment calculates the three-dimensional spatial distance between multiple subtle change events and determines whether spatial clustering exists. It can distinguish between multiple manifestations caused by the same fault source and independent changes caused by different fault sources, thereby making the priority of maintenance work orders more reasonable.

[0044] In a preferred embodiment, the historical inspection image includes a local reference image corresponding to the device instance; after binding the subtle change event with the three-dimensional coordinates of the device instance in the three-dimensional semantic map, the method further includes: Record the number of times the aforementioned subtle change event occurs across multiple consecutive inspection cycles; When the number of occurrences of the same type of subtle change event bound to the three-dimensional coordinates of the same device instance reaches a preset persistence threshold in a continuous inspection cycle, the local reference image of the device instance is determined to be outdated. Update the local reference image corresponding to the device instance based on the current inspection image, and use the updated local reference image as the historical inspection image for the next inspection cycle. Specifically, before determining that the local reference image of the equipment instance is outdated, the repeated occurrence of the same type of minor change event in the same equipment instance during continuous inspection cycles is marked as a continuous change event and merged into the same maintenance work order.

[0045] Specifically, when multiple minor change events are detected on the same device instance, it is necessary to determine whether these change events cluster in three-dimensional space to determine whether they may be caused by the same fault source. In this embodiment, the three-dimensional spatial distance refers to the Euclidean distance between the three-dimensional coordinates bound to two minor change events. The preset clustering threshold is a length value used to determine whether two change events are sufficiently close in space. This threshold is determined based on the short side length of the device bounding box corresponding to the semantic tag of the device instance, and is set to twice this short side length. The reason for setting it to twice the short side length of the bounding box is that the influence range of the same fault source usually does not exceed the short side direction of the device bounding box. Setting it to twice can cover the influence range of the fault source while avoiding misjudging the change events of adjacent devices as spatial clustering. For example, if the short side of the bounding box of a server chassis is 0.05 meters, the preset clustering threshold is 0.1 meters; if the short side of the bounding box of an indicator light panel is 0.02 meters, the preset clustering threshold is 0.04 meters; and if the short side of the bounding box of a cable interface is 0.03 meters, the preset clustering threshold is 0.06 meters. Spatially clustered change event groups refer to two or more minor change events where the three-dimensional spatial distance between any two of them is less than a preset clustering threshold. Spatially dispersed change events refer to multiple minor change events where no two change events have a three-dimensional spatial distance less than the preset clustering threshold, or where only a few change events are close together but do not constitute a cluster as a whole.

[0046] In practice, after the robot detects multiple minor change events for a specific device instance during the current inspection cycle, it first acquires the 3D coordinates associated with each minor change event. Assume three change events are detected: event A, event B, and event C. The 3D coordinates of event A are (x1, y1, z1), event B is (x2, y2, z2), and event C is (x3, y3, z3). The 3D spatial distance dAB between event A and event B, the 3D spatial distance dAC between event A and event C, and the 3D spatial distance dBC between event B and event C are calculated. Then, each distance is compared with a preset clustering threshold. If the 3D spatial distance between at least one pair of change events is less than the preset clustering threshold (e.g., dAB is less than the preset clustering threshold), then event A and event B are identified as a spatially clustered change event group. If the distance between event C and either event A or event B is also less than the preset clustering threshold, then event C is also added to this clustered change event group. If the distance between all pairs of events is greater than or equal to a preset clustering threshold, these events are classified as spatially dispersed change events. Maintenance work orders of different priorities are generated based on the judgment results. If a group of spatially clustered change events exists, it indicates that multiple change events are highly concentrated in space and are likely caused by the same fault source. For example, overheating of the same power module may cause indicator light discoloration and surface deformation to occur simultaneously. In this case, a high-priority maintenance work order is generated to remind maintenance personnel to handle it first. If the change events are spatially dispersed, it indicates that these events may be caused by different reasons and have a relatively lower urgency. A regular-priority maintenance work order is generated.

[0047] This embodiment calculates the three-dimensional spatial distance between multiple subtle change events and determines whether spatial clustering exists. It can distinguish between multiple manifestations caused by the same fault source and independent changes caused by different fault sources, thereby making the priority of maintenance work orders more reasonable.

[0048] In a preferred embodiment, the three-dimensional semantic map further includes a first mapping relationship between the semantic tags of the device instance and a set of change types, wherein the set of change types includes at least one change type that may occur on the device structure corresponding to the semantic tag; The step of performing pixel-level difference comparison on the aligned image includes: Based on the semantic tags of the device instance, query the first mapping relationship to determine the set of change types corresponding to the device instance; Based on the physical characteristics of each change type in the set of change types, the difference comparison parameters corresponding to each change type are determined respectively. The difference comparison parameters include at least one of color channel weights, edge detection operators, and texture difference measures. Based on the difference comparison parameters corresponding to each change type, pixel-level difference comparison is performed on the aligned image to obtain the difference response map corresponding to each change type. The subtle change events and their types for the device instance are determined based on each difference response map.

[0049] Specifically, the change type set refers to the set of at least one change type that may occur on the device structure corresponding to the semantic label. The first mapping relationship refers to the correspondence between the semantic label and the change type set. For example, when the semantic label is an indicator light area, the change type set includes indicator light color changes; when the semantic label is a cable interface, the change type set includes cable displacement; when the semantic label is a device housing, the change type set includes device surface deformation; when the semantic label is a latching area, the change type set includes latch displacement.

[0050] The first mapping relationship can be established either manually or through historical data statistics. Manual annotation involves maintenance personnel listing the possible types of changes to the equipment structure corresponding to each semantic tag based on equipment manuals and practical experience. For example, maintenance personnel might know from the equipment manual that indicator light areas may experience color changes, cable interfaces may experience cable displacement, latch areas may experience latch displacement, and the equipment casing may experience surface deformation; based on this, the first mapping relationship is established. Historical data statistics involve collecting historical maintenance records, statistically analyzing the actual types of changes that have occurred on the equipment structure corresponding to each semantic tag, and including change types with a frequency exceeding a preset frequency threshold into the change type set for that semantic tag. For example, in historical maintenance records, the frequency of color changes in the indicator light area is 95%, and the frequency of surface deformation is 3%. If the preset frequency threshold is 10%, then only color changes are included in the change type set for the indicator light area.

[0051] Difference comparison parameters refer to the parameters used when performing pixel-level difference comparisons, including at least one of color channel weights, edge detection operators, and texture difference measures. Color channel weights refer to the weight coefficients of each channel when calculating the difference between RGB pixel values. Edge detection operators are operators used to extract image edge features, such as the Sobel operator and the Canny operator. Texture difference measures are metrics used to measure local texture differences in an image, such as local binary pattern histogram differences.

[0052] In practical implementation, for a device instance in the 3D semantic map, the first step is to query the first mapping relationship based on its semantic label to determine the set of change types corresponding to the device instance. For example, if the semantic label of the device instance is "indicator area," querying the first mapping relationship yields a set of change types including indicator light color changes. Then, based on the physical characteristics of each change type in the set of change types, the difference comparison parameters corresponding to each change type are determined. For indicator light color changes, the physical characteristic is a significant change in the color channels; therefore, the color channel weights in the difference comparison parameters are set to 0.4 for the red channel, 0.4 for the green channel, and 0.2 for the blue channel. The red and green channels have higher weights because indicator light color changes are mainly reflected in the change in the ratio of the red and green channels, while the blue channel has a lower weight because its contribution to indicator light color changes is smaller. The edge detection operator uses the Canny operator with a high threshold to suppress noise, and the texture difference measure uses the local binary mode histogram difference with a low weight. For cable displacement, the physical characteristic is the displacement of the edge contour; therefore, the color channel weights in the difference comparison parameters can be set equally, the edge detection operator uses the Sobel operator with a low threshold to sensitively capture edges, and the texture difference measure has a high weight. Based on the difference comparison parameters corresponding to each change type, pixel-level difference comparisons are performed on the aligned images. For indicator light color changes, the weighted sum of absolute differences of corresponding pixels is calculated according to the color channel weights. If it exceeds a preset threshold, it is marked as a difference pixel, resulting in a difference response map of the indicator light color change. For cable displacement, an edge detection operator is used to extract the edge binary map. Using the edge binary map of the current inspection image as a reference, a translation search is performed within a preset translation range. After each translation, the number of overlapping pixels between the translated edge binary map and the edge binary map of the historical inspection image is calculated. The translation amount with the largest number of overlapping pixels is taken as the edge position offset. If this offset exceeds a preset offset threshold, the edge region is marked as a difference region, resulting in a difference response map of the cable displacement. Finally, the subtle change events and their change types of the device instance are determined based on each difference response map. For example, if there is a significant difference region in the difference response map of the indicator light color change, it is determined that the device instance has experienced an indicator light color change; if there is a significant difference region in the difference response map of the cable displacement, it is determined that the device instance has experienced cable displacement.

[0053] This embodiment can distinguish between persistent and transient changes by recording the frequency of occurrence of subtle change events in a continuous inspection cycle. For persistent changes, the local reference image is automatically updated to prevent repeated alarms from being generated in subsequent inspections. At the same time, recurring change events are merged into the same work order to reduce the number of alarms.

[0054] In a preferred embodiment, after determining the subtle change events and their change types of the device instance based on each difference response map, the method further includes: Obtain the response quality index of each difference response map in multiple consecutive inspection cycles, wherein the response quality index includes at least one of signal-to-noise ratio and repeatability of change events; When the response quality index of the difference response map corresponding to a certain change type is lower than the preset quality threshold, the difference comparison parameter corresponding to the change type is adjusted by a preset adjustment step size until the response quality index of the difference response map corresponding to the change type is not lower than the preset quality threshold, or the number of adjustments reaches the preset maximum number of adjustments. Update the adjusted difference comparison parameters to the first mapping relationship.

[0055] Specifically, the previous difference comparison parameters were determined based on the semantic tags and change type set of the device instance. However, in actual operation, as the device status and environmental conditions change, the quality of the difference response map corresponding to the same change type may deteriorate. For example, the noise area in the difference response map may increase, the true change area may be submerged, or the detection results of the same change type may be inconsistent in different inspection cycles. Therefore, this implementation method uses closed-loop feedback adjustment to adjust the difference comparison parameters.

[0056] In this embodiment, the response quality indicators include at least one of signal-to-noise ratio (SNR) and repeatability of change events. SNR refers to the ratio between the average response value of the effective change region and the average response value of the noisy region in the difference response map. The effective change region refers to the connected region in the difference response map whose response value exceeds a preset response threshold, and the noisy region refers to the region in the difference response map whose response value does not exceed the preset response threshold. A higher SNR indicates a stronger effective change signal relative to noise, and better quality of the difference response map. Repeatability of change events refers to the proportion of the same change type detected in multiple consecutive inspection cycles. For example, if a certain change type is detected in four out of five consecutive inspection cycles, the repeatability of the change event is 0.8. Higher repeatability indicates more stable detection results for that change type. The preset quality threshold is a value used to determine whether the response quality is acceptable. In this embodiment, the preset quality threshold for SNR is set to 3.0, and the preset quality threshold for repeatability of change events is set to 0.6. The signal-to-noise ratio (SNR) threshold is set to 3.0 because in image difference detection, when the SNR is below 3.0, the difference in response values ​​between the noise region and the effective change region is not obvious, and the reliability of the detection results decreases. The repeatability threshold is set to 0.6 because when it is below this value, it indicates that the detection results are more random and cannot stably reflect changes in the device status.

[0057] In practice, after completing the current inspection cycle, the robot acquires the response quality indicators of each difference response map over multiple consecutive inspection cycles. Assume the semantic label of a device instance is an indicator light area, and the set of change types includes indicator light color changes. The signal-to-noise ratio (SNR) of the difference response map corresponding to this change type in five consecutive inspection cycles is 3.5, 3.2, 2.8, 2.9, and 2.7, respectively. After the fifth inspection cycle is completed, the average SNR of the most recent three consecutive inspection cycles is calculated to be 2.8, which is lower than the preset quality threshold of 3.0. At this point, the difference comparison parameters corresponding to this change type are adjusted according to a preset adjustment step size. The preset adjustment step size refers to the amount of parameter change during each adjustment. In this embodiment, the adjustment step size for color channel weights is set to 0.05, and the adjustment step size for edge detection operator thresholds is set to 5. The adjustment direction is determined based on the parameter type. For color channel weights, if the SNR is low, the weight of the color channel most relevant to the physical characteristics of the change type is increased, while the weights of other channels are decreased. For example, the color change of the indicator light is mainly reflected in the change of the ratio of the red and green channels. Therefore, the weight of the red channel is increased from 0.4 to 0.45, the weight of the green channel is increased from 0.4 to 0.45, and the weight of the blue channel is decreased from 0.2 to 0.1. For the edge detection operator threshold, if the signal-to-noise ratio is low, the threshold is increased to suppress noise. After adjustment, the response quality index of the adjusted difference response map is recalculated. If it is still lower than the preset quality threshold, the adjustment continues at the preset adjustment step size until the response quality index is not lower than the preset quality threshold, or the number of adjustments reaches the preset maximum number of adjustments. In this embodiment, the preset maximum number of adjustments is set to 5. If the response quality index is still lower than the preset quality threshold after reaching the maximum number of adjustments, the current difference comparison parameters are retained and a parameter anomaly prompt message is generated to prompt the maintenance personnel to check the sensor status or environmental conditions. If the response quality index is not lower than the preset quality threshold, the adjusted difference comparison parameters are updated to the first mapping relationship for use in subsequent inspection cycles.

[0058] In a preferred embodiment, the three-dimensional semantic map further includes a second mapping relationship between the semantic tags of the device instance and the light-sensitive area; The step of performing pixel-level difference comparison on the aligned image includes: When the difference in lighting conditions between the historical inspection image and the current inspection image exceeds the preset lighting tolerance range, the second mapping relationship is queried according to the semantic tag of the device instance, and the image area of ​​the device instance is divided into a light-sensitive area and a light-insensitive area. Pixel-level difference comparison is performed directly on the light-insensitive area, and pixel-level difference comparison is performed on the light-sensitive area after light correction; When the difference in lighting conditions does not exceed the preset lighting tolerance range, pixel-level difference comparison is directly performed on the aligned image.

[0059] Specifically, when performing pixel-level difference comparisons on aligned images, changes in lighting conditions may cause pixel value differences not due to changes in device status, but rather to changes in ambient lighting. For example, switching on or off server room lights, changes in the brightness of device indicator lights, and adjustments to the automatic exposure parameters of robot cameras can all lead to differences in image pixel values ​​for the same device instance across different inspection cycles. If these spurious changes are not addressed, they will be misjudged as minor events in pixel-level difference comparisons.

[0060] In this embodiment, the second mapping relationship refers to the correspondence between the semantic tags of a device instance and light-sensitive areas. Light-sensitive areas refer to regions on a device instance that are self-illuminating or sensitive to changes in the angle of incident light. Self-illuminating areas include, for example, indicator light panels. Indicator lights emit different colors of light in different operating states, and their brightness is also affected by their own driving circuitry. When camera exposure parameters change, the pixel values ​​of the indicator light area change accordingly. Regions sensitive to changes in the angle of incident light include, for example, reflective metallic areas on the device surface. The reflective properties of metallic surfaces mean that even a small change in the angle of incident light can lead to a significant change in the intensity of reflected light. Light-insensitive areas refer to regions on a device instance that only reflect ambient light and whose reflective properties do not change significantly with the angle of incident light, such as the matte surface of a device casing or the plastic surface of a cable interface.

[0061] The second mapping relationship can be established either manually or through historical data statistics. Manual annotation involves maintenance personnel labeling areas that emit their own light as light-sensitive areas, areas sensitive to changes in the incident light angle as light-sensitive areas, and areas that only reflect ambient light and whose reflection characteristics do not significantly change with the incident light angle as light-insensitive areas. Historical data statistics involve calculating the light sensitivity of the image region corresponding to each semantic label using historical inspection data, as described later. The light-sensitive and light-insensitive areas are then determined based on a comparison between the light sensitivity and a preset sensitivity threshold.

[0062] The preset lighting tolerance range is a threshold used to determine whether the difference in lighting conditions between two images is negligible. In this embodiment, the lighting condition parameters include ambient light intensity and color temperature. The difference in lighting conditions is calculated by calculating the difference in ambient light intensity and color temperature between the historical inspection image and the current inspection image. If the difference in light intensity exceeds a preset intensity threshold or the difference in color temperature exceeds a preset color temperature threshold, the difference in lighting conditions is determined to exceed the preset lighting tolerance range. In this embodiment, the preset intensity threshold is set to 50 lux, and the preset color temperature threshold is set to 200 Kelvin. These values ​​are chosen because when the change in light intensity is less than 50 lux, the pixel value change does not exceed the preset difference threshold for most cameras; and when the change in color temperature is less than 200 Kelvin, the color shift after white balance correction is not significant.

[0063] In practice, after the robot collects the current inspection image during the current inspection cycle, it obtains the lighting condition parameters corresponding to both the historical inspection images and the current inspection image. These lighting condition parameters can be obtained through the robot's onboard lighting sensor or extracted from the image data, such as estimating the color temperature by calculating the RGB mean of the white reference area in the image. If the lighting condition difference exceeds a preset lighting tolerance range, the robot queries a second mapping relationship based on the semantic label of the device instance to divide the image area of ​​the device instance into lighting-sensitive and lighting-insensitive areas. For example, if the semantic label of the device instance is an indicator light area, querying the second mapping relationship reveals that the image area corresponding to this semantic label is a lighting-sensitive area. If the semantic label of the device instance is a latch area, querying the second mapping relationship reveals that the image area corresponding to this semantic label is a lighting-insensitive area. Pixel-level difference comparison is directly performed on the lighting-insensitive areas, i.e., the comparison is performed according to the aforementioned pixel-level difference comparison method. Pixel-level difference comparison is performed on the lighting-sensitive areas after lighting correction. The purpose of illumination correction is to adjust the pixel values ​​of light-sensitive areas in the current inspection image to match the illumination conditions of historical inspection images, eliminating pixel value changes caused by differences in illumination conditions. If the difference in illumination conditions does not exceed the preset illumination tolerance range, it means that the impact of illumination condition changes on pixel values ​​is negligible, and pixel-level difference comparison can be performed directly on the aligned image.

[0064] In a preferred embodiment, the light-sensitive area is determined in the following way: Obtain statistical data on the pixel value changes of the image region corresponding to the semantic tag of the device instance under different lighting conditions during historical inspections; Based on the pixel value change statistics, the illumination sensitivity of the image region corresponding to the semantic tag is determined; When the light sensitivity is higher than a preset sensitivity threshold, the image region corresponding to the semantic tag is determined as a light-sensitive region; otherwise, it is determined as a light-insensitive region. The correspondence between the semantic tags and the determined light-sensitive or light-insensitive regions is stored in the second mapping relationship.

[0065] In this embodiment, the lighting condition parameters include ambient light intensity and color temperature. The pixel value change statistics refer to the statistical results of the pixel value change amplitude of the image region corresponding to the semantic tag under different lighting condition parameters. The lighting sensitivity is calculated as follows: a first ratio between the pixel value change amplitude and the light intensity change amplitude, and a second ratio between the pixel value change amplitude and the color temperature change amplitude are calculated respectively, and the larger of the first ratio and the second ratio is taken as the lighting sensitivity. The preset sensitivity threshold is determined based on the statistical distribution of the lighting sensitivity of all semantic tag-corresponding image regions in historical inspection data, and the median is taken as the preset sensitivity threshold. The median is chosen because it reflects the typical level of lighting sensitivity, avoiding the influence of extreme values ​​on the threshold.

[0066] In practice, during historical inspections, the robot records the pixel values ​​of an image region corresponding to a specific semantic label under different lighting conditions. For example, for an image region labeled as an indicator light, the average RGB value is (200, 50, 50) when the ambient light intensity is 300 lux and the color temperature is 4000 Kelvin; and (220, 60, 55) when the ambient light intensity is 500 lux and the color temperature is 5000 Kelvin. The pixel value variation is calculated as follows: the variations in the three RGB channels are 20, 10, and 5 respectively. The Euclidean distance is used as the pixel value variation range, i.e., sqrt(20... 2 +10 2 +5 2 The light intensity variation is approximately 22.9. The light intensity variation is calculated to be 200 lux, and the color temperature variation is 1000 Kelvin. The first ratio is 22.9 / 200≈0.115, and the second ratio is 22.9 / 1000≈0.023. The larger value, 0.115, is taken as the light sensitivity of the indicator area. For image areas with the semantic label "lock area," under the same lighting conditions, the pixel value variation may only be around 3. Therefore, the first ratio is 3 / 200=0.015, and the second ratio is 3 / 1000=0.003. The larger value, 0.015, is taken as the light sensitivity. Assuming the median light sensitivity of all semantic label-corresponding image areas in historical data is 0.05, the preset sensitivity threshold is 0.05. The light sensitivity of the indicator area, 0.115, is higher than 0.05, and is therefore identified as a light-sensitive area. The light sensitivity of the latching area is 0.015, which is lower than 0.05, and it is determined to be a light-insensitive area. The correspondence between the semantic tags and the determined light-sensitive or light-insensitive areas is stored in the second mapping relationship.

[0067] In a preferred embodiment, the illumination correction of the light-sensitive area includes: Obtain the illumination response characteristics of the illumination-sensitive area during historical inspections; The change in illumination condition parameters is determined based on the illumination condition parameters corresponding to the historical inspection images and the illumination condition parameters corresponding to the current inspection images. The pixel values ​​of the light-sensitive areas in the current inspection image are corrected based on the light response characteristics and the changes in the light condition parameters.

[0068] In this embodiment, the illumination response feature refers to the mapping relationship between the pixel values ​​of the illumination-sensitive area and the changes in illumination condition parameters. The change in illumination condition parameters refers to the difference between the illumination condition parameters corresponding to the current inspection image and the illumination condition parameters corresponding to historical inspection images.

[0069] In practice, for a specific light-sensitive area, its light response characteristics during historical inspections are acquired. The light response characteristics are acquired by collecting the light condition parameters and corresponding pixel values ​​of the light-sensitive area across multiple historical inspection cycles, and establishing a correspondence between pixel values ​​and light condition parameters. For example, for an indicator light area, historical inspections recorded pixel values ​​as (200, 50, 50) at a light intensity of 300 lux and a color temperature of 4000 Kelvin, (210, 55, 52) at a light intensity of 400 lux and a color temperature of 4500 Kelvin, and (220, 60, 55) at a light intensity of 500 lux and a color temperature of 5000 Kelvin. Based on these data, a mapping relationship between pixel values ​​of the light-sensitive area and changes in light intensity and color temperature can be fitted. In this embodiment, a linear fitting method is used, assuming a linear relationship between pixel values ​​and light intensity and color temperature, and obtaining linear coefficients through least squares fitting. For the red channel, the fitted pixel value changes by 0.1 lux with light intensity and by 0.005 Kelvin with color temperature. For the green channel, the rates of change are 0.05 lux and 0.003 Kelvin, respectively. For the blue channel, the rates of change are 0.025 lux and 0.002 Kelvin, respectively.

[0070] Then, based on the lighting condition parameters corresponding to the historical inspection images and the current inspection image, the change in lighting condition parameters is determined. For example, if the lighting intensity of a historical inspection image is 300 lux and the color temperature is 4000 Kelvin, and the lighting intensity of the current inspection image is 500 lux and the color temperature is 5000 Kelvin, then the change in lighting intensity is 200 lux and the change in color temperature is 1000 Kelvin. Based on the lighting response characteristics and the change in lighting condition parameters, the pixel value correction amount for the lighting-sensitive area is calculated. For the red channel, the correction amount is 0.1×200+0.005×1000, i.e., 20+5=25. For the green channel, the correction amount is 0.05×200+0.003×1000, i.e., 10+3=13. For the blue channel, the correction amount is 0.025×200+0.002×1000, i.e., 5+2=7. The corrected pixel value is obtained by subtracting the corresponding correction amount from the pixel value of the light-sensitive area in the current inspection image. For example, if the pixel value of the light-sensitive area in the current inspection image is (220, 60, 55), the corrected pixel value is (220 minus 25, 60 minus 13, 55 minus 7), which is (195, 47, 48). The corrected pixel value is then compared with the pixel values ​​of the same light-sensitive area in historical inspection images at the pixel level.

[0071] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for intelligent inspection and fault location of data center computer rooms using multimodal information fusion, applied to a mobile inspection robot, characterized in that: include: A three-dimensional semantic map of the data center room is constructed. The three-dimensional semantic map is constructed by fusing LiDAR SLAM data, visual inertial odometry data and depth camera data, and includes device instances, the three-dimensional coordinates of the device instances and semantic labels. During the inspection process, multimodal inspection data is collected, including at least two of visible light images, infrared images, ultrasonic data, and gas concentration data. Based on the pose of the mobile inspection robot, the multimodal inspection data is projected onto the three-dimensional semantic map and bound to the corresponding device instance in the three-dimensional semantic map. For the same device instance in the three-dimensional semantic map, based on the three-dimensional coordinates of the device instance in the three-dimensional semantic map and the observation pose during historical and current inspections, the historical inspection images and the current inspection images are aligned to the same observation area of ​​the device instance, and pixel-level difference comparison is performed on the aligned images to detect subtle change events of the device instance. The subtle change events include at least one of the following: indicator light color change, latch displacement, cable displacement, and device surface deformation. The subtle change event is bound to the three-dimensional coordinates of the device instance in the three-dimensional semantic map to generate a fault location result with three-dimensional coordinates. The device knowledge graph is called based on the subtle change event to infer the cause of the fault and generate a maintenance work order with three-dimensional coordinates and fault cause.

2. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The step of performing pixel-level difference comparison on the aligned image to detect subtle change events in the device instance includes: Pixel-level difference comparison is performed on the aligned image to obtain the two-dimensional change region corresponding to the device instance; The depth data collected by the mobile inspection robot during the current inspection is obtained. Based on the camera parameters when the current inspection image is collected and the observation pose of the mobile inspection robot, the two-dimensional change area is back-projected onto the coordinate system of the three-dimensional semantic map to obtain the three-dimensional spatial position corresponding to the two-dimensional change area. Based on the three-dimensional coordinates and preset dimensions of the device instance in the three-dimensional semantic map, the spatial tolerance range corresponding to the device instance is determined; Determine whether the three-dimensional spatial position falls within the spatial tolerance range corresponding to the device instance. If it does, the two-dimensional change area is determined to be a valid change area, and the subtle change event of the device instance is determined based on the valid change area. If it does not fall within the range, the two-dimensional change area is determined to be an invalid change area and is removed.

3. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The multimodal inspection data includes visible light images, and at least one of infrared images, ultrasonic data, and gas concentration data; the method further includes: Based on the semantic tags of the device instance, the multimodal signal association weights of the device instance are determined, and the multimodal signal association weights represent the degree of physical association between the visible light change event of the device instance and the other modal signals; After detecting the subtle change event, the corresponding multimodal signal association weight is queried based on the event type of the subtle change event and the semantic tag of the device instance; Obtain the feature data of the other modal signals in the current inspection, and calculate the association confidence of the subtle change event based on the association weight of the multimodal signals and the feature data of the other modal signals; The alarm level of the subtle change event is determined based on the correlation confidence level, and a corresponding priority maintenance work order is generated.

4. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The subtle change event can be one or more; after binding the subtle change event to the three-dimensional coordinates of the device instance in the three-dimensional semantic map, the process further includes: When multiple minor change events are detected on the device instance, the three-dimensional spatial distance between the minor change events is calculated based on the three-dimensional coordinates bound to each minor change event. Determine whether there exists at least two subtle change events whose three-dimensional spatial distance is less than a preset aggregation threshold, wherein the preset aggregation threshold is set according to the device size corresponding to the semantic tag of the device instance; If they exist, the at least two minor change events are identified as a spatial clustered change event group, and a high-priority maintenance work order is generated based on the spatial clustered change event group. If none exist, the multiple minor change events are identified as spatially dispersed change events, and a regular priority maintenance work order is generated based on the spatially dispersed change events.

5. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The historical inspection images include local reference images corresponding to the device instance; after binding the subtle change events with the three-dimensional coordinates of the device instance in the three-dimensional semantic map, the process further includes: Record the number of times the aforementioned subtle change event occurs across multiple consecutive inspection cycles; When the number of occurrences of the same type of subtle change event bound to the three-dimensional coordinates of the same device instance reaches a preset persistence threshold in a continuous inspection cycle, the local reference image of the device instance is determined to be outdated. Update the local reference image corresponding to the device instance based on the current inspection image, and use the updated local reference image as the historical inspection image for the next inspection cycle. Specifically, before determining that the local reference image of the equipment instance is outdated, the repeated occurrence of the same type of minor change event in the same equipment instance during continuous inspection cycles is marked as a continuous change event and merged into the same maintenance work order.

6. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The three-dimensional semantic map also includes a first mapping relationship between the semantic tags of the device instance and the set of change types, wherein the set of change types includes at least one change type that may occur on the device structure corresponding to the semantic tag; The step of performing pixel-level difference comparison on the aligned image includes: Based on the semantic tags of the device instance, query the first mapping relationship to determine the set of change types corresponding to the device instance; Based on the physical characteristics of each change type in the set of change types, the difference comparison parameters corresponding to each change type are determined respectively. The difference comparison parameters include at least one of color channel weights, edge detection operators, and texture difference measures. Based on the difference comparison parameters corresponding to each change type, pixel-level difference comparison is performed on the aligned image to obtain the difference response map corresponding to each change type. The subtle change events and their types for the device instance are determined based on each difference response map.

7. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 6, characterized in that, After determining the subtle change events and their types for the device instance based on each difference response map, the process further includes: Obtain the response quality index of each difference response map in multiple consecutive inspection cycles, wherein the response quality index includes at least one of signal-to-noise ratio and repeatability of change events; When the response quality index of the difference response map corresponding to a certain change type is lower than the preset quality threshold, the difference comparison parameter corresponding to the change type is adjusted by a preset adjustment step size until the response quality index of the difference response map corresponding to the change type is not lower than the preset quality threshold, or the number of adjustments reaches the preset maximum number of adjustments. Update the adjusted difference comparison parameters to the first mapping relationship.

8. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 1, characterized in that, The three-dimensional semantic map also includes a second mapping relationship between the semantic tags of the device instance and the light-sensitive area; The step of performing pixel-level difference comparison on the aligned image includes: When the difference in lighting conditions between the historical inspection image and the current inspection image exceeds the preset lighting tolerance range, the second mapping relationship is queried according to the semantic tag of the device instance, and the image area of ​​the device instance is divided into a light-sensitive area and a light-insensitive area. Pixel-level difference comparison is performed directly on the light-insensitive area, and pixel-level difference comparison is performed on the light-sensitive area after light correction; When the difference in lighting conditions does not exceed the preset lighting tolerance range, pixel-level difference comparison is directly performed on the aligned image.

9. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 8, characterized in that, The light-sensitive area is determined in the following way: Obtain statistical data on the pixel value changes of the image region corresponding to the semantic tag of the device instance under different lighting conditions during historical inspections; Based on the pixel value change statistics, the illumination sensitivity of the image region corresponding to the semantic tag is determined; When the light sensitivity is higher than a preset sensitivity threshold, the image region corresponding to the semantic tag is determined as a light-sensitive region; otherwise, it is determined as a light-insensitive region. The correspondence between the semantic tags and the determined light-sensitive or light-insensitive regions is stored in the second mapping relationship.

10. The intelligent inspection and fault location method for data center computer rooms based on multimodal information fusion according to claim 8, characterized in that, The illumination correction of the light-sensitive area includes: Obtain the illumination response characteristics of the light-sensitive area during historical inspections; The change in illumination condition parameters is determined based on the illumination condition parameters corresponding to the historical inspection images and the illumination condition parameters corresponding to the current inspection images. The pixel values ​​of the light-sensitive areas in the current inspection image are corrected based on the light response characteristics and the changes in the light condition parameters.

Images