Data center hardware fault diagnosis method, device, equipment and medium

Abstract

The invention discloses a data center hardware fault diagnosis method, device and equipment and a medium. The method comprises the following steps: synchronously acquiring acoustic signal data and visual image data of target hardware equipment in a data center; according to the acoustic signal data, acoustic features are extracted, target segmentation is carried out on the visual image data through a pre-trained Mask R-CNN model, a plurality of target areas are obtained, and visual features are extracted from the target areas; inputting the acoustic features into the hybrid model to obtain an acoustic fault diagnosis result, and inputting the visual features into an improved YOLOv8 model to obtain a visual fault diagnosis result; the hybrid model comprises a CNN layer and an LSTM layer; the improved YOLOv8 model comprises a backbone network in which a SimAM attention mechanism is introduced and a fault precursor detection head used for identifying non-dominant fault features; and fusing the acoustic fault diagnosis result and the visual fault diagnosis result to obtain a final fault diagnosis result. According to the invention, the comprehensiveness and accuracy of data center hardware fault diagnosis can be improved.

Description

Technical Field

[0001] This invention relates to the field of equipment fault diagnosis technology, and in particular to a method, apparatus, equipment and medium for diagnosing hardware faults in data centers. Background Technology

[0002] With the deepening development of the digital economy, data centers have become key computing power hubs supporting social operations and industry innovation. The continuous and stable operation of the servers, storage devices, network switches, and power supply units within these centers directly affects the reliability of data services, business continuity, and operating costs. If hardware failures are not detected and addressed in a timely manner, they can lead to minor service performance degradation or even system-wide outages, causing incalculable economic losses and reputational damage. Therefore, efficient and accurate fault diagnosis of data center hardware, especially early warning of potential faults, is a core technological challenge in ensuring the high availability and reliability of data centers.

[0003] Currently, fault diagnosis technologies for data center hardware typically rely on various sensors deployed on the equipment (such as temperature sensors and current sensors) to monitor single physical quantities, or on independent image recognition and acoustic analysis methods for detection. These existing technologies have a core flaw: their diagnostic capabilities heavily depend on single-dimensional data or characteristic performance. They are insufficient at capturing early, subtle, and non-obvious fault characteristics exhibited before substantial hardware damage occurs (such as slight abnormalities in indicator light brightness / frequency, minor increases in localized equipment temperature, or specific abnormal noises caused by initial wear of mechanical parts). This makes early fault identification and warning ineffective; alarms are often only triggered after fault characteristics become significant or the hardware has partially failed, missing the optimal window for preventative maintenance and failing to meet the extremely high requirements of modern data centers for "zero-interruption" operations. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for diagnosing hardware faults in data centers. By integrating acoustic and visual data from data center hardware status monitoring, it enables early fault diagnosis of hardware devices, thereby improving the comprehensiveness and accuracy of fault diagnosis.

[0005] In a first aspect, embodiments of the present invention provide a data center hardware fault diagnosis method, including: Simultaneously acquire acoustic signal data and visual image data of target hardware devices within the data center; Based on the acoustic signal data, the acoustic features of the target hardware device are extracted, and the visual image data is segmented using a pre-trained Mask R-CNN model to obtain several target regions, so as to extract the visual features of the target hardware device from each target region. The acoustic features are input into a pre-trained hybrid model to output acoustic fault diagnosis results, and the visual features are input into a pre-trained improved YOLOv8 model to output visual fault diagnosis results. The hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network incorporating a SimAM attention mechanism and a fault precursor detection head for identifying inconspicuous fault features. These inconspicuous fault features include abnormal indicator light flickering, abnormal local temperature, and localized equipment deformation. A consistency analysis is performed on the acoustic fault diagnosis results and the visual fault diagnosis results, and a fusion weight is assigned based on the consistency analysis results. The acoustic fault diagnosis results and the visual fault diagnosis results are then fused according to the assigned fusion weights to obtain the final fault diagnosis result of the target hardware device.

[0006] This invention, through the simultaneous acquisition of acoustic and visual data with distinct physical properties, leverages their complementary information to expand the dimensions of fault information capture from the source, laying the foundation for subsequent robust diagnosis. At the acoustic level, acoustic features are extracted; at the visual level, key areas such as target devices and indicator lights are precisely separated (purified) from the complex background of the server room (cables, cabinets, dust) to obtain visual features. This extraction of both acoustic and visual features avoids interference from irrelevant information in subsequent diagnostic identification. Furthermore, by combining a hybrid model of CNN and LSTM layers, acoustic fault diagnosis is performed based on acoustic features to improve acoustic accuracy. To improve the accuracy of fault diagnosis, this invention improves the YOLOv8 model by adaptively and pixel-level enhancing the weights of key visual features related to faults (such as high-brightness indicator lights and high-temperature areas) using the SimAM attention mechanism. It utilizes a fault precursor detection head to identify weak, non-obvious early fault features such as brightness changes, grayscale differences, and morphological deformation, thereby enhancing the accuracy of visual fault diagnosis. Consistency analysis is used for verification to avoid misjudgments from a single modality dominating the final result. Weight allocation is based on the consistency analysis results, integrating acoustic and visual diagnostic conclusions to output a more reliable and comprehensive final diagnosis. Compared to existing single-dimensional hardware fault diagnosis methods, this invention integrates acoustic and visual data from data center hardware status monitoring for early hardware fault diagnosis, thus improving the comprehensiveness and accuracy of fault diagnosis. In some preferred embodiments of the first aspect, the visual features are input into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results, specifically: The visual features are input into the improved YOLOv8 model; The visual features are enhanced by using the backbone network that incorporates the SimAM attention mechanism to obtain the enhanced visual features. The fault precursor detection head identifies non-obvious fault features in the enhanced visual features, and the non-obvious fault feature identification result is obtained. Based on the non-obvious fault feature identification results, visual fault diagnosis results are obtained and output; wherein, the visual fault diagnosis results include visual fault type, visual fault confidence level, and fault location coordinates.

[0007] This invention combines the SimAM attention mechanism with a fault precursor detection head for identifying insignificant fault features to continuously focus on and enhance features associated with insignificant faults, thereby improving the accuracy of the final visual fault diagnosis results.

[0008] In some preferred embodiments of the first aspect, the visual features are enhanced through the backbone network incorporating the SimAM attention mechanism to obtain enhanced visual features, specifically as follows: The enhanced visual features are input into the backbone network, and the backbone network outputs at least one intermediate feature map. For each pixel in the intermediate feature map, the local mean and local variance of each pixel are calculated based on the pixel value within the preset local window where each pixel is located, and the global mean of the intermediate feature map is calculated based on the pixel values ​​of all pixels in the intermediate feature map. Based on the local mean, local variance, and global mean, calculate the saliency score for each pixel, and generate the attention weight corresponding to each pixel based on the saliency score. The enhanced visual features are obtained by multiplying the original pixel value of each pixel in the intermediate feature map by the attention weight of the corresponding pixel.

[0009] The embodiments of the present invention quantify the difference between each pixel and its surroundings and the global background, thereby accurately amplifying fault features with extremely low computational cost, suppressing noise, and improving the model's sensitivity to minor, local fault features and its ability to resist environmental interference.

[0010] In some preferred embodiments of the first aspect, the fault precursor detection head includes a brightness change enhancement branch, a grayscale difference enhancement branch, and a shape deformation enhancement branch; Specifically, the fault precursor detection head identifies non-obvious fault features in the enhanced visual features to obtain non-obvious fault feature identification results, as follows: Through the brightness change enhancement branch, the brightness change features in the enhanced visual features are extracted and enhanced using temporal difference convolution to obtain the first branch enhancement features; Through the gray-level difference enhancement branch, the gray-level difference features in the enhanced visual features are extracted and enhanced using adaptive threshold convolution to obtain the second branch enhancement features; Through the morphological deformation enhancement branch, edge-aware convolution is used to extract and enhance the morphological change features in the enhanced visual features to obtain the third branch enhancement features; The first branch enhancement features, the second branch enhancement features, and the third branch enhancement features are fused to obtain fused features. Based on the fused features, non-obvious fault features are identified to obtain non-obvious fault feature identification results.

[0011] This invention solves the problem of "weak signals being easily drowned out" by using time-series differential capture of flicker, using adaptive threshold to distinguish subtle temperature differences, and using edge sensing to detect minute deformations, thus transforming the identification of early fault characteristics from "possible" to "reliable".

[0012] In some preferred embodiments of the first aspect, the acoustic features of the target hardware device are extracted based on the acoustic signal data, specifically as follows: The acoustic signal data is denoised using a preset wavelet threshold denoising algorithm. Time-frequency analysis is performed on the denoised acoustic signal data to extract features of different dimensions to generate several acoustic feature vectors; wherein, the features of different dimensions include Mel frequency cepstral coefficients, power spectral density, and zero-crossing rate; The acoustic feature vectors are sequentially concatenated using a preset sliding time window to obtain the acoustic features of the target hardware device.

[0013] This invention transforms and refines raw, noisy acoustic waveform signals into high-quality feature representations with temporal structure that can clearly characterize the health status of devices and are suitable for subsequent model (CNN+LSTM) processing, thereby ensuring the accuracy of subsequent diagnostic results.

[0014] In some preferred embodiments of the first aspect, the step of segmenting the visual image data using a pre-trained Mask R-CNN model to obtain several target regions specifically involves: The first candidate region is extracted from the visual image data by using the preset region proposal network in the Mask R-CNN model based on the preset prior feature template; The RoI Align operation is performed on the candidate regions to uniformly map the first candidate regions to a feature map of a fixed size. The feature map is then subjected to target classification and bounding box regression to determine the region category and bounding box coordinates of the candidate regions, thereby obtaining the second candidate regions. The region categories include the device body, indicator lights, temperature anomaly areas, and background. For the second candidate region, pixel-level prediction is performed through the preset mask branch in the Mask R-CNN model to generate a binary mask representing the contour of the target region. By combining the bounding box coordinates with the binary mask, an independent target region is cropped from the second visual image data; wherein, the target region includes at least one of the device area sub-image, indicator light sub-image, and temperature anomaly sub-image.

[0015] This invention provides a way to stably and automatically separate individual, analyzable images of equipment components (sub-images) from a panoramic view of a computer room containing various devices and a complex background, thereby ensuring the accuracy and purity of subsequent visual feature extraction and diagnostic objects.

[0016] In some preferred embodiments of the first aspect, the step of inputting the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results specifically includes: The acoustic features are input into the hybrid model; The spatial dimension information of the acoustic features is extracted through the CNN layer in the hybrid model to obtain the acoustic spectrum difference results. By analyzing the temporal variation patterns of the acoustic features through the LSTM layer of the hybrid model, the acoustic evolution trend can be obtained. By combining the acoustic spectrum difference results and the acoustic evolution trend, acoustic fault diagnosis results are obtained and output; wherein, the acoustic fault diagnosis results include acoustic fault type, acoustic fault confidence level and fault development stage.

[0017] This invention utilizes both spatial (spectrum) and temporal (trend) information of acoustic signals for diagnosis, thereby determining not only "what kind of fault it is" but also "what stage the fault has developed to," providing crucial temporal dimension information for fault classification and early warning.

[0018] Secondly, embodiments of the present invention provide a data center hardware fault diagnosis device, including a data acquisition module, a feature extraction module, a single-modal result acquisition module, and a final result acquisition module, wherein... The data acquisition module is used to simultaneously acquire acoustic signal data and visual image data of target hardware devices within the data center; The feature extraction module is used to extract the acoustic features of the target hardware device based on the acoustic signal data, and to perform target segmentation on the visual image data through a pre-trained Mask R-CNN model to obtain several target regions, so as to extract the visual features of the target hardware device from each of the target regions. A single-modal result acquisition module is used to input the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results, and to input the visual features into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results; wherein, the hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network incorporating a SimAM attention mechanism and a fault precursor detection head for identifying non-obvious fault features; the non-obvious fault features include abnormal indicator light flickering, abnormal local temperature, and local deformation of the equipment; The final result acquisition module is used to perform consistency analysis on the acoustic fault diagnosis results and the visual fault diagnosis results, and to allocate fusion weights based on the consistency analysis results, so as to fuse the acoustic fault diagnosis results and the visual fault diagnosis results according to the allocated fusion weights to obtain the final fault diagnosis result of the target hardware device.

[0019] This invention employs a data acquisition module to simultaneously collect acoustic and visual data, two types of data with vastly different physical properties. Leveraging their complementary information, this expands the scope of fault information capture from the source, laying the foundation for robust subsequent diagnosis. A feature extraction module extracts acoustic features at the acoustic level and precisely separates (purifies) key areas such as target devices and indicator lights from the complex background of the server room (cables, cabinets, dust) at the visual level to obtain visual features. This extraction of both acoustic and visual features avoids interference from irrelevant information in subsequent diagnostic identification. Finally, a single-modal result acquisition module utilizes a hybrid model combining CNN and LSTM layers to identify acoustic faults based on acoustic features. To improve the accuracy of acoustic fault diagnosis, the system employs an improved YOLOv8 model. The SimAM attention mechanism adaptively and pixel-wise enhances the weights of key fault-related visual features (such as high-brightness indicator lights and high-temperature areas). A fault precursor detection head is used to identify weak, non-obvious early fault features such as brightness changes, grayscale differences, and morphological deformation, thereby improving the accuracy of visual fault diagnosis. The final result acquisition module uses consistency analysis for verification, avoiding the dominance of single-modality misjudgments in the final result. Weight allocation is performed based on the consistency analysis results, integrating acoustic and visual diagnostic conclusions to output a more reliable and comprehensive final diagnostic result.

[0020] Thirdly, embodiments of the present invention provide a terminal device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation of the data center hardware fault diagnosis method as described in any of the above.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device or apparatus where the computer-readable storage medium is located to perform the data center hardware fault diagnosis method as described in any of the above.

[0022] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a data center hardware fault diagnosis method provided in an embodiment of the present invention; Figure 2 This is a structural diagram of a data center hardware fault diagnosis device provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1: like Figure 1 As shown, an embodiment of the present invention provides a data center hardware fault diagnosis method, which includes the following steps: S101, synchronously collects acoustic signal data and visual image data of target hardware devices in the data center; In one specific embodiment, after simultaneously acquiring visual and acoustic data from data center hardware devices, spatiotemporal alignment of the dual-modal data is also required. The process of acquiring dual-modal data and performing spatiotemporal alignment specifically involves: using high-definition industrial cameras (resolution ≥ 1920×1080, frame rate 25fps) to acquire visual data, deployed according to rack partitions, covering core hardware such as servers, switches, power supplies, and storage devices. The acquired content includes: device indicator light status (color, flashing frequency), chassis surface temperature distribution (acquired with the assistance of an infrared imaging module), device physical deformation (such as loose fan casing, detached interfaces), and hardware operating status (such as abnormal hard drive read / write indicator lights, power switch status). The camera has a built-in light sensor that supports adaptive exposure adjustment to adapt to different lighting environments in the data center. A high-sensitivity microphone array (sampling rate ≥44.1kHz, signal-to-noise ratio ≥80dB) is deployed inside and on top of the cabinet to collect acoustic signals generated by the equipment, including: abnormal fan noises (such as irregular noises caused by stuttering or uneven speed), abnormal hard drive read / write noises (such as knocking sounds caused by head failure), power supply discharge noises (such as the buzzing sound of a short circuit precursor), and electrical sparking sounds from poor contact at switch ports. The microphone array supports beamforming technology to focus the acoustic signals of the target device and suppress interference from environmental noise (air conditioning, fresh air system noise). Through timestamp synchronization technology, visual data and acoustic data from the same device at the same time are bound to generate "visual-acoustic" bimodal data pairs to ensure the spatiotemporal consistency of subsequent fusion diagnostics.

[0026] S102, based on the acoustic signal data, extract the acoustic features of the target hardware device, and use a pre-trained Mask R-CNN model to segment the visual image data to obtain several target regions, so as to extract the visual features of the target hardware device from each target region. In this embodiment, the acoustic features of the target hardware device are extracted based on the acoustic signal data. Specifically, the acoustic signal data is denoised using a preset wavelet threshold denoising algorithm; time-frequency analysis is performed on the denoised acoustic signal data to extract features of different dimensions to generate several acoustic feature vectors; wherein the features of different dimensions include Mel frequency cepstral coefficients, power spectral density, and zero-crossing rate; the acoustic feature vectors are time-series concatenated through a preset sliding time window to obtain the acoustic features of the target hardware device.

[0027] In one specific embodiment, the acoustic features of the target hardware device are extracted based on the acoustic signal data. Specifically, a wavelet threshold denoising algorithm is used to filter out environmental noise and acoustic signals from irrelevant devices; time-frequency analysis is performed on the denoised audio signal to extract acoustic features such as Mel-frequency cepstral coefficients (MFCC), power spectral density (PSD), short-time energy, and zero-crossing rate, generating an acoustic feature vector; a sliding window method (window size 0.5s, step size 0.1s) is used to temporally stitch the acoustic features to capture the acoustic change trend of fault precursors.

[0028] In this embodiment, the step of segmenting the visual image data using a pre-trained Mask R-CNN model to obtain several target regions specifically involves: extracting a first candidate region from the visual image data using a preset region proposal network in the Mask R-CNN model based on a preset prior feature template; performing RoI Align operation on the candidate region to uniformly map the first candidate region to a feature map of a fixed size; performing target classification and bounding box regression on the feature map to determine the region category and bounding box coordinates of the candidate region, thereby obtaining a second candidate region; wherein the region category includes device body, indicator light, temperature anomaly region, and background; performing pixel-level prediction on the second candidate region using a preset mask branch in the Mask R-CNN model to generate a binary mask representing the contour of the target region; and combining the bounding box coordinates with the binary mask to crop an independent target region from the second visual image data; wherein the target region includes at least one of a device region sub-image, an indicator light sub-image, and a temperature anomaly sub-image.

[0029] Preferably, before inputting the visual image data into the Mask R-CNN model, the visual image data needs to be preprocessed: an adaptive median filtering algorithm is used to remove noise caused by dust and light reflection in the image; histogram equalization and gamma correction are used to optimize the image quality under low light conditions and improve detail contrast.

[0030] In one specific embodiment, a pre-trained Mask R-CNN model is used to segment the visual image data to obtain several target regions, specifically: (a) The recognition criteria of the Mask R-CNN model (supported by input data) The model does not rely on a single data dimension, but rather performs target recognition based on multi-dimensional visual data from images, ensuring the accuracy and specificity of segmentation. Specifically, this includes: A. Basic image pixel data: The raw images (resolution ≥ 1920×1080) captured by high-definition industrial cameras contain complete visual information such as equipment appearance, cabinet layout, and environmental background, and are the basic data carrier for segmentation; B. Infrared temperature correlation data: The surface temperature distribution data of the chassis collected by the infrared imaging module is superimposed on the original image in the form of a grayscale matrix to form a dual-modal image input of "visible light + infrared" to help identify the physical equipment location corresponding to the temperature abnormal area; C. Pre-set target feature templates: Based on the characteristics of data center hardware, import the prior feature templates of targets such as devices (servers, switches, power modules), indicator lights (different colors, shapes), and interfaces in advance (such as device edge contour parameters, indicator light size range, grayscale range corresponding to infrared temperature anomaly thresholds) to provide recognition references for the model. D. Scene Adaptation Optimization Data: Based on the characteristics of the computer room environment, input image data after adaptive exposure adjustment (the camera's built-in light sensor compensates for low light / strong light interference) to reduce the impact of environmental noise on target recognition.

[0031] (II) Specific Operational Procedures for Target Segmentation Phase 1: Feature Extraction and Region Candidate Generation (RPN Network Role) A. The model first extracts features from the input image through a convolutional neural network (CNN) backbone network (such as ResNet+FPN) to generate multi-scale feature maps (taking into account both the overall outline of the device and detailed features such as indicator lights). B. Based on the feature map, the Region Proposal Network (RPN) traverses the image through a sliding window and filters out candidate regions (RegionProposals) that may contain the target by using preset anchor boxes (adapting to different sizes of targets such as devices, indicator lights, and infrared temperature areas), thus initially eliminating irrelevant backgrounds such as empty areas and gaps between cabinets.

[0032] Phase 2: Object classification, bounding box regression, and region of interest alignment A. For the candidate regions generated by RPN, the RoI Align operation is used to uniformly map candidate regions of different sizes to feature maps of a fixed size, ensuring consistency in subsequent processing; B. Use a fully connected layer to make two types of judgments on the candidate region: ① Target category classification (such as preset categories such as "server chassis", "power indicator", "infrared temperature anomaly area", "background"); ② Bounding box regression (correct the coordinates of the candidate region so that the selected area fits the actual outline of the target). C. Filter out background areas and misidentified areas with confidence levels below the threshold, and retain only target candidate areas with high confidence levels (such as indicator light areas and device main body areas with confidence levels ≥ 0.7).

[0033] Third stage: Pixel-level mask generation (core segmentation operation) A. For high-confidence target regions after classification, the model performs pixel-level prediction of the region through a parallel masking branch (Fully Convolutional Network FCN); B. The mask branch outputs a binary mask with the same size as the target area, where "1" represents a pixel belonging to the target area and "0" represents a background pixel (e.g., pixels within the device area are marked as 1, while surrounding cabinet frames, dust spots, etc. are marked as 0). By combining bounding box coordinates and binary masks, the target region (ROI) is accurately cropped from the original image, resulting in independent device region sub-images, indicator light sub-images, infrared temperature anomaly sub-images, etc., thus completing the segmentation operation.

[0034] It should be noted that fault-related features extracted from the ROI include: indicator light color features (RGB values), flashing frequency features (time series statistics), temperature distribution features (infrared image grayscale value matrix), and device morphology features (edge ​​contours, geometric parameters).

[0035] S103, the acoustic features are input into a pre-trained hybrid model to output acoustic fault diagnosis results, and the visual features are input into a pre-trained improved YOLOv8 model to output visual fault diagnosis results; wherein, the hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network incorporating a SimAM attention mechanism and a fault precursor detection head for identifying inconspicuous fault features; the inconspicuous fault features include abnormal indicator light flickering, abnormal local temperature, and local deformation of the equipment; In this embodiment, the acoustic features are input into a pre-trained hybrid model to output acoustic fault diagnosis results. Specifically, the acoustic features are input into the hybrid model; the spatial dimension information of the acoustic features is extracted through the CNN layer in the hybrid model to obtain acoustic spectrum difference results; the temporal variation law of the acoustic features is analyzed through the LSTM layer in the hybrid model to obtain acoustic evolution trends; and the acoustic spectrum difference results and acoustic evolution trends are combined to obtain and output acoustic fault diagnosis results. The acoustic fault diagnosis results include acoustic fault type, acoustic fault confidence, and fault development stage.

[0036] In one specific embodiment, the acoustic features are input into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results. Specifically, the spatial dimension information of the acoustic features is extracted through a 3-layer convolutional kernel (3×3) to capture the acoustic spectral differences of different faults; the temporal variation law of the acoustic features is learned through a 2-layer bidirectional LSTM network to identify the acoustic evolution trend of fault precursors (such as fan noise from none to present, from weak to strong); the type of equipment acoustic fault (such as fan noise, hard disk failure, power supply discharge), fault confidence (0-1), and fault development stage (early / mid / late stage).

[0037] In this embodiment, the visual features are input into a pre-trained improved YOLOv8 model to enable the improved YOLOv8 model to output visual fault diagnosis results. Specifically, the visual features are input into the improved YOLOv8 model; the visual features are enhanced through the backbone network that incorporates the SimAM attention mechanism to obtain enhanced visual features; non-obvious fault features in the enhanced visual features are identified through the fault precursor detection head to obtain non-obvious fault feature identification results; and visual fault diagnosis results are obtained and output based on the non-obvious fault feature identification results. The visual fault diagnosis results include visual fault type, visual fault confidence level, and fault location coordinates.

[0038] In this embodiment, the visual features are enhanced through the backbone network that introduces the SimAM attention mechanism to obtain enhanced visual features. Specifically, the enhanced visual features are input into the backbone network, and the backbone network outputs at least one intermediate feature map. For each pixel in the intermediate feature map, the local mean and local variance of each pixel are calculated based on the pixel value within a preset local window. The global mean of the intermediate feature map is calculated based on the pixel values ​​of all pixels in the intermediate feature map. The saliency score of each pixel is calculated based on the local mean, local variance, and global mean. The attention weight corresponding to each pixel is generated based on the saliency score. The original pixel value of each pixel in the intermediate feature map is multiplied by the attention weight of the corresponding pixel to obtain the enhanced visual features.

[0039] In one specific embodiment, the visual features are enhanced by the backbone network that introduces the SimAM attention mechanism to obtain the enhanced visual features. Specifically, attention weights are calculated separately for each channel c of the input feature map (each channel corresponds to a type of visual feature, such as brightness features, edge features, and infrared temperature features). The design of independent processing between channels can avoid mutual interference between different types of features and adapt to single-dimensional fault features such as indicator light color and temperature changes.

[0040] Furthermore, for each pixel (i,j) in channel c, a local window is defined (default 3×3, adapted to small targets such as indicator lights and temperature anomaly areas, and can be dynamically adjusted to 5×5 according to the data center hardware size), and its "significance score" s(c,i,j) is calculated. The core formula is designed based on the statistical difference between pixels and local-global data. In the formula, It is the mean value of the local window containing pixel (i,j), reflecting the baseline level of local features around the pixel; It is the global mean of the feature map of the entire channel c, reflecting the overall distribution level of this visual feature; It represents the variance of the local window containing pixel (i,j), reflecting the consistency of features within the local region; To prevent the minimum value of denominator 0 (usually taken as 0), ).

[0041] It should be noted that if pixel (i,j) is a critical fault feature (such as a bright spot on an indicator light), its local window mean value... It will be much higher than the global average. (The numerator increases significantly), and all pixels within the local window except this pixel are background pixels (strong feature consistency, low variance). Smaller denominators result in a significantly increased significance score s(c,i,j); conversely, smaller local and global mean values ​​of background pixels (numerator close to 0) lead to a significantly higher significance score even with larger local variances.

[0042] Furthermore, the saliency score map of each channel is normalized using the Sigmoid function to generate pixel-wise attention weights w(c,i,j)∈[0,1]. After normalization, the weights of key feature pixels are concentrated in the range of 0.8-1.0, while the weights of background noise pixels are concentrated in the range of 0-0.2, further enhancing the weight discrimination. Furthermore, the original feature map is multiplied pixel-by-pixel with the attention weight map, i.e. After weighting, the response values ​​of key fault features are amplified (e.g., the pixel value of indicator light spots is doubled), while the response values ​​of background noise are suppressed (e.g., the pixel value of dust spots approaches 0). The output enhanced feature map can be directly input into the subsequent detection layers of YOLOv8 to improve the recognition accuracy of small targets and abnormal areas.

[0043] For example, indicator lights, as "visual signals" of hardware failure, are characterized by "small size (pixel percentage of a single indicator light ≤ 0.3%), vivid colors (red / green / yellow), and high contrast with the background." SimAM's focusing process is as follows: local-to-global difference amplification. Taking a red light as an example, its R channel pixel value (e.g., 255) is much higher than the surrounding background pixels (the grayscale value of the metal casing of the server rack is about 50-80), causing the local window average of this pixel to be lower. Significantly higher than the global channel mean , numerator It increases exponentially; at the same time, within the indicator's local window (3x3), all pixels except the core pixel are background pixels, resulting in strong feature consistency (variance). The denominator is small, and the final significance score s(c,i,j) is close to the maximum value; background noise suppression: the local mean of background pixels such as cabinet frame, cables, and dust is very close to the global mean (numerator is close to 0). Even if the local variance is large, the weight w(c,i,j) is still close to 0. After weighting, the background features are "masked" to avoid occlusion interference of the small indicator lights; flicker feature enhancement: combined with the temporal detection capability of YOLOv8, SimAM continuously assigns high weight to the high brightness pixels of the "bright frame" during the indicator light flickering process and assigns low weight to the low brightness pixels of the "off frame", which enhances the temporal feature of "bright-off" alternation, so that the model can accurately identify fault precursors such as abnormal flickering frequency (such as the normal 1 time / second becomes 3 times / second).

[0044] For example, areas with abnormal temperatures are captured by an infrared camera and represented by differences in grayscale values ​​(grayscale value ≥200 in high-temperature areas, and 80-120 in normal areas). These areas are characterized by "blurred boundaries and easy confusion with normal high-temperature areas such as equipment heat sinks." SimAM's focusing logic is as follows: grayscale difference quantification; after superimposing the infrared image and the visible light image, the grayscale value of the abnormal temperature area is significantly higher than that of the normal area, and its local window mean... Dominated by pixels in the surrounding normal region, resulting in a difference from the global mean. The difference increases (numerator increases); at the same time, the local window variance of the edge pixels in the abnormal region is small (the features of the normal region are consistent), which further improves the significance score; edge feature enhancement, the edge pixels in the temperature abnormal region have a "sudden change in gray value" (such as jumping from 120 to 200), the difference between their local mean and global mean is greater than that of the pixels inside the region, and the weight w(c,i,j) is higher, which can accurately characterize the boundary contour of the abnormal region and help YOLOv8 locate the abnormal range; normal high temperature region filtering, although the gray value of normal high temperature regions such as equipment heat sinks is high, the difference with the surrounding neighboring pixels is small (numerator is small), and the calculated weight is low, avoiding being misjudged as a temperature abnormality fault, solving the pain point of "confusion between normal high temperature and fault high temperature" in traditional models.

[0045] For example, data center server rooms often suffer from "dense equipment (rack density ≥ 8 units / row) and cluttered backgrounds (tangled cables, labels, dust)." Traditional attention mechanisms are prone to having key features obscured by the background due to global feature learning. SimAM addresses this by employing two key mechanisms: local statistics rather than global learning, focusing only on the difference between a single pixel and its local window, without relying on global feature distribution. Even in scenarios with overlapping racks and cable obstructions, key features can be accurately identified as long as they exhibit local prominence. Furthermore, its parametric design ensures robustness, eliminating the need for parameter fine-tuning with extensive data center data. This avoids overfitting to specific scenarios and allows the model to adapt to different data center rack layouts, equipment types, and lighting conditions, demonstrating strong generalization capabilities.

[0046] It should be noted that SimAM differs from traditional mechanisms. Traditional attention mechanisms (such as SE, CBAM, and ECA) have limitations in data center fault diagnosis scenarios, including "parameter redundancy, coarse granularity, and weak anti-interference." SimAM's design differences are key to its adaptability to complex data center environments and its ability to achieve precise focusing. A comparison is shown in Table 1. Table 1 Comparison of SimAM and Traditional Attention Mechanisms In this embodiment, the fault precursor detection head includes a brightness change enhancement branch, a grayscale difference enhancement branch, and a morphological deformation enhancement branch. Specifically, the fault precursor detection head identifies non-obvious fault features in the enhanced visual features to obtain a non-obvious fault feature identification result. This is achieved by: using the brightness change enhancement branch, extracting and enhancing brightness change features in the enhanced visual features using temporal difference convolution to obtain a first-branch enhancement feature; using the grayscale difference enhancement branch, extracting and enhancing grayscale difference features in the enhanced visual features using adaptive threshold convolution to obtain a second-branch enhancement feature; and using the morphological deformation enhancement branch, extracting and enhancing morphological change features in the enhanced visual features using edge-aware convolution to obtain a third-branch enhancement feature. The first-branch enhancement feature, the second-branch enhancement feature, and the third-branch enhancement feature are then fused to obtain a fused feature. Based on this fused feature, non-obvious fault feature identification is performed to obtain a non-obvious fault feature identification result.

[0047] It should be noted that the fault precursor detection head adopts an architecture design of "multi-branch parallelism + lightweight feature enhancement". It shares the backbone network features with the original fault detection head of YOLOv8 (identifying explicit faults), while adapting to the "weakness, low contrast and easy confusion" characteristics of non-explicit features through an independent branch structure.

[0048] In one specific embodiment, the fault precursor detection head identifies non-obvious fault features in the enhanced visual features to obtain non-obvious fault feature identification results, specifically: (1) The core challenge of non-obvious fault features is "small difference from the background" (e.g., the grayscale difference in the slightly heated area is ≤30, which is much lower than the grayscale difference of obvious faults ≥80). Feature enhancement is achieved through the following methods: For precursors of brightness changes (such as faint flickering indicator lights): calculate the brightness difference between three consecutive frames using temporal differential convolution. This amplifies the subtle changes in the "bright-off" transitions and suppresses static background noise. For grayscale difference precursors (such as slight local temperature rise): Adaptive threshold convolution is used to dynamically calculate the local threshold for each pixel. (μ is the local mean, σ is the local variance, and α and β are adaptive coefficients), regions with gray-level differences higher than the threshold are marked as candidate precursors; For precursors of morphological deformation (such as minor equipment deformation): extract gradient features of the equipment edges through edge-aware convolution and calculate the displacement of edge points. (P_t is the coordinate of the edge point in the current frame, and P_0 is the coordinate of the edge point in the normal state). Areas with displacement exceeding the threshold are identified as candidate deformation areas.

[0049] (2) Non-obvious fault precursors have a "gradual" nature (such as the loosening of the fan casing from none to present, from weak to strong), and timing modeling is needed to avoid false detections caused by single-frame noise: A sliding window method (window size of 5 frames, step size of 1 frame) is used to temporally stitch together the precursor features of consecutive frames to generate a temporal feature vector. ; A lightweight gated recurrent unit (GRU) is introduced to model the temporal feature vector, learn the evolution trend of precursor features (such as gradually increasing brightness variation and gradually expanding deformation region), and output the trend confidence score. The final confidence level is obtained by weighted fusion with the single-frame confidence level. .

[0050] (3) Combining prior knowledge of data center hardware, construct a feature template library of non-obvious fault precursors to achieve accurate matching: The template library includes indicator light flashing frequency templates (e.g., normal flashing frequency 1 time / second, precursor frequency 3 times / second), temperature distribution templates (e.g., normal area grayscale range 80-120, precursor area 121-150), and equipment shape templates (e.g., normal fan edge contour parameters, precursor deformation edge offset threshold). The similarity between the fused feature map and the template library is calculated (using cosine similarity). Candidate regions with a similarity of ≥0.7 are determined as valid precursors, and background interference is further filtered out.

[0051] (4) To address the scarcity of non-obvious fault precursor samples, a hybrid loss function is designed to optimize model training: Classification loss: Focal Loss is adopted. By adjusting the focusing parameter γ=2, the weight of easily classified background samples is reduced and the training priority of difficult-to-classify precursor samples is increased. Regression loss: CIoU Loss is adopted, which takes into account the overlap of candidate boxes, the distance between center points, and the aspect ratio to improve the localization accuracy of subtle precursors; Timing loss: Introducing timing consistency loss This constrains the consistency of precursor features between consecutive frames, reducing false detections caused by temporal fluctuations.

[0052] For example, the core characteristics of a weak flickering indicator light are "abnormal flickering frequency (deviating from the normal range) and small brightness variation amplitude (easily interfered with by light)". The identification process is as follows: Extract the brightness values ​​of the R, G, and B channels from the ROI region (indicator light sub-image) after visual data preprocessing, focusing on fault-related channels (e.g., focusing on the R channel for red lights); calculate the brightness difference of 5 consecutive frames using temporal differential convolution to obtain the brightness change curve (L(t)). The normal flickering curve is a regular sine wave, while the abnormal precursor curve is an irregular fluctuation (e.g., frequency doubling, uneven amplitude); calculate the similarity between the brightness change curve and the "normal flickering template" in the template library. If the similarity is <0.6 and this condition is met for 3 consecutive frames, it is determined to be a precursor to "abnormal weak flickering of the indicator light"; if the confidence of a single frame is 0.6, the confidence increases to 0.75 after accumulating 3 consecutive frames, meeting the effective diagnostic threshold (≥0.7), and the precursor identification result is output.

[0053] For example, the core characteristics of slight localized temperature rise are "small grayscale difference (infrared image), blurred boundaries, and easy confusion with normal heat dissipation areas." The identification process is as follows: Infrared grayscale channel data is extracted from the "visible light + infrared" dual-modal image. The device region ROI is segmented using Mask R-CNN to eliminate background interference such as cabinet gaps. Adaptive threshold convolution is used to calculate the local mean μ and variance σ of each pixel, and the grayscale values ​​are then converted into a single unit. The area is marked as a candidate temperature rise area (corresponding to a temperature increase of 3-5℃); combined with the grayscale range of the normal heat dissipation area of ​​the device in the prior template library (such as the normal grayscale of the heat sink 130-140), the normal heat dissipation area is filtered out, and the candidate area of ​​abnormal slight temperature rise is retained; by monitoring the grayscale change of the candidate area through 5 consecutive frames of images, if the grayscale value continues to rise (≥2 per frame), it is judged as a precursor to "local slight temperature rise", avoiding misjudgment caused by environmental fluctuations.

[0054] For example, the core characteristics of minute device deformation are "small displacement (≤2 pixels), blurred edges, and easy confusion with image noise". The recognition process is as follows: by using edge-aware convolution in the morphological deformation enhancement branch, the gradient features of the device edge contour are extracted to obtain the set of edge points. The current frame's edge point set is compared with the device's normal profile (reference profile) in the template library, and the displacement of each edge point is calculated. If there are more than 5 consecutive edge points with displacement If the area is ≥ 1.5 pixels and forms a continuous region (area ≥ 10 pixels), it is marked as a candidate deformation region. Combined with the weight map output by the SimAM attention module, if the attention weight of the candidate deformation region is ≥ 0.6 (indicating that the region is a critical part of the device), the confidence level is increased by 0.1. When the final confidence level is ≥ 0.7, it is judged as a precursor to "minor device deformation".

[0055] S104, perform consistency analysis on the acoustic fault diagnosis results and the visual fault diagnosis results, and allocate fusion weights according to the consistency analysis results, so as to fuse the acoustic fault diagnosis results and the visual fault diagnosis results according to the allocated fusion weights to obtain the final fault diagnosis result of the target hardware device.

[0056] In one specific embodiment, a consistency analysis is performed on the acoustic fault diagnosis results and the visual fault diagnosis results, and a fusion weight is allocated based on the consistency analysis results. The acoustic fault diagnosis results and the visual fault diagnosis results are then fused according to the allocated fusion weights to obtain the final fault diagnosis result of the target hardware device. Specifically, the output of the visual submodule is obtained as follows: fault type (V_type, such as 6 types including abnormal indicator light / temperature exceeding the standard), fault confidence (V_c, 0-1), fault location coordinates (V_pos), and feature vector (V_feat, dimension D). Obtain the output of the acoustic submodule: fault type (A_type, such as fan noise / hard drive failure, etc., 5 types), fault confidence (A_c, 0-1), fault development stage (A_stage, early / mid / late stage), and feature vector (A_feat, dimension D). Furthermore, V_feat and A_feat are subjected to dimensionality unification and enhancement through 1×1 convolutional layers (ensuring that both have the same dimension, such as 256), generating enhanced visual feature vectors V_feat_enh and acoustic feature vectors A_feat_enh, providing a high-quality data foundation for feature fusion.

[0057] Furthermore, by combining the "credibility + consistency + relevance" of submodule fault identification, a fusion weight is calculated to ensure more accurate weight allocation: Furthermore, the original confidence weighting formula is retained as the basis for weight calculation: ω_v_base = V_c / (V_c + A_c + ε), ω_a_base = A_c / (V_c + A_c + ε) (ε = 1e-6, to avoid the denominator being 0). Furthermore, the matching degree of the fault types of the dual-modal submodule is judged, and the basic weights are adjusted: if V_type and A_type belong to the same fault category (e.g., "Indicator light abnormality" and "Fan abnormal noise" are both classified as "Equipment operation abnormality", based on the preset fault type mapping table), then the consistency coefficient α = 1.2; if V_type and A_type are not related (e.g., "Temperature exceeding the standard" and "Power supply discharge"), then α = 0.8; the adjusted weights are: ω_v_cons = ω_v_base × α, ω_a_cons = ω_a_base × α; Furthermore, the cosine similarity Sim(V_feat, A_feat) of the bimodal feature vectors is calculated to strengthen the weight of highly correlated features. Furthermore, the final fusion weights are ω_v = ω_v_cons × (Sim + 1) / 2, ω_a = ω_a_cons × (Sim + 1) / 2, (Sim∈[0,1], the reliability of the submodule results is verified by feature correlation, the higher Sim is, the more the weights are tilted towards the submodules with higher confidence). Furthermore, based on the final fusion weights, the enhanced bimodal feature vectors are weighted and fused to generate a fused feature vector F_feat: F_feat = ω_v × V_feat_enh + ω_a × A_feat_enh; Furthermore, based on the fault identification results of the dual-modal submodule, the fused feature vector is used for verification and correction, and the final fault information is output: (1) Fault type fusion: If V_c ≥ 0.9 and A_c ≥ 0.85, directly adopt the consistent fault type of the two modes (if both are identified as "fan fault", then the final type is "fan fault"). If only a single submodule has a confidence level ≥ 0.85 (e.g., V_c = 0.92, A_c = 0.7), then the fault type of that submodule is used as a candidate, and the fused feature vector is input into the fully connected layer for secondary judgment to confirm whether it is an associated fault type. If the confidence scores of both modalities are between 0.8 and 0.85, the most likely fault type is output based on the classification results of the fused feature vectors and the fault type mapping table.

[0058] (2) Fault confidence fusion: The final confidence score C_final = ω_v × V_c + ω_a × A_c + β × C_feat (β=0.3, C_feat is the classification confidence score of the fused feature vector). By fusing multiple confidence scores, the reliability of the results is improved, and the misjudgment of a single submodule is avoided from affecting the final output.

[0059] (3) Integration of fault development stage and early warning level: Based on A_stage output from the acoustic submodule, and combined with the severity of fault characteristics from the visual submodule (such as the degree of temperature exceeding the limit and the degree of abnormality in indicator light flashing frequency), the fault development stage is corrected; based on C_final and the corrected fault development stage, an early warning level (0-5) is generated, with the following level determination rules: Table 2 Fault Warning Level Table The final output includes the faulty device ID, the fused fault type (precise to the specific fault, such as "fan bearing wear"), the final confidence level (C_final≥0.8 for a valid diagnosis), the fault development stage, and the early warning level of the fault, providing complete input for subsequent fault classification and response.

[0060] Preferably, as shown in Table 3, a four-level fault classification standard can be formulated based on the fault's impact range, development speed, and severity to achieve refined classification: Table 3 Fault Level Table Preferably, an adaptive response strategy can be formulated based on the fault level to achieve "tiered handling and precise response," including both software and hardware responses: Level 1 Fault Response: Software Response: The system generates a fault log and pushes alarm information to the operation and maintenance management platform (SMS, APP notification), marked "Regular Troubleshooting"; Hardware Response: No automatic operation, maintain normal equipment operation.

[0061] Level 2 Fault Response: Software Response: Generate a detailed fault report (including dual-modal data and diagnostic process), push emergency alarms to maintenance personnel, and set a 72-hour countdown for troubleshooting; Hardware Response: Automatically adjust equipment operating parameters (such as reducing CPU load and starting backup fans) to mitigate the development of the fault.

[0062] Level 3 Fault Response: Software Response: Trigger audible and visual alarms (indicator lights flashing and alarm sounds at the operations and maintenance center), push real-time alarms to the operations and maintenance manager, and set a 2-hour emergency response countdown; Hardware Response: Automatically start data backup (for storage device failures), migrate the core business of the faulty device to redundant devices, and isolate the failure risk.

[0063] Level 4 Fault Response: Software Response: Activate the highest level alarm, link the data center emergency command platform, and notify all relevant operations and maintenance personnel to handle the situation on-site; Hardware Response: Automatically cut off the power to the faulty equipment (to prevent the fault from spreading), activate the backup equipment to take over the business, and ensure the continuity of core business.

[0064] Preferably, a distributed database (MySQL + MongoDB) is used to store the entire process data: MySQL stores structured data: fault classification results, response records, equipment information, and operation and maintenance logs; MongoDB stores unstructured data: raw visual images / videos, raw acoustic audio, preprocessed feature data, and intermediate AI diagnostic results; data management functions support data retrieval, historical fault tracing, export of AI model iteration training data (with de-identification processing), and data backup and recovery.

[0065] This invention, through the simultaneous acquisition of acoustic and visual data with distinct physical properties, leverages their complementary information to expand the dimensions of fault information capture from the source, laying the foundation for subsequent robust diagnosis. At the acoustic level, acoustic features are extracted; at the visual level, key areas such as target devices and indicator lights are precisely separated (purified) from the complex background of the server room (cables, cabinets, dust) to obtain visual features. This extraction of both acoustic and visual features avoids interference from irrelevant information in subsequent diagnostic identification. Furthermore, by combining a hybrid model of CNN and LSTM layers, acoustic fault diagnosis is performed based on acoustic features to improve acoustic accuracy. To improve the accuracy of fault diagnosis, this invention improves the YOLOv8 model by adaptively and pixel-level enhancing the weights of key visual features related to faults (such as high-brightness indicator lights and high-temperature areas) using the SimAM attention mechanism. It utilizes a fault precursor detection head to identify weak, non-obvious early fault features such as brightness changes, grayscale differences, and morphological deformation, thereby enhancing the accuracy of visual fault diagnosis. Consistency analysis is used for verification to avoid misjudgments from a single modality dominating the final result. Weight allocation is based on the consistency analysis results, integrating acoustic and visual diagnostic conclusions to output a more reliable and comprehensive final diagnosis. Compared to existing single-dimensional hardware device fault diagnosis methods, this invention integrates acoustic and visual data from data center hardware device status monitoring for early hardware fault diagnosis, thus improving the comprehensiveness and accuracy of fault diagnosis.

[0066] Example 2: like Figure 2 As shown, this embodiment provides a data center hardware fault diagnosis device, including a data acquisition module 201, a feature extraction module 202, a single-modal result acquisition module 203, and a final result acquisition module 204, wherein... Data acquisition module 201 is used to simultaneously acquire acoustic signal data and visual image data of target hardware devices in the data center; The feature extraction module 202 is used to extract the acoustic features of the target hardware device based on the acoustic signal data, and to perform target segmentation on the visual image data through a pre-trained Mask R-CNN model to obtain several target regions, so as to extract the visual features of the target hardware device from each of the target regions. In this embodiment, the feature extraction module 202 extracts the acoustic features of the target hardware device based on the acoustic signal data. Specifically, the feature extraction module 202 denoises the acoustic signal data using a preset wavelet threshold denoising algorithm; performs time-frequency analysis on the denoised acoustic signal data to extract features of different dimensions to generate several acoustic feature vectors; wherein the features of different dimensions include Mel frequency cepstral coefficients, power spectral density, and zero-crossing rate; and performs time-series splicing of the acoustic feature vectors through a preset sliding time window to obtain the acoustic features of the target hardware device.

[0067] In this embodiment, the feature extraction module 202 performs target segmentation on the visual image data using a pre-trained Mask R-CNN model to obtain several target regions. Specifically, the feature extraction module 202 extracts a first candidate region from the visual image data using a preset region proposal network in the Mask R-CNN model, based on a preset prior feature template. The RoI Align operation is then performed on the candidate region to uniformly map the first candidate region to a fixed-size feature map. Target classification and bounding box regression are then performed on the feature map to determine the region category and bounding box coordinates of the candidate region, resulting in a second candidate region. The region categories include device body, indicator light, temperature anomaly region, and background. For the second candidate region, pixel-level prediction is performed using a preset mask branch in the Mask R-CNN model to generate a binary mask representing the contour of the target region. Combining the bounding box coordinates and the binary mask, an independent target region is cropped from the second visual image data. The target region includes at least one of a device region sub-image, an indicator light sub-image, and a temperature anomaly sub-image.

[0068] The single-modal result acquisition module 203 is used to input the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results, and to input the visual features into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results; wherein, the hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network that introduces the SimAM attention mechanism and a fault precursor detection head for identifying non-obvious fault features; the non-obvious fault features include abnormal indicator light flickering, abnormal local temperature, and local deformation of equipment; In this embodiment, the single-modal result acquisition module 203 inputs the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results. Specifically, the single-modal result acquisition module 203 inputs the acoustic features into the hybrid model; through the CNN layer in the hybrid model, it extracts the spatial dimension information in the acoustic features to obtain acoustic spectrum difference results; through the LSTM layer in the hybrid model, it analyzes the temporal variation law in the acoustic features to obtain the acoustic evolution trend; combining the acoustic spectrum difference results and the acoustic evolution trend, it obtains and outputs acoustic fault diagnosis results; wherein, the acoustic fault diagnosis results include acoustic fault type, acoustic fault confidence, and fault development stage.

[0069] In this embodiment, the single-modal result acquisition module 203 inputs the visual features into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results. Specifically, the single-modal result acquisition module 203 inputs the visual features into the improved YOLOv8 model; through the backbone network that introduces the SimAM attention mechanism, the visual features are enhanced to obtain enhanced visual features; through the fault precursor detection head, non-obvious fault features in the enhanced visual features are identified to obtain non-obvious fault feature identification results; based on the non-obvious fault feature identification results, visual fault diagnosis results are obtained and output; wherein, the visual fault diagnosis results include visual fault type, visual fault confidence, and fault location coordinates.

[0070] The final result acquisition module 204 is used to perform consistency analysis on the acoustic fault diagnosis results and the visual fault diagnosis results, and to allocate fusion weights according to the consistency analysis results, so as to fuse the acoustic fault diagnosis results and the visual fault diagnosis results according to the allocated fusion weights to obtain the final fault diagnosis result of the target hardware device.

[0071] For a more detailed explanation of the working principle and procedures of this embodiment, please refer to the relevant description in Embodiment 1.

[0072] This invention utilizes a data acquisition module 201 to simultaneously acquire acoustic and visual data, two types of data with vastly different physical properties. By leveraging their complementary information, the dimensions of fault information capture are expanded from the source, laying the foundation for subsequent robust diagnosis. A feature extraction module 202 extracts acoustic features at the acoustic level and precisely separates (purifies) key areas such as target devices and indicator lights from the complex background of the computer room (cables, cabinets, dust) at the visual level to obtain visual features. This extraction of both acoustic and visual features avoids interference from irrelevant information in subsequent diagnostic identification. A single-modal result acquisition module 203 uses a hybrid model combining CNN and LSTM layers to perform acoustic analysis based on the acoustic features. To improve the accuracy of acoustic fault diagnosis, the YOLOv8 model is improved by using the SimAM attention mechanism to adaptively and pixel-level enhance the weights of key visual features related to faults (such as bright indicator lights and high-temperature areas). A fault precursor detection head is used to identify weak and non-obvious early fault features such as brightness changes, grayscale differences, and morphological deformation, thereby improving the accuracy of visual fault diagnosis. The final result acquisition module 204 uses consistency analysis for verification to avoid misjudgment from a single modality dominating the final result. Weights are allocated based on the consistency analysis results, and the acoustic and visual diagnostic conclusions are combined to output a more reliable and comprehensive final diagnostic result.

[0073] Example 3: This embodiment provides a terminal device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction that causes the processor to perform the operation of the data center hardware fault diagnosis method as described in any of the above.

[0074] Example 4: This invention provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device or apparatus containing the computer-readable storage medium to perform the data center hardware fault diagnosis method as described in any of the above embodiments.

[0075] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for diagnosing hardware faults in a data center, characterized in that, include: Simultaneously acquire acoustic signal data and visual image data of target hardware devices within the data center; Based on the acoustic signal data, the acoustic features of the target hardware device are extracted, and the visual image data is segmented using a pre-trained Mask R-CNN model to obtain several target regions, so as to extract the visual features of the target hardware device from each target region. The acoustic features are input into a pre-trained hybrid model to output acoustic fault diagnosis results, and the visual features are input into a pre-trained improved YOLOv8 model to output visual fault diagnosis results. The hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network incorporating a SimAM attention mechanism and a fault precursor detection head for identifying inconspicuous fault features. These inconspicuous fault features include abnormal indicator light flickering, abnormal local temperature, and localized equipment deformation. A consistency analysis is performed on the acoustic fault diagnosis results and the visual fault diagnosis results, and a fusion weight is assigned based on the consistency analysis results. The acoustic fault diagnosis results and the visual fault diagnosis results are then fused according to the assigned fusion weights to obtain the final fault diagnosis result of the target hardware device.

2. The data center hardware fault diagnosis method as described in claim 1, characterized in that, The visual features are input into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results, specifically: The visual features are input into the improved YOLOv8 model; The visual features are enhanced by using the backbone network that incorporates the SimAM attention mechanism to obtain the enhanced visual features. The fault precursor detection head identifies non-obvious fault features in the enhanced visual features, and the non-obvious fault feature identification result is obtained. Based on the non-obvious fault feature identification results, visual fault diagnosis results are obtained and output; wherein, the visual fault diagnosis results include visual fault type, visual fault confidence level, and fault location coordinates.

3. The data center hardware fault diagnosis method as described in claim 2, characterized in that, The visual features are enhanced using the backbone network incorporating the SimAM attention mechanism to obtain the enhanced visual features, specifically as follows: The enhanced visual features are input into the backbone network, and the backbone network outputs at least one intermediate feature map. For each pixel in the intermediate feature map, the local mean and local variance of each pixel are calculated based on the pixel value within the preset local window where each pixel is located, and the global mean of the intermediate feature map is calculated based on the pixel values ​​of all pixels in the intermediate feature map. Based on the local mean, local variance, and global mean, calculate the saliency score for each pixel, and generate the attention weight corresponding to each pixel based on the saliency score. The enhanced visual features are obtained by multiplying the original pixel value of each pixel in the intermediate feature map by the attention weight of the corresponding pixel.

4. A data center hardware fault diagnosis method as described in claim 3, characterized in that, The fault precursor detection head includes a brightness change enhancement branch, a grayscale difference enhancement branch, and a shape deformation enhancement branch; Specifically, the fault precursor detection head identifies non-obvious fault features in the enhanced visual features to obtain non-obvious fault feature identification results, as follows: Through the brightness change enhancement branch, the brightness change features in the enhanced visual features are extracted and enhanced using temporal difference convolution to obtain the first branch enhancement features; Through the gray-level difference enhancement branch, the gray-level difference features in the enhanced visual features are extracted and enhanced using adaptive threshold convolution to obtain the second branch enhancement features; Through the morphological deformation enhancement branch, edge-aware convolution is used to extract and enhance the morphological change features in the enhanced visual features to obtain the third branch enhancement features; The first branch enhancement features, the second branch enhancement features, and the third branch enhancement features are fused to obtain fused features. Based on the fused features, non-obvious fault features are identified to obtain non-obvious fault feature identification results.

5. A data center hardware fault diagnosis method as described in claim 1, characterized in that, Based on the acoustic signal data, the acoustic features of the target hardware device are extracted, specifically as follows: The acoustic signal data is denoised using a preset wavelet threshold denoising algorithm. Time-frequency analysis is performed on the denoised acoustic signal data to extract features of different dimensions to generate several acoustic feature vectors; wherein, the features of different dimensions include Mel frequency cepstral coefficients, power spectral density, and zero-crossing rate; The acoustic feature vectors are sequentially concatenated using a preset sliding time window to obtain the acoustic features of the target hardware device.

6. A data center hardware fault diagnosis method as described in claim 1, characterized in that, The pre-trained Mask R-CNN model is used to segment the visual image data to obtain several target regions, specifically: The first candidate region is extracted from the visual image data by using the preset region proposal network in the Mask R-CNN model based on the preset prior feature template; The RoI Align operation is performed on the candidate regions to uniformly map the first candidate regions to a feature map of a fixed size. The feature map is then subjected to target classification and bounding box regression to determine the region category and bounding box coordinates of the candidate regions, thereby obtaining the second candidate regions. The region categories include the device body, indicator lights, temperature anomaly areas, and background. For the second candidate region, pixel-level prediction is performed through the preset mask branch in the Mask R-CNN model to generate a binary mask representing the contour of the target region. By combining the bounding box coordinates with the binary mask, an independent target region is cropped from the second visual image data; wherein, the target region includes at least one of the device area sub-image, indicator light sub-image, and temperature anomaly sub-image.

7. A data center hardware fault diagnosis method as described in claim 1, characterized in that, The step of inputting the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results specifically involves: The acoustic features are input into the hybrid model; The spatial dimension information of the acoustic features is extracted through the CNN layer in the hybrid model to obtain the acoustic spectrum difference results. By analyzing the temporal variation patterns of the acoustic features through the LSTM layer of the hybrid model, the acoustic evolution trend can be obtained. By combining the acoustic spectrum difference results and the acoustic evolution trend, acoustic fault diagnosis results are obtained and output; wherein, the acoustic fault diagnosis results include acoustic fault type, acoustic fault confidence level and fault development stage.

8. A data center hardware fault diagnosis device, characterized in that, It includes a data acquisition module, a feature extraction module, a single-modal result acquisition module, and a final result acquisition module, among which, The data acquisition module is used to simultaneously acquire acoustic signal data and visual image data of target hardware devices within the data center; The feature extraction module is used to extract the acoustic features of the target hardware device based on the acoustic signal data, and to perform target segmentation on the visual image data through a pre-trained Mask R-CNN model to obtain several target regions, so as to extract the visual features of the target hardware device from each of the target regions. A single-modal result acquisition module is used to input the acoustic features into a pre-trained hybrid model so that the hybrid model outputs acoustic fault diagnosis results, and to input the visual features into a pre-trained improved YOLOv8 model so that the improved YOLOv8 model outputs visual fault diagnosis results; wherein, the hybrid model includes CNN layers and LSTM layers; the improved YOLOv8 model includes a backbone network incorporating a SimAM attention mechanism and a fault precursor detection head for identifying non-obvious fault features; the non-obvious fault features include abnormal indicator light flickering, abnormal local temperature, and local deformation of the equipment; The final result acquisition module is used to perform consistency analysis on the acoustic fault diagnosis results and the visual fault diagnosis results, and to allocate fusion weights based on the consistency analysis results, so as to fuse the acoustic fault diagnosis results and the visual fault diagnosis results according to the allocated fusion weights to obtain the final fault diagnosis result of the target hardware device.

9. A terminal device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the data center hardware fault diagnosis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device or apparatus containing the computer-readable storage medium to perform the data center hardware fault diagnosis method as described in any one of claims 1 to 7.

Images