Fault identification method, device, apparatus, medium and program product

Abstract

The application relates to a fault identification method, device, equipment, medium and program product. The method comprises the following steps: acquiring acoustic data of a to-be-tested device and environment data of an environment in which the to-be-tested device is located; extracting acoustic features from the acoustic data according to the environment data; and determining a target fault type of the to-be-tested device according to the acoustic features based on a fault diagnosis model. The above method improves the accuracy and stability of fault identification.

Description

Technical Field

[0001] This application relates to the field of fault identification technology, and in particular to a fault identification method, apparatus, equipment, medium, and program product. Background Technology

[0002] With the rapid development of fault identification technology, the demand for real-time and accurate detection of equipment operating status is increasing. Against this backdrop, acoustic recognition technology has emerged as an effective solution for various fault identification scenarios. It primarily works by acquiring the sound field signals of the equipment and reconstructing the spatial distribution of the sound sources to locate the point of anomaly and identify equipment malfunctions.

[0003] However, in traditional fault identification methods, acoustic recognition methods based on a single mode rely solely on the detection and recognition of acoustic signals. They have poor scene adaptability and are prone to significant fluctuations in detection accuracy due to changes in the scene, resulting in poor stability. Summary of the Invention

[0004] Therefore, it is necessary to provide a fault identification method, device, medium, and program product to address the aforementioned technical problems, thereby improving the accuracy and stability of fault identification.

[0005] Firstly, this application provides a fault identification method, including:

[0006] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0007] Based on environmental data, acoustic features are extracted from acoustic data;

[0008] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0009] In one embodiment, the acoustic features are extracted from the acoustic data based on the environmental data, including: determining the target environment type of the environment in which the device under test is located based on the environmental data; determining an acoustic feature extraction model that matches the target environment type based on a preset matching relationship; and extracting acoustic features from the acoustic data based on the acoustic feature extraction model.

[0010] In one embodiment, the target environment type of the environment in which the device under test is located is determined based on environmental data, including: determining a scene feature vector based on ambient humidity, peak frequency of noise spectrum and device type code of the device under test; and inputting the scene feature vector into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0011] In one embodiment, based on a fault diagnosis model, the target fault type of the device under test is determined according to acoustic features, including: fusing acoustic features, visual features and / or thermal imaging features of the device under test to obtain fused features; visual features are extracted from visual images of the device under test, and thermal imaging features are extracted from thermal imaging data of the device under test. Based on the fault diagnosis model, the target fault type of the device under test is determined according to the fused features.

[0012] In one embodiment, based on the fault diagnosis model, the target fault type of the device under test is determined according to the fusion features, including: determining a consistency score based on acoustic features and thermal imaging features; inputting the fusion features into the fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; determining a target confidence level based on the initial confidence level and the consistency score; and using the candidate fault type as the target fault type if the target confidence level is not less than a preset confidence threshold.

[0013] In one embodiment, determining a consistency score based on acoustic features and thermal imaging features includes: determining the location of a sound source based on acoustic features and determining the location of a thermal center based on thermal imaging features; determining the target distance between the sound source location and the thermal center location; and determining the consistency score based on the target distance.

[0014] Secondly, this application also provides a fault identification device, comprising:

[0015] The acquisition module is used to acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located.

[0016] The extraction module is used to extract acoustic features from acoustic data based on environmental data;

[0017] The determination module is used to determine the target fault type of the device under test based on the fault diagnosis model and acoustic characteristics.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0019] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0020] Based on environmental data, acoustic features are extracted from acoustic data;

[0021] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0022] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0023] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0024] Based on environmental data, acoustic features are extracted from acoustic data;

[0025] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0026] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0027] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0028] Based on environmental data, acoustic features are extracted from acoustic data;

[0029] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0030] The aforementioned fault identification methods, devices, equipment, media, and program products, by acquiring acoustic data of the device under test (DUT) and environmental data of the environment in which the DUT operates, can link the acoustic data during the DUT's operation with the specific environmental conditions of its surroundings, providing a more comprehensive data foundation for subsequent fault identification. By extracting acoustic features from the acoustic data based on environmental data, targeted acoustic feature extraction can be performed on the current environmental conditions, thereby suppressing the interference of environmental variables on the acoustic data and extracting acoustic features that more accurately reflect the DUT's own operating state. Based on a fault diagnosis model, the target fault type of the DUT is determined according to the acoustic features, thus realizing fault matching using acoustic features extracted from acoustic data using environmental data, which helps improve the accuracy and stability of target fault type identification. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a fault identification method in one embodiment;

[0033] Figure 2 This is a flowchart illustrating the steps for determining the target fault type in one embodiment;

[0034] Figure 3 This is a flowchart illustrating the fault identification method in another embodiment;

[0035] Figure 4 This is a structural block diagram of a fault identification device in one embodiment;

[0036] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] In one embodiment, such as Figure 1 As shown, a fault identification method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0039] S110. Acquire the acoustic data of the device under test and the environmental data of the environment in which the device under test is located.

[0040] Acoustic data can be understood as the audio data generated by the device under test (DUT) during operation. Environmental data is used to characterize the environmental conditions of the environment in which the DUT operates.

[0041] In an alternative embodiment, acoustic data of the device under test can be acquired using an audio acquisition device.

[0042] For example, the audio acquisition device may include a microphone array. Optionally, the microphone array may have 128 channels, a sampling rate of 48kHz, a sensitivity of -38dBV / Pa, cover a frequency band of 20Hz-5MHz, and have a circular array aperture of 0.5m. Optionally, the microphone array may include a MEMS (Micro-Electro-Mechanical Systems Sensor). Optionally, the microphone gain may be set to 40dB, and the high-pass filter may be set to 20Hz.

[0043] In an optional embodiment, the acoustic data may be preprocessed, and the preprocessing may include noise reduction processing.

[0044] For example, the acoustic data can be denoised using the following formula to obtain the denoised acoustic data. :

[0045]

[0046] Where X(k) is the spectrum of the noisy signal, that is, the acoustic data before noise reduction; N(k) is the background noise spectrum; α is the over-reduction factor, which can be, for example, 1.2.

[0047] In an optional embodiment, the environmental data may include at least one of ambient humidity and peak frequency of the noise spectrum.

[0048] For example, ambient humidity can be collected using a humidity sensor. The humidity sensor's accuracy can be ±2%RH.

[0049] For example, the peak frequency of the noise spectrum can be extracted from acoustic data, or the noise spectrum of the environment in which the device under test is located can be collected by environmental sensors, and the peak frequency of the noise spectrum can be extracted from the noise spectrum. The noise spectrum can be collected in the range of 20Hz-20kHz.

[0050] Optionally, the environmental data may also include temperature and air pressure data of the environment in which the device under test is located. Temperature data can be acquired using a temperature sensor with an accuracy of ±0.3℃. Air pressure data can be acquired using a air pressure sensor with an accuracy of ±0.1 kPa.

[0051] S120. Extract acoustic features from acoustic data based on environmental data.

[0052] In an optional embodiment, the target environment type of the environment in which the device under test is located can be determined based on environmental data; an acoustic feature extraction model that matches the target environment type can be determined based on a preset matching relationship; and acoustic features can be extracted from acoustic data based on the acoustic feature extraction model.

[0053] The target environment type can be understood as the environment type corresponding to the environment data.

[0054] The preset matching relationship can be understood as the pre-built matching relationship between different target environment types and different acoustic feature extraction models.

[0055] For example, the target environment type may include at least one of reverberation scene type, open scene type and extreme noise type. This application does not limit the specific classification method and specific classification content of the target environment type.

[0056] In one implementation, the acoustic feature extraction model matching the reverberation scene type can be a first feature extraction model based on improved delay superposition beamforming, which can be expressed as:

[0057] ;

[0058] in, The first acoustic feature extraction model extracts the acoustic features, i.e., the output sound field power spectrum (in dB), which represents the acoustic features extracted in the spatial angle. The power intensity of the sound source at that location; This indicates the azimuth angle, which ranges from 0° to 360° in the horizontal direction. The pitch angle is represented by -90° to 90° in the vertical direction; M represents the total number of microphones, i.e., the number of microphone array channels defined in the system hardware architecture, for example, 128; m represents the microphone channel index, which ranges from 1 to 128, corresponding to the m-th microphone in a 128-channel microphone array.

[0059] in, This represents the adaptive weight of the m-th microphone, with a value ranging from 0 to 1. It can be optimized using the minimum mean square error algorithm and is used to suppress reverberation interference and enhance the target sound source. This represents the preprocessed acoustic data collected by the m-th microphone at time t, i.e., the time domain acoustic signal. This indicates that the m-th microphone, relative to the reference microphone (array center), is in spatial angle... The direction of sound wave propagation delay; t represents the time variable, the unit is s, corresponding to the sampling time of the acoustic data, the sampling rate is 48kHz, that is, t=n / 48000, where n is the sampling point index.

[0060] For example, the sound wave propagation delay can be determined using the following formula. :

[0061] ;

[0062] in, represents the distance difference between the m-th microphone and the reference microphone; c represents the speed of sound.

[0063] In another implementation, the acoustic feature extraction model matching the open scene type can be a second feature extraction model based on the MUSIC (Multiple Signal Classification) algorithm, which can be expressed as:

[0064] ;

[0065] in, This represents the acoustic features extracted by the second acoustic feature extraction model, namely the MUSIC spatial spectrum (unitless, relative intensity values), with the spectral peaks corresponding to... This indicates the location of the fault sound source. In open scenes, the spectral peaks are sharper, resulting in higher positioning accuracy.

[0066] in, Represents the array manifold vector, and represents the spatial angle. The phase delay vector of the sound wave as it arrives at each microphone; This represents the conjugate transpose of the array manifold vector, used for inner product calculations in matrix operations; Represents the noise subspace matrix; This represents the conjugate transpose of the noise subspace matrix.

[0067] For example, the vector elements of an array manifold vector can be represented as:

[0068] ;

[0069] Where j represents the imaginary unit; f represents the signal frequency; This indicates the time delay of sound wave propagation.

[0070] For example, the noise subspace matrix The covariance matrix of acoustic data can be used to... Eigenvalue decomposition is performed, and the eigenvectors corresponding to the smallest eigenvalues ​​are used to construct a representation of the spatial distribution characteristics of the background noise. The smallest eigenvalues ​​can include the minimum eigenvalue and eigenvalues ​​smaller than a preset value. The covariance matrix is... It can be represented as:

[0071] ;

[0072] in, This represents the signal vector corresponding to the preprocessed acoustic data.

[0073] For example, The dimension can be: 1*128; The dimension can be 128*K, where K represents the number of noise feature values; The dimension can be 128*1; The dimension can be K*128.

[0074] In another implementation, the acoustic feature extraction model that matches the type of extreme noise can be a third feature extraction model of an attention-based neural network. This third feature extraction model, through attention enhancement, can effectively reduce feature bias in the extracted features caused by extreme noise.

[0075] In an optional embodiment, a scene feature vector can be determined based on ambient humidity, peak frequency of the noise spectrum, and device type encoding of the device under test; the scene feature vector is then input into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0076] For example, a scene feature vector can be represented as:

[0077]

[0078] Where F represents the scene feature vector; f peak H represents the peak frequency of the noise spectrum; D represents the ambient humidity; and onehot represents the unique hot coding.

[0079] For example, the scene classification model can be a traditional machine learning model or a neural network model; this application does not limit the specific model type of the scene classification model. Optionally, the scene classification model can be a random forest classifier. The classifier parameters of the random forest classifier can be set by a technician according to needs or experience, or determined through a large number of experiments; this application does not limit this. For example, the number of decision trees in the classifier parameters can be 50, and the maximum depth can be 10.

[0080] S130. Based on the fault diagnosis model, determine the target fault type of the device under test according to the acoustic characteristics.

[0081] In an optional embodiment, acoustic features can be input into a fault diagnosis model to determine the target fault type of the device under test.

[0082] In another optional embodiment, acoustic features can be fused with the visual features and / or thermal imaging features of the device under test (DUT) to obtain fused features; visual features are extracted from visual images of the DUT, and thermal imaging features are extracted from thermal imaging data of the DUT; based on the fault diagnosis model, the target fault type of the DUT is determined according to the fused features.

[0083] Among them, visual images of the device under test can be acquired through a thermal imaging module; visual images of the device under test can be acquired through a visual camera.

[0084] For example, the thermal imaging module can have a resolution of 640*512; a temperature measurement range of -20℃ to 150℃; an acquisition accuracy of ±0.5℃; a frame rate of 30fps; and a thermal imaging emissivity of 0.95 with atmospheric transmittance compensation enabled. For example, the visual camera can be a 20-megapixel camera with a focal length of 8mm. For example, the visual camera, thermal imaging module, and audio acquisition device can be mechanically fixed together to ensure a spatial deviation of no more than 5mm.

[0085] Optionally, after acquiring visual images, thermal imaging data, and acoustic data, spatial coordinate mapping can be performed on the visual images, thermal imaging data, and acoustic data to achieve spatial calibration. The coordinate transformation can be performed using the following formula:

[0086]

[0087] Where R is a 3*3 rotation matrix and T is a 3*1 translation vector, the spatial mapping error after calibration is no greater than 0.1m.

[0088] Optionally, the fused features can be input into the fault diagnosis model to determine the target fault type of the device under test.

[0089] Optionally, the fused features can be input into the fault diagnosis model to determine the candidate diagnostic results of the device under test; based on the consistency scores between the different dimensional features corresponding to the device under test, the candidate diagnostic results are optimized to obtain the final diagnostic results, which are used to characterize the target fault type of the device under test.

[0090] The aforementioned fault identification method, by acquiring acoustic data of the device under test (DUT) and environmental data of its environment, can link the acoustic data during DUT operation with the specific environmental conditions, providing a more comprehensive data foundation for subsequent fault identification. By extracting acoustic features from the acoustic data based on environmental data, targeted acoustic feature extraction can be performed for current environmental conditions, thereby suppressing the interference of environmental variables on the acoustic data and extracting acoustic features that more accurately reflect the operating state of the DUT. Based on a fault diagnosis model, the target fault type of the DUT is determined according to the acoustic features, thus realizing fault matching using acoustic features extracted from acoustic data using environmental data, which helps improve the accuracy and stability of target fault type identification.

[0091] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment in which the target fault type is further refined.

[0092] refer to Figure 2The steps for determining the target fault type shown include:

[0093] S210. Perform feature fusion on the acoustic features, as well as the visual features and / or thermal imaging features of the device under test, to obtain fused features; the visual features are extracted from the visual images of the device under test, and the thermal imaging features are extracted from the thermal imaging data of the device under test.

[0094] In an optional embodiment, acoustic features may be processed before feature fusion to improve the accuracy and efficiency of subsequent recognition.

[0095] Optionally, the extracted acoustic features can be mapped to a pixel matrix to obtain a sound field heatmap; the global mean and global standard deviation of the sound field heatmap can be determined; and candidate sound source features can be determined based on the global mean and global standard deviation. The sound field heatmap can be understood as the sound source power intensity corresponding to each pixel (x, y), in dB; the global mean is the average power of each pixel.

[0096] For example, candidate sound source features A(x,y) can be determined according to the following formula:

[0097] A(x,y)>mean(A)+3*std(A);

[0098] Where A represents the sound field heatmap; mean(A) represents the global mean of the sound field heatmap, used to distinguish between background noise and potential sound sources; std(A) represents the global standard deviation of the sound field heatmap, used to filter out weak noise interference; “mean(A)+3*std(A)” can be understood as the threshold that significantly deviates from the background noise.

[0099] Among them, the sound field heat map can be a 256*256 pixel sound field heat map. The larger the pixel value, the more prominent the sound source.

[0100] For example, candidate sound source features can be used as acoustic features after feature processing, or the original acoustic features and candidate sound source features can be used as acoustic features after feature processing.

[0101] The following examples illustrate the extraction processes for thermal imaging features and visual features, respectively. It should be noted that this should not be construed as a limitation on the specific extraction process.

[0102] In an optional embodiment, bilinear interpolation can be performed on isolated bad pixels in the thermal imaging data to normalize the thermal imaging data; valid thermal imaging data can be determined from the normalized thermal imaging data; and thermal imaging features can be extracted from the valid thermal imaging data.

[0103] Specifically, if the temperature difference between the target point and its adjacent locations is not less than a preset temperature value, the target point can be determined as an isolated defective point. The preset temperature value can be set by technicians according to their needs or experience, or determined through a large number of experiments; this application does not impose any limitations on it. For example, the preset temperature value can be 5°C.

[0104] For example, bilinear interpolation can be performed using the following formula:

[0105]

[0106] The thermal imaging data after bad pixel processing includes the original temperature T(x,y) of different locations (x,y).

[0107] For example, the original temperature T(x,y) can be normalized using the following formula to obtain the normalized thermal imaging data T. norm (x,y):

[0108] ;

[0109] Where T represents the original temperature; This indicates the lowest temperature value in a single frame of thermal imaging, for example, the lowest temperature on the surface of the device in a certain frame of thermal imaging is 15°C; This indicates the highest temperature value in a single frame of thermal imaging. For example, the highest temperature at the fault point in a certain frame of thermal imaging is 85℃.

[0110] For example, candidate thermal imaging data with normalized temperatures greater than a preset normalization threshold can be selected from normalized thermal imaging data. Continuous regions between candidate thermal imaging data are preserved through neighborhood connectivity analysis to obtain effective thermal imaging data. The preset temperature threshold can be set by a technician based on needs or experience, or determined through extensive experimentation; this application does not impose any limitations on it. For example, the preset normalization threshold can be 0.85. The candidate thermal imaging data Rthermal can be expressed as:

[0111]

[0112] For example, the center coordinates (x, y) of the effective thermal imaging data can be determined. T ,y T ) and area S T Based on a neural network model, according to the center coordinates (x T ,y T ) and area S T Extracting thermal imaging features, i.e., thermal feature vector F, from effective thermal imaging data. T Among them, the thermal eigenvector F T The vector dimension can be 1024*1.

[0113] In an optional embodiment, edge detection can be performed on the visual image based on the Canny algorithm to obtain a visual feature map V; the visual feature map V is then cropped to retain the device region corresponding to the device under test, resulting in a cropped edge feature map; and features are extracted from the cropped edge feature map to obtain a visual feature vector F. v Among them, the visual feature vector F v The vector dimension can be 512*1.

[0114] For example, the visual feature map V can be input into a visual processing model to identify the target box corresponding to the device region, and then cropped using the target box as the boundary to retain the device region, resulting in a cropped edge feature map. The target box can be identified based on the YOLOv8 model. The visual feature map V is used to characterize the edge intensity of pixel (x,y), with the edge intensity value ranging from 0 to 255.

[0115] Understandably, based on the aforementioned processing, the acoustic feature F can be obtained. A Thermal imaging features F T and visual features F v Correspondingly, the fusion feature F can be determined using the following formula. 融合 :

[0116]

[0117] Among them, F 融合 This represents the fused features, i.e., the fused feature vector, which can have a dimension of 2048*1; F A The acoustic features are represented by an acoustic feature vector, which can have a dimension of 2048*1, w A F represents the acoustic feature weights; T This represents thermal imaging features, i.e., thermal imaging feature vectors, which can have dimensions of 1024*1, w T F represents the feature weights of thermal imaging; v This represents visual features, i.e., visual feature vectors, with dimensions of 512*1, w v This represents the weight of visual features.

[0118] For example, acoustic feature weights w A The value of is in the range of 0-1 and can be learned through a multi-layer perceptron (MLP). Optionally, the acoustic feature weights w can be determined based on the target environment type. A Specifically, when the target environment type is a reverberant scene or an open scene, the acoustic feature weights w can be determined. AThe value range is 0.6 to 0.7; when the target environment type is extreme noise type, the acoustic feature weight w can be determined. A The value range is 0.3-0.4.

[0119] Optional, acoustic feature weights w A It can be determined using the following formula:

[0120]

[0121] The Multilayer Perceptron (MLP) can be understood as a neural network module containing two hidden layers (1024 and 512 nodes respectively) and using ReLU activation function. By using F... A and F T The acoustic features are concatenated and fused into a 3072*1 vector to correlate acoustic features with thermal imaging features. The concatenated vector is then input into a multilayer perceptron (MLP) to obtain predicted weights. After normalization and softmax, the acoustic feature weights w are obtained. A .

[0122] For example, thermal imaging feature weights w T The value range is 0-1, and w A Complementary, i.e., w T =1-w A Based on a complementary approach, thermal imaging features are made more reliable under extreme noise conditions.

[0123] For example, the visual feature weights can use preset weight values, such as 0.2, to balance multimodal features and achieve spatial calibration.

[0124] S220. Based on the fault diagnosis model, determine the target fault type of the device under test according to the fusion characteristics.

[0125] The target fault type can include normal types and abnormal types. Abnormal types can include at least one of surge arrester discharge types and insulator crack types, and this application does not limit the specific classification of the target fault type.

[0126] In an optional embodiment, a consistency score can be determined based on acoustic features and thermal imaging features; the fused features are input into the fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; a target confidence level is determined based on the initial confidence level and the consistency score; and if the target confidence level is not less than a preset confidence threshold, the candidate fault type is used as the target fault type.

[0127] The preset reliability threshold can be set by technicians according to their needs or experience, or determined through a large number of experiments; this application does not impose any limitations on it. For example, the preset reliability threshold can be 0.85.

[0128] For example, the product of the initial confidence level and the consistency score can be used as the target confidence level. The target fault types for power equipment may include at least one of the following: surge arrester discharge, insulator crack, transformer winding short circuit, and cable joint overheating; the target fault types for mechanical equipment may include at least one of the following: motor bearing wear, abnormal gear meshing, pump body cavitation, and conveyor belt misalignment.

[0129] Optionally, the location of the sound source can be determined based on acoustic characteristics, and the location of the thermal center can be determined based on thermal imaging characteristics; the target distance between the sound source location and the thermal center location can be determined; and a consistency score can be determined based on the target distance.

[0130] For example, the target distance d can be determined according to the following formula:

[0131]

[0132] Where d represents the target distance; (x A ,y A (x) indicates the location of the sound source; T ,y T ) indicates the location of the thermal center.

[0133] For example, if the target distance is not greater than a preset distance threshold, a consistency score is determined as the base score; if the target distance is greater than the preset distance threshold, the difference between the target distance and the preset distance threshold is determined as the target ratio of the difference between the base score and the target distance threshold; and a consistency score is determined based on the base score and the target ratio.

[0134] The consistency score Q can be determined using the following formula:

[0135] ;

[0136] Where Q represents the consistency score; d represents the target distance; and D represents the preset distance threshold.

[0137] For example, if the target distance is not greater than a preset distance threshold and the target confidence is not less than a preset confidence threshold, the candidate fault type can be used as the target fault type.

[0138] In the above steps, feature fusion of acoustic, visual, and thermal imaging features integrates multidimensional information reflecting the mechanical state, appearance, and temperature distribution of equipment into a unified fused feature representation. By determining a consistency score based on acoustic and thermal imaging features, the correlation between the sound source localization result and the abnormal heating area in physical space is independently verified, thereby achieving a quantitative assessment of the inherent consistency of the multimodal perception results. By inputting the fused features into the fault diagnosis model, candidate fault types and their initial confidence levels are obtained. Then, a target confidence level is determined based on the initial confidence level and the consistency score. If the target confidence level is not less than a preset confidence threshold, the candidate fault type is used as the target fault type. This fusion and correction of data-driven model prediction results with physical rule-based verification results improves the accuracy of the output results.

[0139] Based on the above embodiments, this application also provides an optional embodiment in which the training process of the fault diagnosis model is described in detail.

[0140] In an optional embodiment, master and slave devices can be associated and joint training samples can be obtained. Based on the training samples and corresponding fault labels, an initial fault diagnosis model is trained and its model weights are updated to obtain a trained fault diagnosis model. The fault diagnosis model can output the target fault type. Optionally, the fault diagnosis model can also output at least one of the following: acoustic field heatmap, temperature curve, fault coordinates, confidence level corresponding to the target fault type, and maintenance suggestions. The master and slave devices can include a master device and a slave device.

[0141] Optionally, the fault diagnosis model can be updated based on incremental learning. This involves weighted summation of the model weights for the old samples and the new samples to obtain the updated model weights. The model weight for the old samples can be 0.3, and the model weight for the new samples can be 0.7. This application does not impose any limitations on the specific weight allocation method.

[0142] The following is an exemplary description of the training process for a fault diagnosis model.

[0143] For example, the spatial coordinates and / or signal feature vectors of the master and slave devices are used as clustering samples; clustering parameters are set, and the clustering samples are divided into two clusters by minimizing the objective function; the cross-correlation value between the acoustic data of the master device and the acoustic data of the slave device is determined; the acoustic data of the slave device is corrected according to the cross-correlation value to synchronize it with the signal of the master device; the optimized multimodal fusion weights are determined by minimizing the squared difference error between the fused features and the true labels, and the multimodal fusion weights are used as model parameters for fault diagnosis.

[0144] The spatial coordinates may include the installation location of the microphone array or sensor, and the signal feature vector may include the spectral characteristics of the acoustic signal.

[0145] For example, clustering parameters can be set, and the objective function can be minimized using the K-means algorithm to divide the clustered samples into two clusters. The objective function can be expressed as:

[0146]

[0147] Among them, C i Represents the i-th cluster, i=1,2; x represents the spatial coordinates and / or signal feature vectors of the master and slave devices, i.e., clustering samples; u i The i-th cluster is represented by the reference coordinates / features of the master and slave devices. The clustering parameter K can be set to 2, which means that the devices are fixedly divided into master devices and slave devices. The device corresponding to the cluster center u1 is the master device and serves as the reference for signal acquisition, while the device corresponding to the cluster center u2 is the slave device.

[0148] For example, the cross-correlation value can be expressed as:

[0149]

[0150] in, Indicates the acoustic signal of the main device; This indicates the acoustic signal from the device.

[0151] Among them, the one that maximizes the cross-correlation value can be determined. As a correction delay ; the acoustic signal from the device Adjusted to .

[0152] For example, the squared error between the fused features and the true label can be expressed as:

[0153]

[0154] in, This represents the optimized multimodal feature weights; Y represents the fusion feature with fusion weight w as a parameter; 真实 This represents the true fault label, that is, the feature vector or classification label corresponding to the fault type of the sample.

[0155] For example, the optimized multimodal feature weights It can be determined based on the least squares method and according to the following formula:

[0156]

[0157] Where X is the feature matrix composed of acoustic features, visual features and thermal imaging features corresponding to the training samples.

[0158] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment in which the fault identification method is described in detail.

[0159] refer to Figure 3 The diagram shown is a flowchart of a fault identification method in another embodiment, including:

[0160] S301. Acquire the acoustic data of the device under test and the environmental data of the environment in which the device under test is located. The environmental data includes the ambient humidity and the peak frequency of the noise spectrum.

[0161] S302. Determine the scene feature vector based on the ambient humidity, peak frequency of the noise spectrum, and the device type code of the device under test.

[0162] S303. Input the scene feature vector into the scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0163] S304. Based on the preset matching relationship, determine the acoustic feature extraction model that matches the target environment type.

[0164] S305. Acquire visual images of the device under test and extract visual features from the visual images; acquire thermal imaging data of the device under test and extract thermal imaging features from the thermal imaging data.

[0165] S306. Determine the location of the sound source based on acoustic characteristics, and determine the location of the thermal center based on thermal imaging characteristics.

[0166] S307. Determine the consistency score based on the target distance between the sound source location and the thermal center location.

[0167] S308. Perform feature fusion on acoustic features, visual features and thermal imaging features to obtain fused features.

[0168] S309. Input the fused features into the fault diagnosis model to obtain the candidate fault type and the initial confidence level of the candidate fault type.

[0169] S310. Determine the target confidence level based on the initial confidence level and consistency score; if the target confidence level is not less than the preset confidence threshold, the candidate fault type shall be used as the target fault type.

[0170] S305 can be executed synchronously with S301, or it can be executed before or after S301. No restrictions are imposed on this unless otherwise specified in the application.

[0171] Based on the above embodiments, a verification embodiment is provided. In this verification embodiment, the fault identification method is executed by a terminal device, which can be an industrial-grade edge computing terminal with a quad-core processor, 16GB of memory, and support for 5G / Ethernet communication. Simultaneously, GPS (Global Positioning System) timing is performed via a time synchronization module, with a UTC (Coordinated Universal Time) synchronization error not exceeding 100ns. A hardware-triggered 10us pulse ensures that the timestamp deviation of multi-source data is not greater than 1ms. To achieve synchronized data acquisition, a visual frame signal is used as a reference, binding one frame of visual image, one frame of thermal imaging image, and 1024 points of acoustic data, along with environmental parameter tags.

[0172] Based on the above fault identification method, 200 defect samples were tested in a 10kV power line inspection scenario, and the following test results were obtained: the average positioning error was 0.15m, the maximum positioning error was 0.2m; the fault identification accuracy was 96.3%, which is 35.1% higher than the traditional single acoustic detection method; the response time was 1.8s, which can meet the real-time requirements.

[0173] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0174] Based on the same inventive concept, this application also provides a fault identification device for implementing the fault identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more fault identification device embodiments provided below can be found in the limitations of the fault identification method described above, and will not be repeated here.

[0175] In one exemplary embodiment, such as Figure 4 As shown, a fault identification device is provided, comprising: an acquisition module 410, an extraction module 420, and a determination module 430, wherein:

[0176] The acquisition module 410 is used to acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located.

[0177] Extraction module 420 is used to extract acoustic features from acoustic data based on environmental data;

[0178] The determination module 430 is used to determine the target fault type of the device under test based on the fault diagnosis model and acoustic characteristics.

[0179] In one embodiment, the extraction module 420 includes: a first determining unit, configured to determine the target environment type of the environment in which the device under test is located based on environmental data; a second determining unit, configured to determine an acoustic feature extraction model that matches the target environment type based on a preset matching relationship; and a first extraction unit, configured to extract acoustic features from acoustic data based on the acoustic feature extraction model.

[0180] In one embodiment, the first determining unit includes: a first determining subunit, configured to determine a scene feature vector based on ambient humidity, peak frequency of noise spectrum and device type encoding of the device under test; and a first input subunit, configured to input the scene feature vector into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0181] In one embodiment, the determining module 430 includes: a fusion unit, configured to perform feature fusion on acoustic features, visual features of the device under test, and / or thermal imaging features of the device under test to obtain fused features; visual features are extracted from the visual image of the device under test, and thermal imaging features are extracted from the thermal imaging data of the device under test; and a third determining unit, configured to determine the target fault type of the device under test based on the fused features according to a fault diagnosis model.

[0182] In one embodiment, the third determining unit includes: a second determining subunit, used to determine a consistency score based on acoustic features and thermal imaging features; a second input subunit, used to input fused features into a fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; a third determining subunit, used to determine a target confidence level based on the initial confidence level and the consistency score; and a fourth determining subunit, used to use the candidate fault type as the target fault type if the target confidence level is not less than a preset confidence threshold.

[0183] In one embodiment, the second determining subunit is specifically used to: determine the location of the sound source based on acoustic features, and determine the location of the thermal center based on thermal imaging features; determine the target distance between the sound source location and the thermal center location; and determine a consistency score based on the target distance.

[0184] Each module in the aforementioned fault identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0185] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a fault identification method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0186] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0188] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0189] Based on environmental data, acoustic features are extracted from acoustic data;

[0190] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0191] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the target environment type of the environment in which the device under test is located based on environmental data; determining an acoustic feature extraction model that matches the target environment type based on a preset matching relationship; and extracting acoustic features from the acoustic data based on the acoustic feature extraction model.

[0192] In one embodiment, when the processor executes the computer program, it also performs the following steps: determining a scene feature vector based on ambient humidity, peak frequency of the noise spectrum, and device type encoding of the device under test; and inputting the scene feature vector into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0193] In one embodiment, when the processor executes the computer program, it further performs the following steps: performing feature fusion on acoustic features, visual features of the device under test, and / or thermal imaging features of the device under test to obtain fused features; extracting visual features from visual images of the device under test, and extracting thermal imaging features from thermal imaging data of the device under test; and determining the target fault type of the device under test based on the fused features according to a fault diagnosis model.

[0194] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a consistency score based on acoustic features and thermal imaging features; inputting the fused features into a fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; determining a target confidence level based on the initial confidence level and the consistency score; and using the candidate fault type as the target fault type if the target confidence level is not less than a preset confidence threshold.

[0195] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the location of a sound source based on acoustic features, and determining the location of a thermal center based on thermal imaging features; determining a target distance between the sound source location and the thermal center location; and determining a consistency score based on the target distance.

[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0197] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0198] Based on environmental data, acoustic features are extracted from acoustic data;

[0199] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0200] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the target environment type of the environment in which the device under test is located based on environmental data; determining an acoustic feature extraction model that matches the target environment type based on a preset matching relationship; and extracting acoustic features from the acoustic data based on the acoustic feature extraction model.

[0201] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a scene feature vector based on ambient humidity, peak frequency of the noise spectrum, and device type encoding of the device under test; and inputting the scene feature vector into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0202] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: performing feature fusion on acoustic features, visual features of the device under test, and / or thermal imaging features of the device under test to obtain fused features; extracting visual features from visual images of the device under test, and extracting thermal imaging features from thermal imaging data of the device under test; and determining the target fault type of the device under test based on the fused features according to a fault diagnosis model.

[0203] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a consistency score based on acoustic features and thermal imaging features; inputting the fused features into the fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; determining a target confidence level based on the initial confidence level and the consistency score; and using the candidate fault type as the target fault type if the target confidence level is not less than a preset confidence threshold.

[0204] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the location of a sound source based on acoustic features, and determining the location of a thermal center based on thermal imaging features; determining a target distance between the sound source location and the thermal center location; and determining a consistency score based on the target distance.

[0205] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0206] Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located;

[0207] Based on environmental data, acoustic features are extracted from acoustic data;

[0208] Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

[0209] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the target environment type of the environment in which the device under test is located based on environmental data; determining an acoustic feature extraction model that matches the target environment type based on a preset matching relationship; and extracting acoustic features from the acoustic data based on the acoustic feature extraction model.

[0210] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a scene feature vector based on ambient humidity, peak frequency of the noise spectrum, and device type encoding of the device under test; and inputting the scene feature vector into a scene classification model to obtain the target environment type of the environment in which the device under test is located.

[0211] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: performing feature fusion on acoustic features, visual features of the device under test, and / or thermal imaging features of the device under test to obtain fused features; extracting visual features from visual images of the device under test, and extracting thermal imaging features from thermal imaging data of the device under test; and determining the target fault type of the device under test based on the fused features according to a fault diagnosis model.

[0212] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a consistency score based on acoustic features and thermal imaging features; inputting the fused features into the fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; determining a target confidence level based on the initial confidence level and the consistency score; and using the candidate fault type as the target fault type if the target confidence level is not less than a preset confidence threshold.

[0213] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the location of a sound source based on acoustic features, and determining the location of a thermal center based on thermal imaging features; determining a target distance between the sound source location and the thermal center location; and determining a consistency score based on the target distance.

[0214] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0215] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0216] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fault identification method, characterized in that, The method includes: Acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located; Based on the environmental data, acoustic features are extracted from the acoustic data; Based on the fault diagnosis model, the target fault type of the device under test is determined according to the acoustic characteristics.

2. The method according to claim 1, characterized in that, The step of extracting acoustic features from the acoustic data based on the environmental data includes: Based on the environmental data, the target environment type of the environment in which the device under test is located is determined; Based on the preset matching relationship, determine the acoustic feature extraction model that matches the target environment type; Based on the acoustic feature extraction model, acoustic features are extracted from the acoustic data.

3. The method according to claim 2, characterized in that, The step of determining the target environment type of the environment in which the device under test is located based on the environmental data includes: The scene feature vector is determined based on the ambient humidity, the peak frequency of the noise spectrum, and the device type code of the device under test; The scene feature vector is input into the scene classification model to obtain the target environment type of the environment in which the device under test is located.

4. The method according to any one of claims 1-3, characterized in that, The method of determining the target fault type of the device under test based on the acoustic characteristics using the fault diagnosis model includes: The acoustic features, visual features and / or thermal imaging features of the device under test are fused to obtain fused features; the visual features are extracted from the visual images of the device under test, and the thermal imaging features are extracted from the thermal imaging data of the device under test. Based on the fault diagnosis model, the target fault type of the device under test is determined according to the fusion features.

5. The method according to claim 4, characterized in that, The determination of the target fault type of the device under test based on the fault diagnosis model and the fused features includes: A consistency score is determined based on the acoustic features and the thermal imaging features; The fused features are input into the fault diagnosis model to obtain candidate fault types and initial confidence levels of the candidate fault types; The target confidence level is determined based on the initial confidence level and the consistency score. If the target confidence level is not less than a preset confidence threshold, the candidate fault type is taken as the target fault type.

6. The method according to claim 5, characterized in that, The determination of the consistency score based on the acoustic features and the thermal imaging features includes: Based on the acoustic characteristics, the location of the sound source is determined, and based on the thermal imaging characteristics, the location of the thermal center is determined. Determine the target distance between the sound source location and the thermal center location; The consistency score is determined based on the target distance.

7. A fault identification device, characterized in that, The device includes: The acquisition module is used to acquire acoustic data of the device under test and environmental data of the environment in which the device under test is located; An extraction module is used to extract acoustic features from the acoustic data based on the environmental data; The determination module is used to determine the target fault type of the device under test based on the acoustic characteristics and the fault diagnosis model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images