Power equipment state monitoring and fault diagnosis system based on multi-modal data

Abstract

The invention provides a power equipment state monitoring and fault diagnosis system based on multi-modal data, and relates to the field of data processing, and the system comprises a data collection module which comprises an acoustic array, an image collection device and an infrared thermal induction device, and the acoustic array comprises sound collection devices arranged at a plurality of positions of power equipment; the state monitoring module is used for determining abnormal sound coordinates according to the sound data acquired by the acoustic array; the data acquisition module is further used for controlling the image acquisition device to acquire images based on the abnormal sound coordinates and controlling the infrared thermal induction device to acquire infrared thermal induction data; and the fault diagnosis module is used for performing fault diagnosis based on the sound data acquired by the acoustic array, the image acquired by the image acquisition device and the infrared thermal induction data acquired by the infrared thermal induction device, and has the advantage of improving the efficiency and accuracy of state monitoring and fault diagnosis of the power equipment.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a power equipment condition monitoring and fault diagnosis system based on multimodal data. Background Technology

[0002] As power systems develop towards higher voltage, larger capacity, and greater intelligence, the operational reliability of power equipment (such as transformers, generators, and high-voltage circuit breakers) is directly related to grid security and the stability of energy supply.

[0003] Current power equipment condition monitoring mainly relies on single-type sensor data or simple rule-based diagnosis based on thresholds, which has the following fundamental limitations: Single-modal data can only reflect the local state of equipment and cannot capture the multi-dimensional characteristics of faults. For example, vibration sensors can detect rotor imbalance or bearing wear, but cannot identify partial discharge caused by stator winding insulation deterioration; infrared thermal imaging can locate overheated areas of equipment, but cannot distinguish whether the temperature rise is caused by excessive load or internal fault. Single-modal data lacks correlation analysis, which can easily lead to misdiagnosis of faults. Threshold methods can only identify faults that have reached a dangerous level, but cannot capture the weak characteristics in the early stages of a fault. For example, the pulse amplitude in the early stages of partial discharge may be lower than the threshold, but the insulation has already begun to deteriorate at this time. By the time the fault is detected to be exceeding the limit, it has already entered a rapid development phase, and the maintenance window is greatly shortened.

[0004] Complex faults are often caused by the coupling of multiple factors (such as mechanical vibration leading to insulation wear, which in turn causes partial discharge), and threshold methods are difficult to use to analyze the correlation of changes in multiple parameters. For example, a drop in pressure in the operating mechanism of a high-voltage circuit breaker and an abnormal current in the opening and closing coils may occur simultaneously, but traditional methods cannot determine the causal relationship between the two, making it difficult to locate the root cause of the fault.

[0005] Therefore, there is a need to provide a power equipment condition monitoring and fault diagnosis system based on multimodal data to improve the efficiency and accuracy of power equipment condition monitoring and fault diagnosis. Summary of the Invention

[0006] This invention provides a power equipment condition monitoring and fault diagnosis system based on multimodal data, comprising: a data acquisition module, including an acoustic array, an image acquisition device, and an infrared thermal sensing device, wherein the acoustic array includes sound acquisition devices disposed at multiple locations on the power equipment; a condition monitoring module, used to determine the coordinates of abnormal noises based on the sound data acquired by the acoustic array; the data acquisition module is also used to control the image acquisition device to acquire images and control the infrared thermal sensing device to acquire infrared thermal sensing data based on the coordinates of the abnormal noises; and a fault diagnosis module, used to perform fault diagnosis based on the sound data acquired by the acoustic array, the images acquired by the image acquisition device, and the infrared thermal sensing data acquired by the infrared thermal sensing device.

[0007] Furthermore, the installation of the acoustic array includes: identifying multiple fault types and multiple initial detection locations for the power equipment; for each fault type, collecting abnormal noise data corresponding to the fault type, wherein the abnormal noise data includes sound data for each initial detection location; for any two detection locations, calculating the abnormal noise similarity between the two initial detection locations based on the sound data for each fault type corresponding to the two initial detection locations; based on the abnormal noise similarity between any two initial detection locations, eliminating redundant initial detection locations and determining valid detection locations; and using an adaptive search butterfly optimization algorithm, determining multiple installation locations for the acoustic array based on the sound data for each fault type corresponding to the valid detection locations, and then installing it.

[0008] Furthermore, using an adaptive search butterfly optimization algorithm, multiple installation locations for the acoustic array are determined based on the sound data corresponding to each fault type at the effective detection locations. This includes: S11, for each effective detection location, calculating the sampling probability of the effective detection location based on the sound data corresponding to each fault type; S12, initializing the population based on the sampling probability of each effective detection location, where each butterfly in the population represents a combination of effective detection locations; S13, calculating the fragrance intensity of each butterfly in the population; S14, calculating the global anomaly similarity corresponding to the population; S15, updating the fitness probability based on the global anomaly similarity corresponding to the population; S16, generating random numbers, and determining the search method based on the random numbers and the fitness probability, where the search method is either a global search or a local search; S17, updating the population based on the search method; S18, determining whether the termination condition is met. If yes, determining the acoustic array based on the global optimal solution; otherwise, executing S13.

[0009] Furthermore, based on the sound data corresponding to each fault type at the effective detection location, the sampling probability of the effective detection location is calculated, including: for each fault type, generating a sound feature vector corresponding to the fault type at the effective detection location based on the sound data corresponding to the fault type at the effective detection location; for any two fault types, calculating the cosine distance between the sound feature vectors of the two fault types at the effective detection location; acquiring historical fault data and determining the fault occurrence probability of each fault type based on the historical fault data; and, based on the fault occurrence probability of each fault type, performing a weighted average of the cosine distance between the sound feature vectors of any two fault types at the effective detection location to obtain the weighted average cosine distance corresponding to the effective detection location, and calculating the sampling probability of the effective detection location.

[0010] Further, the fragrance intensity of each butterfly in the population is calculated, including: for each fault type, determining the effective detection locations in the butterfly that respond to the fault type based on the sound data corresponding to each effective detection location included in the butterfly; for any two fault types, calculating the similarity of the butterfly's responses to the two fault types based on the effective detection locations in the butterfly that respond to the two fault types; taking a weighted average of the similarity of the butterfly's responses to the two fault types based on the fault occurrence probability of each fault type, obtaining the weighted average similarity of the butterfly; calculating the detection coverage of the butterfly for multiple fault types of the power equipment based on the effective detection locations in the butterfly that respond to the fault type; and calculating the fragrance intensity of the butterfly based on the number of effective detection locations in the butterfly, the weighted average similarity of the butterfly, and the detection coverage of the butterfly for multiple fault types of the power equipment.

[0011] Furthermore, the global anomaly similarity for the population is calculated, including: for each butterfly in the population, generating a position matrix for the butterfly based on the effective detection positions of each fault type, where each row vector of the position matrix corresponds to a fault type, and the row vectors of the position matrix represent the effective detection positions of the fault type; for any two butterflies in the population, calculating the cosine similarity of the position matrices corresponding to the two butterflies; and calculating the global anomaly similarity for the population based on the cosine similarity of the position matrices corresponding to any two butterflies in the population, where a higher global anomaly similarity indicates a higher fitness probability.

[0012] Furthermore, the status monitoring module determines the abnormal noise coordinates based on the sound data collected by the acoustic array, including: acquiring abnormal noise data for multiple calibration coordinates collected by the acoustic array; for each calibration coordinate, determining multiple responding sound acquisition devices corresponding to the calibration coordinate based on the abnormal noise data collected by the acoustic array; determining the current multiple responding sound acquisition devices based on the sound data collected by the acoustic array; determining the target calibration coordinate based on the current multiple responding sound acquisition devices and the multiple responding sound acquisition devices corresponding to each calibration coordinate; and determining the abnormal noise coordinates based on the target calibration coordinates.

[0013] Furthermore, the status monitoring module determines the abnormal noise coordinates based on the target calibration coordinates, including: for each target calibration coordinate, calculating the matching degree of the target calibration coordinates based on the current multiple response sound acquisition devices and the multiple response sound acquisition devices corresponding to the target calibration coordinates; and determining the abnormal noise coordinates based on each target calibration coordinate and the matching degree of the target calibration coordinates.

[0014] Furthermore, the fault diagnosis module performs fault diagnosis based on the sound data collected by the acoustic array, the images collected by the image acquisition device, and the infrared thermal sensing data collected by the infrared thermal sensing device. This includes: determining the number of abnormal modes based on the sound data collected by the acoustic array, the images collected by the image acquisition device, and the infrared thermal sensing data collected by the infrared thermal sensing device; determining the risk level based on the number of abnormal modes; and executing response actions.

[0015] Furthermore, the fault diagnosis module is also used to: construct a knowledge graph, wherein the knowledge graph is used to record the abnormal sound coordinates corresponding to each fault type and the abnormal features corresponding to each mode; and determine the current fault type based on the sound data collected by the acoustic array, the images collected by the image acquisition device, the infrared thermal sensing data collected by the infrared thermal sensing device, and the knowledge graph.

[0016] Compared with existing technologies, the power equipment condition monitoring and fault diagnosis system based on multimodal data provided by this invention has at least the following beneficial effects: 1. This system, through the distributed arrangement of acoustic arrays, can accurately capture abnormal noise signals from different locations of power equipment and calculate the three-dimensional coordinates of the sound source (abnormal noise coordinates), thus achieving spatial localization of the fault point. Based on these coordinates, the system dynamically controls the image acquisition device to focus on the abnormal area for high-definition imaging, while simultaneously triggering the infrared thermal sensing device to collect temperature distribution data of that area. This multimodal collaborative mechanism of "acoustic localization - visual / infrared verification" avoids the limitations of traditional single-modal detection (such as the inability of acoustics to identify structural damage and the difficulty of images to detect hidden overheating). Through dual verification at both the spatial and feature levels, it significantly improves the accuracy of fault type identification, and is particularly suitable for the early detection of complex defects (such as the simultaneous presence of discharge and overheating).

[0017] 2. Image and infrared data acquisition is triggered only when the acoustic array detects an anomaly, rather than continuous full-scale monitoring. This design significantly reduces the inactive working time of image and infrared sensors, lowers data storage and processing pressure (e.g., image data volume can be reduced by more than 70%), and avoids false alarms caused by environmental interference (such as background noise and solar radiation temperature rise). Furthermore, the precise guidance of the anomaly coordinates allows image and infrared data to be highly focused on the fault area, reducing interference from irrelevant areas and further improving the efficiency and reliability of subsequent fault diagnosis.

[0018] 3. From abnormal noise detection to multimodal data acquisition and fault diagnosis, the system achieves full-process automation. After the acoustic array analyzes the sound data in real time and generates the coordinates of the abnormal noise, it automatically links with image and infrared devices to complete data acquisition. The fault diagnosis module then quickly outputs the fault type and risk level based on multimodal data fusion analysis. This closed-loop process requires no manual intervention, and the response time can be controlled within seconds. Attached Figure Description

[0019] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein: Figure 1 This is a schematic diagram of a power equipment condition monitoring and fault diagnosis system based on multimodal data, as shown in some embodiments of this specification. Figure 2 This is a flowchart illustrating the determination of multiple mounting locations of an acoustic array according to some embodiments of this specification; Figure 3 This is a schematic diagram of the process for calculating the fragrance intensity of each butterfly in a population, based on some embodiments of this specification. Figure 4 This is a flowchart illustrating the process of determining the coordinates of abnormal noises according to some embodiments of this specification. Detailed Implementation

[0020] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0021] Figure 1 This is a schematic diagram of a power equipment condition monitoring and fault diagnosis system based on multimodal data, as shown in some embodiments of this specification. Figure 1 As shown, a power equipment condition monitoring and fault diagnosis system based on multimodal data can include a data acquisition module, a condition monitoring module, and a fault diagnosis module.

[0022] The data acquisition module may include an acoustic array, an image acquisition device, and an infrared thermal sensing device.

[0023] Image acquisition devices (such as industrial cameras and webcams) are used to capture visible light images of power equipment. These devices employ two-axis (horizontal / vertical) or three-axis (pitch / yaw / roll) motorized pan-tilt heads to adjust the image acquisition angle. Infrared thermal sensors (such as infrared thermal imagers) are used to detect the surface infrared radiation intensity of power equipment. They can be mounted on the motorized pan-tilt head of the image acquisition device to achieve synchronous angle adjustment.

[0024] The acoustic array includes sound acquisition devices (e.g., miniature microphones) located at multiple positions on the power equipment.

[0025] Specifically, the installation location and number of sound acquisition devices can vary depending on the type of power equipment. For example, for transformers, 8-12 microphones can be arranged around the transformer tank, with a radius of 1-2 meters and a height level with the windings. An array can be placed at the top (oil pump / fan area), middle (winding area), and bottom (radiator area) of the transformer. Another example is high-voltage switchgear, where 4-6 microphones are vertically arranged on the front (operating mechanism area) of the switchgear, spaced 0.3-0.5 meters apart, with the microphones embedded in the cabinet panel to avoid electromagnetic interference. Yet another example is motors, where 4 miniature microphones (5mm in diameter) are attached to the motor end cover and bearing housing surface, spaced 10-15cm apart.

[0026] In some embodiments, installing an acoustic array includes: Identify multiple fault types and initial detection locations for power equipment; For each type of fault, abnormal noise data corresponding to the fault type is collected. The abnormal noise data includes sound data at each initial detection location. The sound data at the initial detection location may include data collected by the sound acquisition device at multiple test time points. For any two detection positions, the similarity of abnormal sounds between the two initial detection positions is calculated based on the sound data corresponding to each fault type at the two initial detection positions. Based on the similarity of abnormal sounds between any two initial detection locations, redundant initial detection locations are eliminated to determine the effective detection locations. The acoustic array is installed by using an adaptive search butterfly optimization algorithm based on sound data corresponding to each fault type at effective detection locations.

[0027] Specifically, multiple fault types in power equipment can be identified by the abnormal noises emitted when the fault occurs, such as bearing wear, loose couplings, jamming operating mechanisms, transformer winding insulation breakdown, and switchgear contact erosion. These fault types can be determined manually. Multiple initial detection locations need to cover the components where these fault types occur, such as high-voltage / medium-voltage / low-voltage windings, winding end insulation, core laminations, core grounding plates, and core clamps. Multiple initial detection locations can be determined manually based on historical data, or they can be evenly distributed across the power equipment.

[0028] Specifically, the similarity of abnormal sounds between two initial detection locations can be calculated using the following procedure: For each initial detection location and each fault type, extract the features of the sound data corresponding to the fault type at the initial detection location (e.g., time domain (RMS value, kurtosis), frequency domain (band energy, dominant frequency), etc.) to generate the sound feature vector corresponding to the fault type at the initial detection location. For any two initial detection locations and each fault type, calculate the cosine similarity between the sound feature vectors of the two initial detection locations corresponding to the fault types; Based on historical fault data, determine the probability of occurrence for each fault type; For any two initial detection locations, the cosine similarity of the sound feature vectors corresponding to each fault type is calculated based on the probability of each fault type as the weight of each fault type, and the cosine similarity of each fault type is calculated based on the weight of each fault type. This cosine similarity is then used as the similarity of the abnormal sounds between the two initial detection locations.

[0029] If the similarity of the abnormal sounds between two initial detection positions is greater than the similarity threshold (e.g., 0.7), then the mean of the similarity between each of the two initial detection positions and the other initial detection positions is calculated. The initial detection position with the larger mean is taken as the redundant initial detection position, and the initial detection position with the smaller mean is taken as the valid detection position.

[0030] Understandably, by calculating the similarity of abnormal sounds between initial detection locations and eliminating redundant points, the number of sensors can be reduced by 30%-50%. For example, for two types of faults—transformer winding insulation breakdown and core lamination vibration—if the cosine similarity of the sound feature vectors of two detection locations exceeds a threshold, the point with more significant acoustic characteristics is retained, avoiding duplicate monitoring. This approach reduces hardware costs by approximately 40% while maintaining the fault detection rate. The probability of fault occurrence is determined based on historical fault data, and this probability is used as a weight to calculate weighted similarity. Sound features corresponding to high-probability faults are prioritized for retention, ensuring that monitoring resources are allocated to high-risk areas. In practical applications, this improves the accuracy of identifying critical faults by 25%.

[0031] Figure 2 This is a flowchart illustrating the determination of multiple mounting locations for an acoustic array according to some embodiments of this specification, such as... Figure 2 As shown, in some embodiments, an adaptive search butterfly optimization algorithm is used to determine multiple installation locations of the acoustic array based on sound data corresponding to each fault type at effective detection locations, including: S11. For each valid detection location, calculate the sampling probability of the valid detection location based on the sound data of each fault type corresponding to the valid detection location. S12. Initialize the population based on the sampling probability of each valid detection location, where each butterfly in the population represents a combination of valid detection locations. S13. Calculate the fragrance intensity of each butterfly in the population; S14. Calculate the global anomaly similarity for the population; S15. Update the fitness probability based on the global anomaly similarity corresponding to the population; S16. Generate random numbers, and determine the search method based on the random numbers and the fitness probability, wherein the search method is either a global search or a local search; S17. Population update based on search method; S18. Determine if the termination condition is met. If yes, determine the acoustic array based on the global optimal solution. If not, execute S13.

[0032] In some embodiments, the sampling probability of a valid detection location is calculated based on the sound data corresponding to each fault type at the valid detection location, including: For each fault type, based on the sound data corresponding to the fault type at the valid detection location, a sound feature vector corresponding to the fault type at the valid detection location is generated. Specifically, the method of generating the sound feature vector corresponding to the fault type at the valid detection location is similar to that of generating the initial sound feature vector corresponding to the fault type at the detection location, and will not be elaborated here. For any two fault types, calculate the cosine distance between the sound feature vectors of the two fault types corresponding to the effective detection locations; Acquire historical fault data and, based on this data, determine the probability of occurrence for each fault type. Based on the probability of occurrence of each fault type, the cosine distance between the sound feature vectors of any two fault types corresponding to the effective detection location is weighted and averaged to obtain the weighted average cosine distance corresponding to the effective detection location, and the sampling probability of the effective detection location is calculated.

[0033] Specifically, for any two fault types, the average probability of occurrence of the two fault types can be used as the weight of the cosine distance between the sound feature vectors of the effective detection positions corresponding to the two fault types. Based on the weight of the cosine distance between the sound feature vectors of the effective detection positions corresponding to the two fault types, a weighted average of the cosine distance between the sound feature vectors of the effective detection positions corresponding to the two fault types is calculated to obtain the weighted average cosine distance corresponding to the effective detection positions.

[0034] The ratio of the weighted average cosine distance corresponding to a valid detection location to the sum of the weighted average cosine distances corresponding to all valid detection locations can be used as the sampling probability of a valid detection location.

[0035] Understandably, for each fault type, sound feature vectors are extracted from the sound data at the effective detection locations. This process is consistent with the feature extraction method for the initial detection locations, ensuring the accuracy of feature representation. Next, the cosine distance between the sound feature vectors of any two fault types at the effective detection locations is calculated. This distance reflects the degree of difference in sound features between different faults; a larger distance indicates a more distinct distinction between the two fault sound features. Simultaneously, the probability of occurrence for each fault type is determined based on historical fault data, serving as an indicator of fault importance. Then, a weighted average is calculated on the cosine distances of the sound feature vectors of any two fault types at the effective detection locations. The weights are taken as the average of the probability of occurrence for both fault types. This ensures that fault combinations with high probability of occurrence and significant differences in sound features occupy a more significant position in the calculation. The resulting weighted average cosine distance corresponding to the effective detection location comprehensively reflects the location's ability to distinguish between different faults and the probability of fault occurrence. Finally, the ratio of the weighted average cosine distance corresponding to the effective detection location to the sum of the distances corresponding to all effective detection locations is used as the sampling probability. A higher sampling probability indicates greater value of the detection location in fault monitoring.

[0036] During population initialization, the algorithm randomly selects butterflies based on the sampling probability of each valid detection location. Specifically, for each butterfly in the population (i.e., each combination of detection locations), the algorithm determines whether a specific detection location is included in the combination based on the sampling probability of each valid detection location. Detection locations with higher sampling probabilities are more likely to be selected into the combination, while those with lower sampling probabilities are less likely to be selected. In this way, the butterflies (detection location combinations) in the initialized population will be more likely to include detection locations that are more critical to fault detection, thereby improving the overall quality and search efficiency of the population.

[0037] For example, create an empty set for each butterfly. This set will be used to store the combination of detection positions corresponding to that butterfly. Generate an n-dimensional random number vector, where each random number is uniformly distributed in the interval [0, 1]. Here, n represents the total number of valid detection positions. For each detection position in the set of valid detection positions, compare the generated random number with the sampling probability of that valid detection position. If the random number is less than the probability, add that valid detection position to the combination corresponding to the current butterfly.

[0038] By initializing the population based on sampling probabilities, multiple different combinations of effective detection locations can be generated, providing the algorithm with a rich solution space. These different combinations cover a variety of possible detection location layouts, helping the algorithm find the global optimum in subsequent search processes, rather than being limited to local optima. Since the sampling probabilities consider factors such as differences in fault characteristics and the probability of fault occurrence, the initialized population is better suited to the actual needs of power equipment fault monitoring. Butterfly combinations in the population are more likely to cover high-fault areas and detection locations with obvious fault characteristics, thereby improving the overall quality of the population and search efficiency.

[0039] Figure 3 This is a schematic flowchart illustrating the calculation of the fragrance intensity of each butterfly in a population, based on some embodiments shown in this specification. Figure 3 As shown, in some embodiments, calculating the scent intensity of each butterfly in the population includes: For each fault type, based on the sound data corresponding to each fault type for each valid detection location included in the butterfly, determine the valid detection location in the butterfly that responds to the fault type; For any two fault types, find the effective detection locations in the butterfly that respond to the two fault types, and calculate the similarity of the butterfly's responses to the two fault types. Based on the probability of occurrence of each fault type, the weighted average of the similarity between the two fault types of the butterfly response is calculated to obtain the weighted average similarity of the butterfly. Based on the effective detection locations of the butterfly response fault types, calculate the detection coverage of the butterfly for multiple fault types of power equipment. The fragrance intensity of butterflies is calculated based on the number of effective detection locations in the butterflies, the weighted average similarity of the butterflies, and the detection coverage of butterflies for various fault types of power equipment.

[0040] Specifically, for each valid detection location included in the butterfly, sound data collected at the valid detection location under normal operating conditions is acquired, and a sound feature vector corresponding to the valid detection location under normal conditions is generated. For each fault type, based on the sound data corresponding to the fault type at the valid detection location, a sound feature vector corresponding to the fault type at the valid detection location is generated. The Euclidean distance between the sound feature vector corresponding to the fault type at the valid detection location and the sound feature vector corresponding to the normal state at the valid detection location is calculated. Valid detection locations with an Euclidean distance greater than a threshold (e.g., 3, 5, etc.) are considered valid detection locations in the butterfly that respond to the fault type.

[0041] The similarity of the butterfly's response to the two fault types is determined by the ratio of the number of valid detection locations included in the intersection of the valid detection locations responding to the two fault types in the butterfly to the union of the valid detection locations responding to the two fault types in the butterfly.

[0042] For any two fault types, the average probability of occurrence of the two fault types can be used as the weight of the similarity between the two fault types in the butterfly response. Based on the weight of the similarity between the two fault types in the butterfly response, a weighted average of the similarity between the two fault types in the butterfly response is calculated to obtain the weighted average similarity of the butterfly response.

[0043] For each fault type, if the number of valid detection locations responding to the fault type in the butterfly is greater than a threshold (e.g., 3, etc.), then the butterfly covers that fault type. The ratio of the number of fault types covered by the butterfly to the total number of fault types is calculated as the butterfly's detection coverage rate for multiple fault types of the power equipment.

[0044] The fewer the number of effective detection locations in a butterfly, the smaller the weighted average similarity of the butterfly, and the higher the detection coverage of the butterfly for multiple fault types of power equipment, the stronger the butterfly's fragrance.

[0045] For example, the intensity of a butterfly's scent can be calculated using the following formula: ;in, Let be the fragrance intensity of the i-th butterfly, and c be the sensory modality, a constant taking values ​​in the range [0, 1] used to adjust the perceived intensity of the fragrance. Let be the number of valid detection positions in the i-th butterfly. Let be the weighted average similarity score for the i-th butterfly. Let be the detection coverage of the i-th butterfly for various fault types of power equipment, and a be the power exponent, which is also a constant that takes values ​​in the range [0, 1] and is used to control the rate at which the fragrance intensity changes with the intensity of stimulation.

[0046] Understandably, by comparing the Euclidean distance of the sound feature vectors of effective detection locations under normal operating conditions and fault types, locations exceeding a threshold are identified as locations responding to the fault type. This method accurately filters out detection points that actually respond to the fault, avoiding interference from invalid locations, laying the foundation for subsequent accurate assessment of the butterfly's fault perception capabilities, and helping to more accurately locate the fault location of power equipment. The similarity is calculated by the ratio of the intersection to the union of the effective detection locations responding to two fault types, intuitively and reasonably reflecting the overlap and differences in the butterfly's perception of different fault types. A weighted average similarity is obtained by combining the mean of the occurrence probabilities of the two fault types as weights, comprehensively considering the actual probability of the fault occurring, making the similarity measurement more consistent with real-world scenarios. Based on the comparison of the number of effective detection locations responding to a fault type with a threshold, it is determined whether the butterfly covers that fault type, and then the ratio of the number of covered fault types to the total number is calculated as the detection coverage rate. This indicator comprehensively considers the butterfly's overall detection capability for multiple fault types of power equipment, clearly demonstrating the butterfly's detection breadth in complex fault environments. The weighted average similarity measures the degree of overlap in the butterfly's perception of different fault types. A smaller value indicates that the butterfly's response to different fault types varies significantly, meaning it can clearly distinguish multiple fault characteristics. For example, in electrical equipment, the sound characteristics of short-circuit and overload faults may differ. If a butterfly can perceive these two faults with low similarity, it means it has a strong ability to distinguish fault types. This ability to differentiate is crucial in complex fault scenarios because equipment may experience multiple faults simultaneously or sequentially. Accurately distinguishing fault types helps to quickly locate the root cause of the problem. Therefore, low similarity corresponds to high fragrance intensity, reflecting the incentive for this superior characteristic.

[0047] In some embodiments, calculating the global anomaly similarity corresponding to the population includes: For each butterfly in the population, a position matrix is ​​generated based on the effective detection positions of each fault type in the butterfly. Each row vector of the position matrix corresponds to a fault type, and the row vectors of the position matrix represent the effective detection positions of the fault types. For any two butterflies in the population, calculate the cosine similarity of the position matrices corresponding to the two butterflies; Based on the cosine similarity of the position matrices of any two butterflies in the population, the global anomaly similarity of the population is calculated. The higher the global anomaly similarity, the greater the fitness probability.

[0048] Specifically, in the row vector of the position matrix, each element corresponds to a valid detection position in the butterfly. The value of the element is the response identifier of the fault type corresponding to the valid detection position. If the valid detection position is a valid detection position for the fault type, the response identifier is 1; otherwise, it is 0.

[0049] The mean of the cosine similarity of the position matrices corresponding to any two butterflies can be used as the global anomaly similarity for the population.

[0050] For example, the fitness probability can be calculated using the following formula: ;in, Let be the probability of the t-th iteration. The probability is a preset value (e.g., 0.7, 0.8, etc.). Let be the global anomaly similarity in the t-th iteration.

[0051] Understandably, by constructing a position matrix with fault types as row vectors and effective detection locations as elements, the detection capabilities of butterflies are transformed into structured data. Each element, represented by 0 or 1, indicates the response state of the detection location to a specific fault type, preserving spatial distribution information while avoiding interference from numerical scale differences. This design allows cosine similarity to accurately measure the directional difference between two butterflies in fault detection coverage, rather than simply positional distance, providing a reliable basis for assessing population diversity. Global anomaly similarity, by aggregating the mean cosine similarity of all butterfly pairs, reflects the overall homogeneity of the population's coverage of fault types. When the similarity is high (i.e., most butterflies are concentrated in detecting the same type of fault), the system increases the adaptive probability, prioritizing the retention of individuals with differentiated detection capabilities to guide the algorithm to explore uncovered fault types; when the similarity is low (i.e., the butterfly detection range is dispersed), the basic exploration intensity is maintained to avoid excessive dispersion of search resources. This adaptive mechanism effectively prevents the algorithm from getting trapped in local optima, and is particularly suitable for detection scenarios with multiple fault types and high complexity. Global anomaly similarity reflects the degree of difference in detection capabilities among individuals in the population; a higher value indicates a more dispersed detection range and greater diversity among individuals. Increasing the fitness probability in this case prioritizes retaining individuals with differentiated detection capabilities and triggers a global search with a higher probability through a random number decision mechanism, accelerating the exploration of uncovered fault regions. Conversely, when global anomaly similarity is low (individual detection capabilities converge), the fitness probability decreases, tending to select local searches (such as neighborhood perturbation or gradient optimization), focusing on the refined development of the current region. This dynamic switching mechanism effectively avoids the "premature convergence" or "low search efficiency" problems caused by fixed search strategies in traditional algorithms.

[0052] The specific implementation methods of S16-S18 are similar to those of the existing butterfly optimization algorithm, and will not be described in detail here.

[0053] The status monitoring module can be used to determine the coordinates of abnormal noises based on the sound data collected by the acoustic array.

[0054] Figure 4 This is a flowchart illustrating the process of determining the coordinates of abnormal noises according to some embodiments of this specification, such as... Figure 4 As shown, the status monitoring module determines the coordinates of abnormal noises by including: Acquire abnormal noise data from multiple calibration coordinates collected by an acoustic array. The calibration coordinates can be the coordinates of locations on electronic devices where abnormal noise is likely to occur. Multiple calibration coordinates can be determined manually. For each calibration coordinate, based on the abnormal noise data of the calibration coordinate collected by the acoustic array, determine the multiple sound acquisition devices corresponding to the calibration coordinate; Based on the sound data collected by the acoustic array, determine the current multiple response sound acquisition devices; Based on the current multiple response sound acquisition devices and the multiple response sound acquisition devices corresponding to each calibration coordinate, the target calibration coordinates are determined; Based on the target calibration coordinates, determine the coordinates of the abnormal noise.

[0055] Specifically, the method for determining the sound acquisition device for the response is similar to the method for determining the effective detection location of the response fault type in the butterfly, and will not be elaborated here.

[0056] For each calibration coordinate, based on the multiple sound acquisition devices corresponding to the calibration coordinate, a response position vector corresponding to the calibration coordinate is generated. Each element of the response position vector corresponds to a valid detection position of a sound acquisition device. The value of the element is the response identifier of the calibration coordinate corresponding to the valid detection position. If the valid detection position is a valid detection position of the response calibration coordinate, the response identifier is 1; otherwise, it is 0.

[0057] The current response position vector is determined by referring to the method used to generate the response position vector corresponding to the calibration coordinates.

[0058] Calculate the cosine similarity between the response position vector corresponding to the calibration coordinates and the current response position vector, and use the calibration coordinates with a cosine similarity greater than the cosine similarity threshold (e.g., 0.7) as the target calibration coordinates.

[0059] In some embodiments, the condition monitoring module determines the abnormal noise coordinates based on the target calibration coordinates, including: For each target calibration coordinate, the matching degree of the target calibration coordinate is calculated based on the current multiple response sound acquisition devices and the multiple response sound acquisition devices corresponding to the target calibration coordinate. For example, the cosine similarity between the response position vector corresponding to the target calibration coordinate and the current response position vector is used as the matching degree of the target calibration coordinate. Based on the matching degree between each target calibration coordinate and the target calibration coordinate, the abnormal sound coordinates are determined. For example, the matching degree of the target calibration coordinates is used as a weight to perform weighted fusion of multiple target calibration coordinates to obtain the abnormal sound coordinates.

[0060] Understandably, by pre-calibrating the locations of electronic devices prone to abnormal noises (calibration coordinates) and establishing a vector model of their response relationship with the sound acquisition device, the system can quickly compare the similarity between the current abnormal noise data and the calibration data. Utilizing cosine similarity to quantify the matching degree of spatial response patterns allows for efficient selection of the most probable target calibration coordinates, avoiding the inefficiency of relying on manual experience or point-by-point scanning in traditional methods, and significantly shortening the abnormal noise localization time. For multiple target calibration coordinates that meet the similarity threshold, the system generates the final abnormal noise coordinates through weighted fusion (using matching degree as the weight), rather than simply taking the average or selecting the coordinate with the highest similarity. This processing method fully considers the correlation strength between different calibration coordinates and the current abnormal noise, and is particularly suitable for scenarios where the abnormal noise source is located in the intermediate region between multiple calibration coordinates or has blurred boundaries, effectively improving the accuracy of the localization results.

[0061] The data acquisition module is also used to control the image acquisition device to acquire images based on the coordinates of the abnormal noise, and to control the infrared thermal sensing device to acquire infrared thermal sensing data.

[0062] Specifically, the image acquisition device adjusts the shooting angle according to the coordinates of the abnormal noise, so that the coordinates of the abnormal noise are located in the center of the image, avoiding the loss of key areas due to field of view deviation.

[0063] The fault diagnosis module can be used to diagnose faults based on sound data acquired by an acoustic array, images acquired by an image acquisition device, and infrared thermal sensing data acquired by an infrared thermal sensing device.

[0064] Specifically, it includes: The number of abnormal modes is determined based on sound data acquired by an acoustic array, images acquired by an image acquisition device, and infrared thermal sensing data acquired by an infrared thermal sensing device. Based on the number of abnormal modes, the risk level is determined and response actions are executed.

[0065] Specifically, feature analysis is performed on the sound data (time-domain waveform or frequency-domain spectrum) collected by the acoustic array to extract key indicators such as sound pressure level, frequency components (e.g., 50Hz octave), and impulse pulse intensity. If the sound pressure level exceeds the normal operating threshold of the equipment (e.g., sound pressure level of motor bearing >85dB), abnormal frequency components appear in the spectrum (e.g., increased sidelobes at gearbox meshing frequency), or a discharge pulse signal is detected (e.g., high-frequency pulses of corona discharge from 10kHz to 100kHz), then an acoustic modal abnormality is determined.

[0066] After preprocessing (such as denoising and enhancement) the images acquired by the image acquisition device, key features are extracted through target detection (such as the YOLO algorithm) or image segmentation (such as U-Net), such as the area of ​​contamination on the insulator surface, crack length, and degree of corrosion of metal components. If the contamination coverage area exceeds a threshold (such as >30%), the crack length exceeds a safety limit (such as >5mm), or obvious damage is detected (such as broken insulator skirts), the image is determined to be modally abnormal.

[0067] A heat map is generated from the infrared thermal sensing data, and key temperature indicators such as maximum temperature, regional temperature difference, and temperature distribution uniformity are extracted. If the maximum temperature exceeds the allowable temperature rise of the equipment (e.g., motor winding temperature rise > 80K), the regional temperature difference exceeds the threshold (e.g., regional temperature difference > 15K), or the heat map shows local overheating (e.g., hot spot area > 10cm²), then an infrared modal anomaly is determined.

[0068] Abnormal mode count calculation: Count the number (0~3) of modes that simultaneously satisfy the abnormal rules among the above three modes. For example: If only the acoustics are abnormal, while the image and infrared are normal, then the number of abnormal modes is 1; If the acoustics and imaging are abnormal, but the infrared is normal, then the number of abnormal modes is 2; If all three modes are abnormal, then the number of abnormal modes is 3.

[0069] Based on the number of abnormal modes and combined with multimodal data correlation analysis, risk levels are classified and response strategies are formulated to achieve a tiered response from "early warning" to "emergency response": 1. Risk Level I (Low Risk) Triggering condition: Minor anomaly in a single mode (number of anomalous modes = 1).

[0070] Response action: Platform Alarms: Generate text or pop-up alarms on the monitoring platform, and record the abnormal time, mode, and characteristic values.

[0071] Data logging: Store raw data (such as sound spectrum, images, heat maps) in a historical database for subsequent analysis.

[0072] 2. Risk Level II (Medium Risk) Triggering condition: Bimodal association anomaly (number of anomalous modalities = 2).

[0073] Response action: Audible and visual alarm: Triggers the on-site audible and visual alarm to alert maintenance personnel.

[0074] Push maintenance suggestions: Generate targeted suggestions based on abnormal mode combinations (such as "check the cleaning status of insulators and measure insulation resistance" or "investigate discharge points and reinforce grounding devices").

[0075] 3. Risk Level III (High Risk) Triggering condition: Trimodal composite anomaly.

[0076] Response action: Emergency power outage: Automatically triggers circuit breaker tripping or remotely cuts off equipment power to prevent the fault from escalating (such as short circuit caused by insulator breakdown).

[0077] Defect coordinate location: By combining the spatial coordinates of the discharge point located by the acoustic array, the location of the damage identified by the image, and the overheated area marked by the infrared thermal image, a three-dimensional defect map is generated.

[0078] In some embodiments, the fault diagnosis module is further configured to: Construct a knowledge graph, which is used to record the abnormal sound coordinates corresponding to each fault type and the abnormal features corresponding to each mode; Based on sound data collected by the acoustic array, images collected by the image acquisition device, infrared thermal sensing data collected by the infrared thermal sensing device, and knowledge graph, the current fault type is determined.

[0079] Specifically, the fault diagnosis module achieves accurate identification of fault types by constructing a knowledge graph. This knowledge graph integrates multimodal fault knowledge in a structured form, containing two core layers of relationships: First, it horizontally records the spatial coordinates of acoustic abnormal sounds corresponding to various fault types (such as insulator pollution overheating, corona discharge, mechanical cracks, etc.) (the three-dimensional position of the discharge point located by the acoustic array), and vertically associates the abnormal feature thresholds and manifestations of each mode (acoustic, image, infrared) (such as the pulse frequency range in acoustics, the pollution coverage area threshold in images, and the temperature rise gradient in infrared); Second, it vertically establishes the mapping relationship between fault types and multimodal abnormal features. For example, the "pollution overheating" fault is simultaneously associated with the acoustic mode "50-100kHz high-frequency discharge pulse", the image mode "surface pollution coverage >30%", and the infrared mode "local temperature rise >15K".

[0080] In actual diagnosis, the fault diagnosis module first performs spectral analysis on the sound data collected by the acoustic array to locate the coordinates of abnormal noises and extract pulse features; simultaneously, it performs target detection on the image to identify structural anomalies such as dirt and cracks; and it performs temperature gradient analysis on the infrared thermogram to calculate the regional temperature difference. Subsequently, using the real-time acquired multimodal data as input, a three-way matching is performed in the knowledge graph: the possible fault area is located by the coordinates of the acoustic abnormal noise, and then the image and infrared data are compared to see if they meet the modal anomaly feature combination of the corresponding fault type in that area (such as the simultaneous occurrence of acoustic pulses, image dirt, and infrared temperature rise at coordinate X). If the three features completely match the record of a certain fault type in the knowledge graph, then it is determined to be that fault type (such as "dirt overheating"); if there is a partial match, fuzzy reasoning is performed by combining historical data and dynamic threshold adjustments, and finally the most likely fault type and confidence level are output, providing a basis for subsequent risk classification and handling.

[0081] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A power equipment condition monitoring and fault diagnosis system based on multimodal data, characterized in that, include: The data acquisition module includes an acoustic array, an image acquisition device, and an infrared thermal sensing device, wherein the acoustic array includes multiple sound acquisition devices disposed at locations on the power equipment. The status monitoring module is used to determine the coordinates of abnormal noises based on the sound data collected by the acoustic array; The data acquisition module is also used to control the image acquisition device to acquire images based on the abnormal sound coordinates, and to control the infrared thermal sensing device to acquire infrared thermal sensing data. The fault diagnosis module is used to diagnose faults based on sound data acquired by the acoustic array, images acquired by the image acquisition device, and infrared thermal sensing data acquired by the infrared thermal sensing device.

2. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 1, characterized in that, Installing an acoustic array includes: Identify multiple fault types and initial detection locations for power equipment; For each fault type, abnormal noise data corresponding to the fault type is collected, where the abnormal noise data includes sound data at each initial detection location; For any two detection positions, the similarity of abnormal sounds between the two initial detection positions is calculated based on the sound data corresponding to each fault type at the two initial detection positions. Based on the similarity of abnormal sounds between any two initial detection locations, redundant initial detection locations are eliminated to determine the effective detection locations. The acoustic array is installed by using an adaptive search butterfly optimization algorithm based on sound data corresponding to each fault type at effective detection locations.

3. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 2, characterized in that, Using an adaptive search butterfly optimization algorithm, multiple installation locations for the acoustic array are determined based on sound data corresponding to each fault type at effective detection locations, including: S11. For each valid detection location, calculate the sampling probability of the valid detection location based on the sound data of each fault type corresponding to the valid detection location. S12. Initialize the population based on the sampling probability of each valid detection location, where each butterfly in the population represents a combination of valid detection locations. S13. Calculate the fragrance intensity of each butterfly in the population; S14. Calculate the global anomaly similarity for the population; S15. Update the fitness probability based on the global anomaly similarity corresponding to the population; S16. Generate random numbers, and determine the search method based on the random numbers and the fitness probability, wherein the search method is either a global search or a local search; S17. Population update based on search method; S18. Determine if the termination condition is met. If yes, determine the acoustic array based on the global optimal solution. If not, execute S13.

4. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 3, characterized in that, Based on the sound data corresponding to each fault type at the effective detection locations, the sampling probability of the effective detection locations is calculated, including: For each fault type, an effective sound feature vector corresponding to the fault type is generated based on the sound data of the effective detection location corresponding to the fault type. For any two fault types, calculate the cosine distance between the sound feature vectors of the two fault types corresponding to the effective detection locations; Acquire historical fault data and, based on this data, determine the probability of occurrence for each fault type. Based on the probability of occurrence of each fault type, the cosine distance between the sound feature vectors of any two fault types corresponding to the effective detection location is weighted and averaged to obtain the weighted average cosine distance corresponding to the effective detection location, and the sampling probability of the effective detection location is calculated.

5. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 3, characterized in that, Calculate the scent intensity of each butterfly in the population, including: For each fault type, based on the sound data corresponding to each fault type for each valid detection location included in the butterfly, determine the valid detection location in the butterfly that responds to the fault type; For any two fault types, find the effective detection locations in the butterfly that respond to the two fault types, and calculate the similarity of the butterfly's responses to the two fault types. Based on the probability of occurrence of each fault type, the weighted average of the similarity between the two fault types of the butterfly response is calculated to obtain the weighted average similarity of the butterfly. Based on the effective detection locations of the butterfly response fault types, calculate the detection coverage of the butterfly for multiple fault types of power equipment. The fragrance intensity of butterflies is calculated based on the number of effective detection locations in the butterflies, the weighted average similarity of the butterflies, and the detection coverage of butterflies for various fault types of power equipment.

6. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 3, characterized in that, Calculate the global anomaly similarity for the population, including: For each butterfly in the population, a position matrix is ​​generated based on the effective detection positions of each fault type in the butterfly. Each row vector of the position matrix corresponds to a fault type, and the row vectors of the position matrix represent the effective detection positions of the fault types. For any two butterflies in the population, calculate the cosine similarity of the position matrices corresponding to the two butterflies; Based on the cosine similarity of the position matrices of any two butterflies in the population, the global anomaly similarity of the population is calculated. The higher the global anomaly similarity, the greater the fitness probability.

7. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to any one of claims 2-6, characterized in that, The status monitoring module determines the coordinates of abnormal noises based on the sound data collected by the acoustic array, including: Acquire abnormal noise data from multiple calibration coordinates collected by the acoustic array; For each calibration coordinate, based on the abnormal noise data of the calibration coordinate collected by the acoustic array, determine the multiple sound acquisition devices corresponding to the calibration coordinate; Based on the sound data collected by the acoustic array, determine the current multiple response sound acquisition devices; Based on the current multiple response sound acquisition devices and the multiple response sound acquisition devices corresponding to each calibration coordinate, the target calibration coordinates are determined; Based on the target calibration coordinates, determine the coordinates of the abnormal noise.

8. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 7, characterized in that, The status monitoring module determines the coordinates of abnormal noises based on the target calibration coordinates, including: For each target calibration coordinate, the matching degree of the target calibration coordinate is calculated based on the current multiple response sound acquisition devices and the multiple response sound acquisition devices corresponding to the target calibration coordinate; The coordinates of the abnormal noise are determined based on the matching degree between each target calibration coordinate and the target calibration coordinate.

9. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to any one of claims 1-6, characterized in that, The fault diagnosis module performs fault diagnosis based on sound data acquired by the acoustic array, images acquired by the image acquisition device, and infrared thermal sensing data acquired by the infrared thermal sensing device, including: The number of abnormal modes is determined based on sound data acquired by an acoustic array, images acquired by an image acquisition device, and infrared thermal sensing data acquired by an infrared thermal sensing device. Based on the number of abnormal modes, the risk level is determined and response actions are executed.

10. The power equipment condition monitoring and fault diagnosis system based on multimodal data according to claim 8, characterized in that, The fault diagnosis module is also used for: Construct a knowledge graph, wherein the knowledge graph is used to record the abnormal sound coordinates corresponding to each fault type and the abnormal features corresponding to each mode; Based on sound data collected by the acoustic array, images collected by the image acquisition device, infrared thermal sensing data collected by the infrared thermal sensing device, and knowledge graph, the current fault type is determined.

Images