Power information multi-modal data dynamic integration method and system

Abstract

The invention provides an electric power information multi-modal data dynamic integration method and system, relates to the technical field of electric power information data integration, and realizes multi-modal data fusion through a technical chain of asynchronous receiving, feature extraction, elastic alignment, dynamic verification and intelligent compensation. The method comprises the following steps: acquiring a sensor numerical value, an infrared image and an environment audio heterogeneous data stream in real time; extracting each modal feature and calculating a historical deviation coefficient; sensor data is used as a reference axis, and a multi-modal maximum feature coincidence interval is determined through sliding window matching; performing cross-modal feature cross validation in the extended dynamic interaction interval, and marking credible fusion features; redundant feature compensation is triggered for the modals which do not pass verification, and an integrated feature vector is generated; and if continuous failure occurs, marking the data source as a low-confidence data source. Through elastic time alignment and a dynamic verification mechanism, the problem of native asynchronization of power multi-modal data is solved, the feature fusion reliability is improved, and accurate fault information can be provided.

Description

Technical Field

[0001] The present invention relates to the technical field of power information data integration, and in particular to a method and system for dynamically integrating multi-modal data of power information. Background Art

[0002] In the field of intelligent operation and maintenance of power systems, multi-modal data integration technology has become the core supporting means for fault diagnosis. With the improvement of the power equipment monitoring system, the real-time acquisition of heterogeneous data sources such as sensor values, infrared thermal images, and audio vibrations has become the norm. These data have significant differences in time scale, physical dimension, and information density. The biggest challenge currently faced by the industry lies in the dynamic fusion reliability problem of asynchronous multi-modal data - when sensor data with different acquisition frequencies needs to be integrated with unstructured data such as images and audio in real time, the existing technologies usually adopt methods of forced alignment on the time axis or fixed-weight fusion. These methods have two fatal defects: First, forced synchronization will artificially split the inherent temporal correlation of each modal data. For example, the spatio-temporal coupling characteristics of the acoustic signal and infrared hot spot of partial discharge in a transformer may be lost due to alignment errors. Second, the fixed-weight strategy cannot adapt to the dynamic fluctuations of the quality of multi-modal data. When a certain sensor fails temporarily due to electromagnetic interference, low-quality data is still equally included in the integration calculation, resulting in distorted diagnostic results.

[0003] In recent years, although some studies have tried to introduce timestamp calibration or sliding window technology, these improvements are essentially "remedial measures afterwards" and have not solved the fundamental problem of multi-modal asynchrony from the data acquisition source. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the huge differences in time scale, spatial dimension, and information density of heterogeneous data in the prior art. The present invention provides a method and system for dynamically integrating multi-modal data of power information. Through a technical chain of asynchronous reception, feature extraction, elastic alignment, dynamic verification, and intelligent compensation, the native asynchrony problem of power multi-modal data is solved, the reliability of feature fusion is improved, and accurate fault information can be provided.

[0005] To solve the above technical problem, the present invention provides a method for dynamically integrating multi-modal data of power information, including the following steps: Real-time obtain data streams of at least two heterogeneous modalities in a power system, including sequences of device sensor values, sequences of infrared images on the surface of the device, and environmental audio waveforms. Each modal data stream is collected with an independent clock cycle and is not pre-synchronized. For each modal data stream, extract the heterogeneous feature set within the current time window, including the mutation gradient of sensor values, the contour area of hot spots in infrared images, and the energy peak of specific frequency bands in audio waveforms, and calculate the deviation coefficient of each feature under the historical normal state; Taking the sensor value sequence as the reference time axis, perform a sliding window match between the peak moment of the feature deviation coefficient of other modal data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference; According to the maximum feature coincidence time interval, expand the original data stream of each modality in the time dimension to generate a dynamic interaction interval, the interval length is 1.2 - 1.5 times the original time window, and perform cross-validation on the extracted heterogeneous feature set within this interval. If the deviation directions of the features of at least two modalities are consistent, it is marked as a credible fusion feature; When a certain modality fails the cross-validation within the dynamic interaction interval, trigger the extraction of redundant features of other modalities within adjacent time windows, logically combine the redundant features with the current credible fusion features to generate a compensated integrated feature vector; if the compensated features still cannot pass the verification, mark the modality data as a low-confidence data source and skip its time window matching process in subsequent integration.

[0006] In an embodiment of the present invention, the process of extracting the mutation gradient of the sensor values includes: Perform three-level progressive filtering on the original sensor data stream, first eliminate the instantaneous pulse noise caused by electromagnetic interference, then smooth the periodic fluctuations caused by mechanical vibrations, and finally retain the real signal whose amplitude exceeds the baseline value; Automatically update the baseline value at regular intervals, use the sliding median filtering method to establish the upper and lower threshold bands, and trigger mutation detection when the data points break through the threshold band continuously for multiple times; Within the time window where a mutation is detected, calculate the maximum slope value of the signal change rate, and at the same time record the cumulative time during which this slope continuously exceeds the critical value. Multiply the maximum slope value by the cumulative time to obtain the mutation gradient feature value.

[0007] In an embodiment of the present invention, the process of extracting the contour area of hot spots in the infrared image includes: Segment the device body area for each frame of infrared image to exclude background interference, and then perform horizontal and vertical double scans on the device surface temperature matrix to mark the temperature mutation areas; Based on the heat conduction characteristics of the device material, when the temperature difference between adjacent pixels is detected to exceed the normal heat transfer limit of this material, automatically lower the segmentation threshold to ensure that small hot spots are not missed; Perform multi-frame continuity inspection on the preliminarily identified hot spot areas, and only retain the areas where the area expansion speed is within the preset growth interval in multiple consecutive frames; Take the circumscribed rectangle of the verified hot spot area, and use the ratio of its area to the area of the reference area on the device surface as the characteristic output value.

[0008] In an embodiment of the present invention, the process of extracting the energy peak value in a specific frequency band of the audio waveform includes: Collect the ambient sound for 30 seconds at the initial stage of device startup, and automatically generate a base spectrum template including ambient noise and electromagnetic noise; Real-time monitor the pulse bands that suddenly appear in the audio stream, and these bands need to meet both: the duration is shorter than the device vibration period, and the main frequency deviates from the core frequency band of the base template; Perform three-layer fine division on the suspicious band. First, detect the sudden increase in energy in the full frequency band, then locate the resonance point in the sub-frequency band, and finally extract the frequency range with the most concentrated energy aggregation as the characteristic frequency band; Only when the energy value of this frequency band continuously exceeds the historical average value within a certain time and coincides with the mechanical action time of the device, it is recorded as an effective feature.

[0009] In an embodiment of the present invention, the process of calculating the deviation coefficient of each feature in the historical normal state includes: Collect the historical feature data of each mode during the normal operation of the device, establish a hierarchical statistical model according to different working conditions, and record the distribution interval and typical fluctuation mode of the feature values for each level; Match the feature value of the currently extracted time window to the corresponding working condition level, and calculate the percentage of its deviation from the median value of this level as the instantaneous deviation degree; For the continuously deviated features, apply a progressive coefficient according to the deviation duration, set time nodes, grow linearly before the time nodes, and change to square root growth after the time nodes; Multiply the instantaneous deviation degree by the progressive coefficient, and then superimpose the collaborative deviation degrees of other modes, and finally output the standardized deviation coefficient between 0 and 10.

[0010] In an embodiment of the present invention, the sliding window matching ensures accuracy in the following way: Take the sensor numerical mutation point as the center, and extend 50 milliseconds forward and backward as the reference time anchor point; The first-level rough matching compresses other mode data into a simplified sequence at 10-millisecond intervals for quick alignment, and the second-level fine matching performs millisecond-level fine-tuning within the range of ±20 milliseconds of the rough matching result; It is required that the occurrence time of the hot spot in the image mode must be included in the duration of the audio pulse, and the time differences between both of them and the sensor mutation are less than the upper limit of the device physical response delay; When the time differences of the features of each mode exceed the system allowable error, the matching result of the sensor and the infrared image is preferably adopted, and the audio data is converted for auxiliary verification purposes.

[0011] In one embodiment of the present invention, the process of cross - validating the extracted heterogeneous feature set includes: Establish the mapping relationship between each modal feature and the physical state of the device, including: the mutation gradient of the sensor value is associated with the change of the contact state of the conductive component, the contour area of the infrared hot spot is associated with the local temperature rise of the device, and the peak value of the audio energy is associated with the vibration damage of the mechanical structure; It is required that the physical quantity indicators of at least two modalities exceed their safety thresholds simultaneously and the change directions are the same, that is, they are marked as credible features; For the currently marked credible features, it is necessary to reverse - check whether there are progressive abnormal signs in a previous period of time. If there are progressive abnormal signs, secondary verification is started and it is determined as a sudden device failure.

[0012] In one embodiment of the present invention, when a certain modality fails to pass the cross - validation within the dynamic interaction interval, the process of triggering the extraction of redundant features of other modalities within adjacent time windows includes: Identify the modal data stream that fails to pass the verification, record its timestamp deviation value and feature deviation direction, and at the same time freeze the original feature output of this modality in the current window; Centered on the reference time axis, expand 1 acquisition period forward and backward respectively, and extract the following redundant features from other verified modalities: For the sensor modality: extract the gradient change rate of the forward window and the numerical fluctuation amplitude of the backward window; For the infrared modality: extract the morphological similarity of the hot spot contours in adjacent windows and the area change trend; For the audio modality: extract the energy attenuation coefficient and frequency shift ratio of the same frequency band in the front and rear windows.

[0013] In one embodiment of the present invention, the process of logically combining the redundant features with the current credible fusion features to generate a compensated integrated feature vector includes: Mark the verified credible fusion features as first - level features, mark the redundant features with a high degree of matching with historical data as second - level features, and mark the remaining redundant features as third - level features; Perform weighted summation on the first - level features, second - level features and third - level features according to preset fixed weights, where: the weight of the first - level features is greater than that of the second - level features, and the weight of the second - level features is greater than that of the third - level features, and output the weighted result as the final integrated feature vector.

[0014] To solve the above - mentioned technical problems, the present invention also provides a power information multi - modal data dynamic integration system, including: A multi-source heterogeneous data acquisition module is used to obtain data streams of at least two heterogeneous modalities in the power system in real time, including device sensor numerical sequences, device surface infrared image sequences, and environmental audio waveforms. Each modality data stream is acquired with an independent clock cycle and is not pre-synchronized. A dynamic feature extraction module is used to extract a heterogeneous feature set within the current time window for each modality data stream, including the mutation gradient of sensor values, the hot spot contour area in infrared images, and the energy peak value in a specific frequency band of the audio waveform, and calculate the deviation coefficient of each feature from its historical normal state. A time axis alignment module is used to take the sensor numerical sequence as the reference time axis, and perform a sliding window match between the peak moments of the feature deviation coefficients of other modality data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference. A cross-validation module is used to expand each modality's original data stream in the time dimension to generate a dynamic interaction interval according to the maximum feature coincidence time interval. The interval length is 1.2 - 1.5 times the original time window, and cross-validate the extracted heterogeneous feature set within this interval. If the feature deviation directions of at least two modalities are consistent, they are marked as credible fusion features. A redundancy compensation module is used to trigger the extraction of redundant features of other modalities within adjacent time windows when a certain modality fails the cross-validation within the dynamic interaction interval, and logically combine the redundant features with the current credible fusion features to generate a compensated integrated feature vector. When the compensated features still cannot pass the validation, the data of this modality is marked as a low-confidence data source, and its time window matching process is skipped in subsequent integrations.

[0015] The above technical solution of the present invention has the following advantages compared with the prior art: The present invention provides a method and system for dynamically integrating multi-modal power information, which realizes the native asynchronous processing and cross-modal feature fusion of multi-modal power data through a technical chain of asynchronous reception, feature extraction, elastic alignment, dynamic verification, and intelligent compensation.

[0016] Aiming at the problem that forced synchronization destroys the timing correlation, the present invention adopts a dual mechanism of peak matching of feature deviation coefficients and dynamic interaction interval expansion. By extracting the feature deviation coefficients of each modality data and identifying their peak moments, a sliding window match based on feature significance is established on the reference time axis. This method retains the inherent timing coupling relationship of different modality data; subsequently, cross-validation is performed by dynamically expanding the time interval, which not only avoids the rigid cutting of fixed time alignment but also provides an elastic matching space for the spatio-temporal coupling of multi-modal features; this feature-driven adaptive synchronization method can capture the physical association between multi-modal data more essentially than traditional timestamp calibration or fixed window techniques.

[0017] In view of the deficiencies of fixed-weight fusion, the technical solution of the present invention constructs a dynamic fault-tolerant chain of "cross-validation - redundancy compensation - confidence evaluation". When the quality of a certain modal data fluctuates, first, the consistency verification of the deviation direction of multi-modal features is used to screen credible features. When the verification fails, the redundant feature compensation mechanism of adjacent windows is triggered to form integrated features after logical combination. Finally, the window skipping strategy is implemented for the modality that continuously fails the verification. This hierarchical processing method realizes multiple verifications of data feature fusion. Compared with the equal fusion strategy of the prior art, it can automatically suppress the interference of low-quality data while retaining the contribution degree of valid information, enabling multi-modal data to accurately represent the fault information of power equipment. Description of the Drawings

[0018] To make the content of the present invention easier to be clearly understood, the following further details the present invention according to the specific embodiments of the present invention in conjunction with the drawings, where: Figure 1 is the flowchart of the steps of the dynamic integration method for multi-modal power information data of the present invention; Figure 2 is the structural framework diagram of the dynamic integration system for multi-modal power information data of the present invention. Detailed Embodiments

[0019] The following further illustrates the present invention in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited do not limit the present invention.

[0020] Refer to Figure 1 As shown, the present invention discloses a dynamic integration method for multi-modal power information data, which is characterized in that it includes the following steps: S10. Real-time acquire data streams of at least two heterogeneous modalities in the power system, including device sensor numerical sequences, device surface infrared image sequences, and environmental audio waveforms. Each modal data stream is collected with an independent clock cycle and is not pre-synchronized.

[0021] As mentioned above, with the improvement of the power equipment monitoring system, the real-time acquisition of heterogeneous data sources such as sensor numerical values, infrared thermal imaging, and audio vibrations has become normal. For example, in the scenario of partial discharge monitoring of transformers, current and voltage sensors may collect data at a millisecond-level frequency, while the sampling frequency of an infrared thermal imager may be only a few frames per second, and the audio waveform extraction frequency band is continuous. Traditional methods will force the alignment of timestamps, resulting in the smoothing of transient features of high-frequency vibration signals. When collecting each heterogeneous data in this embodiment, the independent clock cycle of each modality is retained to ensure the integrity of the original time sequence features of the data and avoid information loss caused by artificial forced synchronization, laying a foundation for subsequent adaptive fusion.

[0022] S20. For each modal data stream, extract the heterogeneous feature set within the current time window, including the mutation gradient of sensor values, the contour area of hot spots in infrared images, and the energy peak of specific frequency bands in audio waveforms, and calculate the deviation coefficient of each feature under the historical normal state.

[0023] Specifically, extract comparable physical features from different modal data and quantify their degree of abnormality. Taking transformer monitoring as an example: extract the mutation gradient from sensor values (such as current and voltage) to reflect the rapid change of electrical parameters, extract the contour area of hot spots from infrared images to identify local overheating areas, and extract the energy peak of specific frequency bands (such as ultrasonic waves) from audio waveforms to detect partial discharge signals; calculate the deviation coefficient of these features under the historical normal state to quantify the abnormality of the current data and avoid the noise interference caused by directly fusing the original data.

[0024] S30. Taking the sensor value sequence as the reference time axis, perform sliding window matching between the peak moments of the feature deviation coefficients of other modal data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference.

[0025] Specifically, introduce a dynamic alignment mechanism of "reference time axis + sliding window matching". Taking electrical quantity data as the reference, other modal data are elastically matched through the peak moments of the feature deviation coefficients. Without forcibly aligning the timestamps, the best matching interval of multi-modal data can be found; assume that a mutation is detected in the current sensor data (reference time axis), and the infrared image captures a hot spot a few milliseconds later. Traditional methods would discard some data due to the time difference, while in this step, the peak moments of the feature deviation coefficients of each modality are found through sliding window matching to determine the maximum feature coincidence time interval. In this way, even if the data acquisition times are not exactly the same, the spatio-temporal correlation of key features can be retained.

[0026] In summary, through the above S10 - S30, the asynchronous processing chain formed has produced a significant synergistic effect: the original timing information retained by S10 provides a real-time background for the feature extraction of S20, and the modality-specific features extracted by S20 provide a reliable comparison benchmark for the intelligent matching of S30.

[0027] This processing logic of "native asynchrony - feature optimization - dynamic matching" fundamentally solves the three major problems faced by the existing technologies: First, it avoids the loss of high-frequency features caused by forced synchronization. For example, it can completely retain the correlation features between the microsecond-level current mutation and the millisecond-level mechanical vibration wave at the moment of circuit breaker opening. Second, through modality-adaptive feature extraction, the signal-to-noise ratio of weak fault features is significantly improved, making the cross-modal weak correlation signals of early faults prominent. Thirdly, the dynamic time offset estimation enables the system to automatically adapt to the operating characteristics of different devices. For example, for circuit breaker mechanisms with different operating speeds, the system can accurately capture the correlation rules of their electromechanical characteristics.

[0028] S40. According to the maximum feature coincidence time interval, the original data streams of each modality are extended in the time dimension to generate a dynamic interaction interval, and the length of the interval is 1.2 - 1.5 times that of the original time window. Then, cross - validation is performed on the extracted heterogeneous feature set within this interval. If the deviation directions of the features of at least two modalities are the same, they are marked as credible fusion features.

[0029] Specifically, based on the maximum feature coincidence interval, a window with a length 1.2 - 1.5 times that of the feature coincidence time interval is used to expand the window, providing a physical - level cross - validation space for heterogeneous data to ensure that key features are not missed due to minor time deviations. For example, when the overheating feature of a cable joint appears in an infrared image, the system does not mechanically search for current sensor readings at the same millisecond level, but detects whether there are electrical features such as an increase in the dielectric loss angle or acoustic features of partial discharge within the expanded interval. This design retains the causal temporal relationship of multimodal data in the real physical scenario. And cross - validation is performed on the extracted heterogeneous feature set within this interval. If the time difference between the current mutation and the hot spot appears within the expanded interval and their deviation directions are the same (such as a sudden increase in current accompanied by an expansion of the hot spot), they are marked as credible fusion features. If they are not the same (such as only a hot spot but no current anomaly), it may be a false alarm and further verification is required.

[0030] S50. When a certain modality fails to pass the cross - validation within the dynamic interaction interval, redundant feature extraction of other modalities within adjacent time windows is triggered, and the redundant features are logically combined with the current credible fusion features to generate a compensated integrated feature vector. If the compensated features still cannot pass the verification, the modality data is marked as a low - confidence data source, and the time - window matching process of this modality is skipped in subsequent integration.

[0031] Specifically, dynamic processing of data quality fluctuations can avoid the influence of low - quality data on the fusion result. When a certain modality of data fails, the system does not simply discard or down - weight it, but performs logical reconstruction through associated features of adjacent windows. Suppose that during a certain monitoring, the audio sensor fails due to electromagnetic interference, resulting in the audio features not passing the cross - validation. At this time, the system will trigger redundant feature extraction, such as extracting historical audio features from adjacent time windows for logical combination to try to recover valid information. If it still cannot pass the verification after compensation, it is determined that this modality is a low - confidence data source, and its matching process is skipped in subsequent integration to prevent incorrect data from affecting the overall diagnostic result.

[0032] In summary, through the above S40 and S50, at the basic theory level, a "native asynchronous - feature - driven" multimodal integration framework is established for the first time, breaking through the theoretical limitations of traditional synchronous assumptions; at the technical performance level, the dynamic interaction interval design enables the system to capture cross - modal transient features that cannot be recognized by traditional methods; at the engineering applicability level, through the asynchronous compensation mechanism, the system can reduce the misdiagnosis rate under common on - site problems such as temporary sensor failure and electromagnetic interference. In scenarios such as transformer partial discharge monitoring and cable overheating warning, it can more accurately capture the coupling features of multimodal data and improve the accuracy of fault diagnosis.

[0033] Specifically, in S20, each modal feature extraction method will directly determine the reliability of the feature input in subsequent multimodal fusion and will affect the judgment of the entire data dynamic integration. When extracting each modal feature, it is necessary to improve the accuracy of its extraction. Therefore, this embodiment further discloses the extraction processes of the mutation gradient of the sensor numerical value, the hot - spot contour area of the infrared image, and the energy peak value in a specific frequency band of the audio waveform.

[0034] Specifically, in order to achieve the accuracy of extracting the mutation gradient feature of the sensor numerical value, in this embodiment, a progressive - processing method is adopted. Through dynamic baseline update and comprehensive calculation of the mutation gradient, high - anti - interference anomaly detection is realized. The extraction process includes: Perform three - level progressive filtering on the original sensor data stream. At the first level, first eliminate the instantaneous pulse noise caused by electromagnetic interference, then smooth the periodic fluctuations caused by mechanical vibration, and finally retain the real signal whose amplitude exceeds the baseline value. Through hierarchical filtering, while retaining real fault signals (such as winding overheating), environmental noise is suppressed to the greatest extent.

[0035] Automatically update the baseline value every once in a while. Use the sliding median filtering method to establish upper and lower threshold bands. For example, when the sensor detects temperature, it is necessary to adapt to the diurnal temperature difference or seasonal changes and adjust its baseline value. When the data points break through the threshold band continuously for multiple times, mutation detection is triggered to prevent false alarms caused by occasional interference.

[0036] Within the time window when a mutation is detected, calculate the maximum slope value of the signal change rate, and at the same time record the cumulative time during which this slope continuously exceeds the critical value. Multiply the maximum slope value by the cumulative time to obtain the mutation gradient feature value. Combine the mutation intensity and duration to avoid misjudging short - term interference as a fault (such as instantaneous load fluctuations), and at the same time capture slowly developing hidden dangers (such as progressive temperature rise caused by insulation aging).

[0037] Specifically, in order to achieve the accuracy of extracting the hot - spot contour area feature of the infrared image, in this embodiment, combined with the material heat - conduction characteristics and multi - frame verification, the detection rate of tiny hot spots is improved. The extraction process includes: Segment the device body area for each frame of infrared image to exclude background interference (sunlight reflection, background heat source interference), and then perform horizontal and vertical double scanning on the device surface temperature matrix to mark the temperature mutation areas, so as to avoid misjudging environmental thermal noise (such as uneven sunlight) as device heating.

[0038] Based on the thermal conductivity characteristics of the device material, when the temperature difference between adjacent pixels is detected to exceed the normal heat transfer limit of the material, the segmentation threshold is automatically lowered, which can further narrow the separation interval to ensure that small hot spots are not missed.

[0039] Perform multi-frame continuity inspection on the initially identified hot spot areas, and only retain the areas where the area expansion speed is within the preset growth interval in multiple consecutive frames to exclude instantaneous thermal interference (such as short-term thermal reflection caused by a bird flying by), and distinguish real overheating from instantaneous noise through time series verification.

[0040] Take the circumscribed rectangle of the verified hot spot area, and use the ratio of its area to the area of the reference area on the device surface as the characteristic output value, and perform normalization processing to adapt to the general judgment of devices of different sizes.

[0041] Specifically, in order to achieve the accuracy of extracting the energy peak value in a specific frequency band of the audio waveform, in this embodiment, through base noise elimination and three-layer frequency band focusing, local features are accurately captured, and the extraction process includes: Collect 30 seconds of ambient sound at the initial stage of device startup to automatically generate a base frequency spectrum template containing ambient noise and electromagnetic noise, with the purpose of excluding the interference of these fixed frequency bands in subsequent monitoring to avoid misjudging normal noise (such as the running sound of an air conditioner) as a discharge signal.

[0042] Real-time monitor the pulse bands that suddenly appear in the audio stream. These bands need to simultaneously meet the conditions: the duration is shorter than the device vibration period, and the main frequency deviates from the core frequency band of the base template, excluding the factors of device self-interference, and screening possible pulse signals.

[0043] Perform three-layer fine division on the suspicious bands. First, detect the sudden increase in energy in the full frequency band, then locate the resonance points in the sub-frequency bands, and finally extract the frequency range with the most concentrated energy as the characteristic frequency band, gradually focusing on the real mutation characteristics to avoid the influence of broadband interference on the judgment.

[0044] Only when the energy value of this frequency band continuously exceeds the historical average value within a certain period of time and coincides with the device mechanical action time, it is recorded as an effective feature to ensure that the detected signal is strongly related to the device state and exclude random interference.

[0045] Specifically, the characteristic manifestations of power equipment are strongly correlated with the operating conditions (e.g., the oil temperature of a transformer changes with the load). Therefore, in the process of calculating the deviation coefficients of each characteristic under the historical normal state, a physical mapping relationship between the public and the characteristic needs to be established. In this embodiment, during the normal operation stage of the equipment, historical characteristic data of each mode is collected in advance, and a hierarchical statistical model is established according to different operating conditions (such as light load, full load, start-stop transient). The distribution interval and typical fluctuation mode of the characteristic values are recorded for each level, which can avoid misjudging the operating condition switch as a fault.

[0046] Match the characteristic values of the extracted current time window to the corresponding operating condition levels, and calculate the percentage of its deviation from the median value of this level as the instantaneous deviation degree. The essence of equipment abnormality is the deviation from the normal behavior mode under its current operating condition, rather than simply exceeding a fixed threshold. In this embodiment, using percentage quantification is more in line with the physical characteristics of the fault.

[0047] For continuously deviating characteristics, apply a progressive coefficient according to the deviation duration, set time nodes, increase linearly before the time node to quickly capture sudden faults, and change to square root growth after the time node to avoid false alarms triggered by short-term fluctuations, while continuously monitoring slow deterioration.

[0048] Multiply the instantaneous deviation degree by the progressive coefficient, and then superimpose the collaborative deviation degrees of other modes. Power equipment failures usually manifest as multi-physical quantity coupling anomalies (such as partial discharge simultaneously triggering temperature rise, ultrasonic and electromagnetic signals). Multi-modal consistency is a strong feature of faults. Excessive multi-modal collaborative verification can significantly improve reliability, and finally output a standardized deviation coefficient between 0 and 10.

[0049] Specifically, in the process of time alignment of multi-modal data of power equipment, the accuracy of sliding window matching directly determines the correct alignment of key characteristics of each mode on the time axis. If the matching accuracy is insufficient, it may lead to time misalignment of sound, heat, and electrical signals, and then cause two serious consequences: one is to wrongly associate irrelevant events; the other is to miss real coupling characteristics. To address this key issue, this embodiment proposes a set of hierarchical and progressive precise matching mechanisms, from benchmark anchor point positioning to multi-level matching verification, and finally realizes time alignment with credible physical meaning.

[0050] First, establish a reference time anchor point centered on the sensor value mutation point. This is because the change of electrical parameters of power equipment is often the most direct manifestation of faults and has the highest time reference value. Expand 50 milliseconds forward and backward to form a reference interval, which not only covers the duration of typical fault signals but also leaves a buffer space for subsequent matching.

[0051] In the first-level rough matching stage, high-frequency modal data is compressed into a simplified sequence at 10-millisecond intervals for fast alignment. This downsampling process significantly reduces the computational complexity while retaining key time features. Since the acoustic features of power equipment failures usually last for dozens of milliseconds or more, a 10-millisecond resolution is sufficient to capture their start and end times. After rough matching, the system performs millisecond-level fine-tuning within a range of ±20 milliseconds. This design targets the small drift problems of different sensor hardware clocks. In the fine matching stage, sub-sampling-level alignment is achieved through interpolation algorithms to ensure that the physical causal relationship between the occurrence time of the hot spot and the current mutation is not masked by clock errors.

[0052] During the matching process, it is specifically required that the occurrence time of the hot spot in the image modality must be included within the duration of the audio pulse. This constraint stems from the physical propagation law of power equipment failures. When partial discharge occurs inside the equipment, the acoustic signal propagates instantaneously in the form of mechanical waves, while the accumulation of heat takes a longer time. Therefore, a reasonable physical scenario should be that the start time of the acoustic signal is earlier than the occurrence of the hot spot, and the hot spot duration should cover part of the acoustic signal. If it is detected that the hot spot appears earlier than the acoustic signal, there is likely a matching error. Also, it is set that the time difference between each modality must be less than the upper limit of the physical response delay of the equipment. For example, the thermal inertia of transformer oil temperature to winding overheating is usually at the second level. If it is claimed that the time difference between "current mutation" and "temperature rise" is only 10 milliseconds, it obviously violates the principles of thermodynamics. This verification based on physical laws fundamentally excludes unreasonable time matching combinations.

[0053] When the time difference between the modal features of each modality exceeds the allowable error, the matching result of the sensor and the infrared image is preferentially adopted. This is based on the different reliability characteristics of the two types of data. Sensor values (such as voltage and current) directly reflect the changes in the electrical state, and the time resolution can reach the microsecond level. Although the infrared thermal image is limited by the frame rate (usually 30 - 100Hz), the heat diffusion process itself has inertia and the influence of time jitter is small. In contrast, the audio signal is easily affected by environmental noise (such as transient sounds generated by switch operations), and reflection superposition may occur on the propagation path, resulting in a decrease in time positioning accuracy. At this time, the audio data is converted for auxiliary verification purposes, which not only retains its verification value for fault characteristics (such as the frequency domain characteristics of discharge ultrasonic waves), but also avoids its negative impact on the time alignment accuracy.

[0054] Specifically, the process of cross-verifying the extracted heterogeneous feature set includes: First, in line with the actual usage scenarios, a clear mapping relationship between various modal features and the physical state of the equipment is established. This is the theoretical basis for cross-validation. Specifically: the mutation gradient of sensor values is correlated with the change in the contact state of conductive components; the contour area of the infrared hot spot corresponds to the local temperature rise of the equipment; and the peak audio energy is related to the vibration damage of the mechanical structure. This mapping is not a simple data correlation but a physical modeling based on the fault mechanism of power equipment, endowing the features of different modalities with comparable physical meanings.

[0055] When the system detects that the physical quantity indicators of at least two modalities simultaneously exceed the safety threshold and change in the same direction, it will be marked as a credible feature. This design makes full use of the coupling characteristics of multiple physical quantities. For example, when the transformer winding is deformed, the high-frequency energy of the vibration signal will increase (mechanical deformation), and at the same time, the change in winding resistance will cause an increase in current harmonics (electrical characteristics). The two modal features change synchronously in the time domain and in the same direction, thus constituting a strong credible feature.

[0056] This verification method based on physical consistency can effectively eliminate false alarms in a single modality. For example, environmental temperature fluctuations may cause abnormal infrared hot spots, but if there are no corresponding changes in electrical or vibration characteristics, the system will not misjudge it as an equipment failure.

[0057] For the currently marked credible features, the system will further conduct a reverse verification of the progressive abnormal signs in historical data. This is a mechanism for differentiating between sudden and progressive faults. Many power equipment faults do not occur suddenly but go through a deterioration process. For example, when a cable joint is loose, it will first show intermittent temperature fluctuations and eventually develop into continuous overheating. When the system detects a sudden feature (such as a sudden temperature rise), by retrospectively analyzing the trend changes (such as the gradually increasing amplitude of temperature fluctuations) in a previous period of time, the continuity of the fault evolution can be confirmed.

[0058] If such progressive signs exist, a secondary verification process will be initiated, including extending the observation window, increasing the detection of auxiliary features, etc., to finally determine it as a real equipment fault rather than an instantaneous interference. This mechanism is particularly suitable for identifying slowly developing defects such as insulation aging and mechanical fatigue, making up for the deficiency of traditional threshold detection methods being insensitive to progressive faults.

[0059] Specifically, the cross-validation process of this embodiment realizes three key breakthroughs through a three-layer progressive analysis of "physical mapping establishment → multi-modal consistency verification → historical trend backtracking": First, it elevates the feature fusion at the data level to the correlation verification at the physical law level, making the diagnostic results have a clear interpretation of the equipment state; second, through the mutual verification of multiple modalities, it significantly reduces the false alarm rate caused by single-sensor failure or environmental interference; third, the time-domain tracking of the fault development process realizes the accurate differentiation between sudden and progressive faults.

[0060] Specifically, when a certain modality fails to pass the cross-verification within the dynamic interaction interval, the process of triggering the extraction of redundant features of other modalities within adjacent time windows includes: First, record the timestamp deviation value and the feature deviation direction of this modality. These two parameters have important physical meanings: the timestamp deviation reflects the sensor clock synchronization error or signal propagation delay, while the feature deviation direction implies possible fault types. The design of freezing the original feature output of the current window avoids the continuous propagation of incorrect data and provides a reference benchmark for subsequent redundancy analysis.

[0061] In the redundant feature extraction stage, expand 1 acquisition cycle forward and backward centered on the reference time axis. The setting of this time range takes into account the inertial characteristics of the fault characteristics of power equipment - for example, the acoustic emission signals generated by partial discharges usually last for several acquisition cycles, and the heat diffusion process has a significant time continuity.

[0062] For the sensor modality, extract the gradient change rate of the forward window and the numerical fluctuation amplitude of the backward window. These two features respectively reflect the dynamic characteristics and steady-state characteristics of fault development; The morphological similarity and area change trend of the infrared modality capture the spatio-temporal laws of hot spot evolution; The energy attenuation coefficient and frequency shift ratio of the audio modality can distinguish real fault signals (with specific attenuation and frequency shift patterns) from environmental noise.

[0063] This cross-time-window redundant feature extraction essentially reconstructs the missing information integrity through the time-domain correlation of fault signals.

[0064] Specifically, the process of generating the compensated integrated feature vector by logically combining the redundant features with the current credible fusion features includes: Take the verified primary features as the core benchmark, considering that they have passed the multi-modal physical consistency test and have the highest reliability; the secondary features in the redundant features (with a high degree of matching with historical data) reflect the repetitive or periodic characteristics of faults. For example, bearing damage often shows a periodic increase in vibration signals; the tertiary features contain more uncertainties but may still contain valuable fault information.

[0065] In the hierarchical weighting strategy, the primary features obtaining the maximum weight conforms to the principle of "verified information first", the second-highest weight of the secondary features reflects respect for historical laws, and the lower weight of the tertiary features realizes the cautious utilization of uncertain information.

[0066] This weight assignment is not a simple empirical setting, but is based on the principle of belief propagation in information theory: accurate information that is known should dominate in decision-making, while speculative information should participate in decision-making with limitations. The final weighted summation process actually constructs a dynamic feature credibility field. Even if some modalities fail, the system can still form a reasonable integrated judgment based on the remaining reliable information.

[0067] Referring to Figure 2 as shown, to implement the above dynamic integration method for multi-modal power information data, the present invention also discloses a dynamic integration system for multi-modal power information data, including: A multi-source heterogeneous data acquisition module for real-time acquiring data streams of at least two heterogeneous modalities in the power system, including device sensor numerical sequences, device surface infrared image sequences, and environmental audio waveforms. Each modality data stream is acquired with an independent clock cycle and is not pre-synchronized. A dynamic feature extraction module for extracting heterogeneous feature sets within the current time window for each modality data stream, including the mutation gradient of sensor values, the hot spot contour area in infrared images, and the energy peak value in a specific frequency band of the audio waveform, and calculating the deviation coefficient of each feature in the historical normal state. A time axis alignment module for using the sensor numerical sequence as the reference time axis, and performing a sliding window match between the peak moments of the feature deviation coefficients of other modality data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference. A cross-validation module for expanding each modality's original data stream in the time dimension to generate a dynamic interaction interval according to the maximum feature coincidence time interval, where the interval length is 1.2 to 1.5 times the original time window, and performing cross-validation on the extracted heterogeneous feature sets within this interval. If the feature deviation directions of at least two modalities are consistent, they are marked as credible fusion features. A redundancy compensation module for triggering the extraction of redundant features of other modalities within adjacent time windows when a certain modality fails to pass cross-validation within the dynamic interaction interval, logically combining the redundant features with the current credible fusion features to generate a compensated integrated feature vector. When the compensated features still cannot pass the validation, the modality data is marked as a low-confidence data source, and its time window matching process is skipped in subsequent integrations.

[0068] Specifically, the dynamic integration system for multi-modal power information data described in this embodiment can execute the above method. The specific execution process can refer to the description of the method and will not be repeated here.

[0069] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0070] Obviously, the above embodiments are merely examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A dynamic integration method for multi-modal power information data, characterized in that: Including the following steps: Obtain the data streams of at least two heterogeneous modalities in the power system in real time, including device sensor numerical sequences, device surface infrared image sequences, and environmental audio waveforms. Each modality's data stream is collected with an independent clock cycle and is not pre-synchronized; For each modality's data stream, extract the heterogeneous feature set within the current time window, including the mutation gradient of sensor values, the hot spot contour area in the infrared image, and the energy peak value in a specific frequency band of the audio waveform, and calculate the deviation coefficient of each feature under the historical normal state; Taking the sensor numerical sequence as the reference time axis, perform a sliding window match between the peak moment of the feature deviation coefficient of other modality data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference; According to the maximum feature coincidence time interval, expand the original data stream of each modality in the time dimension to generate a dynamic interaction interval, and the interval length is 1.2 - 1.5 times that of the original time window. Cross-validate the extracted heterogeneous feature set within this interval. If the deviation directions of the features of at least two modalities are consistent, mark them as credible fusion features; When a certain modality fails the cross-validation within the dynamic interaction interval, trigger the extraction of redundant features of other modalities within adjacent time windows, and logically combine the redundant features with the current credible fusion features to generate a compensated integrated feature vector; If the compensated features still cannot pass the verification, mark the data of this modality as a low-confidence data source and skip its time window matching process in subsequent integration.

2. The dynamic integration method for multi-modal power information data according to claim 1, wherein: The process of extracting the mutation gradient of the sensor values includes: Perform three-level progressive filtering on the original sensor data stream. First, eliminate the instantaneous pulse noise caused by electromagnetic interference, then smooth the periodic fluctuations caused by mechanical vibration, and finally retain the real signal whose amplitude exceeds the baseline value; Automatically update the baseline value every once in a while, use the sliding median filtering method to establish the upper and lower threshold bands, and trigger mutation detection when the data points break through the threshold band continuously for multiple times; Within the time window where a mutation is detected, calculate the maximum slope value of the signal change rate, and at the same time record the cumulative time during which this slope continuously exceeds the critical value. Multiply the maximum slope value by the cumulative time to obtain the mutation gradient feature value.

3. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: The process of extracting the hot spot contour area of the infrared image includes: Segment the device body area for each frame of infrared image to exclude background interference, and then perform horizontal and vertical double scans on the device surface temperature matrix to mark the temperature mutation areas; Based on the heat conduction characteristics of the device material, when the temperature difference between adjacent pixels is detected to exceed the normal heat transfer limit of this material, automatically lower the segmentation threshold to ensure that small hot spots are not missed; Perform multi-frame continuity inspection on the initially identified hot spot areas, and only retain the areas whose area expansion speed is within the preset growth interval in multiple consecutive frames; Take the circumscribed rectangle of the verified hot spot area, and use the ratio of its area to the area of the device surface reference area as the feature output value.

4. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: The process of extracting the energy peak value in a specific frequency band of the audio waveform includes: Collect 30 seconds of ambient sound at the initial stage of device startup, and automatically generate a base spectrum template including ambient noise and electromagnetic noise; Real-time monitor the suddenly emerging pulse bands in the audio stream, and these bands need to meet both of the following conditions: the duration is shorter than the device vibration period, and the main frequency deviates from the core frequency band of the base template; Conduct a three-layer fine division of the suspicious bands. First, detect the sudden increase in the energy of the full frequency band, then locate the resonance points of the sub-frequency bands, and finally extract the frequency range with the most concentrated energy aggregation as the characteristic frequency band; Only when the energy value of this frequency band continuously exceeds the historical average value within a certain period of time and coincides with the mechanical action time of the device, it is recorded as an effective feature.

5. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: The process of calculating the deviation coefficients of each feature in the historical normal state includes: Collect the historical feature data of each mode during the normal operation of the device, establish a hierarchical statistical model according to different working conditions, and record the distribution interval of the feature values and the typical fluctuation patterns for each level; Match the feature values of the currently extracted time window to the corresponding working condition levels, and calculate the percentage of its deviation from the median value of this level as the instantaneous deviation degree; For the continuously deviating features, apply a progressive coefficient according to the deviation duration, set time nodes, with linear growth before the time nodes and square root growth after the time nodes; Multiply the instantaneous deviation degree by the progressive coefficient, and then superimpose the collaborative deviation degrees of other modes, and finally output the standardized deviation coefficient between 0 and 10.

6. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: The sliding window matching ensures accuracy through the following methods: Centering on the sensor value mutation point, expand 50 milliseconds forward and backward as the reference time anchor point; The first-level rough matching compresses the data of other modes into a simplified sequence at 10-millisecond intervals for fast alignment, and the second-level fine matching performs millisecond-level fine-tuning within the range of ±20 milliseconds of the rough matching result; It is required that the occurrence time of the hot spot in the image mode must be included in the duration of the audio pulse, and the time differences between both of them and the sensor mutation are less than the upper limit of the device's physical response delay; When the time differences of the features of each mode exceed the system allowable error, give priority to the matching result of the sensor and the infrared image, and the audio data is converted for auxiliary verification purposes.

7. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: The process of cross-verifying the extracted heterogeneous feature set includes: Establish the mapping relationship between the features of each mode and the physical state of the device, including: the mutation gradient of the sensor value is associated with the change of the contact state of the conductive components, the contour area of the infrared hot spot is associated with the local temperature rise of the device, and the audio energy peak is associated with the vibration damage of the mechanical structure; It is required that the physical quantity indicators of at least two modes exceed their safety thresholds at the same time and the change directions are the same, that is, they are marked as credible features; For the currently marked credible features, it is necessary to reverse-check whether there are progressive abnormal signs in a previous period of time. If there are progressive abnormal signs, start the secondary verification and determine it as a sudden device failure.

8. The dynamic integration method for multi-modal power information data according to claim 1, characterized in that: When a certain mode fails to pass the cross-verification within the dynamic interaction interval, the process of triggering the extraction of redundant features of other modes within the adjacent time window includes: Identify the data stream of the mode that fails to pass the verification, record its timestamp deviation value and the feature deviation direction, and at the same time freeze the original feature output of this mode in the current window; Centering on the reference time axis, expand 1 acquisition cycle forward and backward, and extract the following redundant features from other verified modes: For the sensor mode: extract the gradient change rate of the forward window and the numerical fluctuation amplitude of the backward window; For the infrared modality: Extract the morphological similarity of the hot spot contours in adjacent windows and the trend of area change; For the audio modality: Extract the energy attenuation coefficient and frequency shift ratio in the same frequency band of the front and rear windows.

9. The dynamic integration method for multi-modal power information data according to claim 1, wherein: The process of logically combining redundant features with the current trusted fusion features to generate a compensated integrated feature vector includes: Mark the verified trusted fusion features as first-level features, mark the redundant features with a high degree of matching with historical data as second-level features, and mark the remaining redundant features as third-level features; Perform weighted summation on the first-level features, second-level features, and third-level features according to preset fixed weights, where: the weight of the first-level features is greater than that of the second-level features, and the weight of the second-level features is greater than that of the third-level features, and output the weighted result as the final integrated feature vector.

10. A dynamic integration system for multi-modal power information data, characterized in that: Including: A multi-source heterogeneous data acquisition module for real-time acquiring data streams of at least two heterogeneous modalities in the power system, including device sensor numerical sequences, infrared image sequences on the device surface, and environmental audio waveforms. Each modality data stream is acquired with an independent clock cycle and is not pre-synchronized; A dynamic feature extraction module for extracting a heterogeneous feature set within the current time window for each modality data stream, including the mutation gradient of sensor values, the area of hot spot contours in infrared images, and the energy peak in a specific frequency band of audio waveforms, and calculating the deviation coefficient of each feature under the historical normal state; A time axis alignment module for using the sensor numerical sequence as the reference time axis, performing sliding window matching between the peak moments of the feature deviation coefficients of other modality data and the reference time axis to determine the maximum feature coincidence time interval of each modality relative to the reference; A cross-validation module for expanding the original data streams of each modality in the time dimension according to the maximum feature coincidence time interval to generate a dynamic interaction interval, the interval length being 1.2 - 1.5 times that of the original time window, and performing cross-validation on the extracted heterogeneous feature set within this interval. If the feature deviation directions of at least two modalities are the same, they are marked as trusted fusion features; A redundancy compensation module for triggering the extraction of redundant features of other modalities within adjacent time windows when a certain modality fails to pass cross-validation within the dynamic interaction interval, logically combining the redundant features with the current trusted fusion features to generate a compensated integrated feature vector. When the compensated features still cannot pass the verification, mark the data of this modality as a low-confidence data source and skip its time window matching process in subsequent integrations.

Images