Power transmission and transformation equipment abnormal intelligent diagnosis method and device based on adaptive multi-modal weight

Abstract

This invention belongs to the technical field of computer vision, specifically relating to a method and device for intelligent diagnosis of anomalies in power transmission and transformation equipment based on adaptive multimodal weights. The method includes: acquiring multimodal data and preprocessing each modal data to obtain a signal quality vector for each modal data; calculating the weights corresponding to the signal quality vectors of each modal data; extracting and fusing the feature vectors of each modal data at the current time to obtain fused features at the current time, using these features as input to an anomaly detection network to obtain anomaly identification results; acquiring a fused feature sequence of a given historical window length and using it as input to a prediction model to predict a comprehensive anomaly confidence sequence for future times, comparing it with a preset threshold, and triggering an early warning if the result is greater than the preset threshold. This invention solves the technical problems of fixed weights, poor environmental adaptability, and lack of early warning in existing methods.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision, specifically relating to a method and device for intelligent diagnosis of anomalies in power transmission and transformation equipment based on adaptive multimodal weights. Background Technology

[0002] With the intelligent development of power systems, the monitoring of the operational status of power transmission and transformation equipment is gradually transitioning from manual inspection to automated and intelligent diagnosis. In the condition monitoring and anomaly diagnosis of power transmission and transformation equipment, commonly used methods mainly rely on single-modal or simply fused multi-modal data. For example: Single-modal video monitoring: Video is captured from the equipment by installing a visible light camera, and image recognition or video analysis technology is used to detect abnormalities such as appearance defects and damage to equipment components; Simple fusion of multimodal data: Some systems directly use fixed weights to perform modal fusion based on data collected by visible light, infrared thermal imaging and audio sensors, or rely on human experience to adjust the modal weights to obtain comprehensive anomaly detection results.

[0003] Referring to Chinese patent document CN120147316A, a method, apparatus, device, and storage medium for detecting thermal defects in transmission lines are disclosed. The method includes: acquiring infrared and visible light images of the transmission line; extracting features from the infrared and visible light images to obtain infrared image features and visible light image features; performing high-heat target detection on the infrared image to obtain a high-heat target saliency weight; calculating adaptive weights for the infrared and visible light images based on the high-heat target saliency weights; performing weighted fusion of the infrared and visible light image features based on the infrared and visible light image adaptive weights to obtain a fused feature image; and using a deformable attention detection model to detect thermal defect targets on the fused feature image to obtain a defect target detection image.

[0004] However, existing technologies have the following shortcomings in multimodal fusion: Fixed modal weights have poor adaptability: Under environmental changes such as illumination, weather, and noise, the quality of data from different modalities will fluctuate significantly, but fixed weights cannot be dynamically adjusted according to the actual quality, which can easily lead to unstable detection results. Low efficiency in modal data utilization: Most fusion methods still participate in fusion calculations when modal data has low quality or even severe interference, which increases the risk of misjudgment; Lack of trend prediction capability: Existing technologies focus more on the detection and localization of current anomalies, but lack the ability to predict the evolution trend of anomalies and thus cannot issue early warnings.

[0005] Based on this, the present invention designs an intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights to solve the technical problems of fixed weights, poor environmental adaptability, and lack of early warning in existing methods. Summary of the Invention

[0006] The present invention aims to overcome at least one of the defects of the prior art and provide an intelligent diagnosis method for power transmission and transformation equipment that can adaptively adjust multimodal weights, enhance the robustness of anomaly detection, and have the ability to predict the anomaly evolution trend, so as to solve the technical problems of fixed weights, poor environmental adaptability, and lack of early warning in the existing methods.

[0007] The present invention also discloses an apparatus loaded with an intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights.

[0008] The detailed technical solution of this invention is as follows: A method for intelligent diagnosis of anomalies in power transmission and transformation equipment based on adaptive multimodal weights, the method comprising: S1. Acquire multimodal data and preprocess each modal data to obtain the signal quality vector of each modal data; wherein, the multimodal data includes visible light modal data, infrared modal data and audio modal data with consistent time reference; S2. Based on the signal quality vector of each modal data, calculate the weight of each modal data accordingly; S3. Extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time. Use it as the input of the anomaly detection network to obtain the anomaly recognition result. S4. Obtain the fusion feature sequence of a given historical window length and use it as the input of the prediction model to predict the comprehensive anomaly confidence sequence for the next K time moments. Compare the comprehensive anomaly confidence at the next time moment with a preset threshold. If it is greater than the preset threshold, trigger an early warning.

[0009] According to a preferred embodiment of the present invention, in step S1, the multimodal data includes visible light modal data, infrared modal data, and audio modal data with a consistent time reference, specifically: For visible light image frames Infrared image frames Audio frame fragments ,in, Indicates the acquisition timestamps of visible light image frames and infrared image frames. Represents the timestamp of audio frame segment acquisition, and defines the alignment window. , Indicate the width of the time alignment window and construct the synchronization frame sequence:

[0010] In formula (1): Indicates a sequence of synchronization frames; This indicates any sampling point within the aligned window after interpolation.

[0011] According to a preferred embodiment of the present invention, in step S1, the signal quality vector of the visible light modal data The calculation is as follows:

[0012] In formula (2): This represents the average brightness of a visible light image frame; Represents a visible light image frame; Indicates the number of visible light image frames; This represents the luminance variance of a visible light image frame; express The signal quality vector of the visible light image frame at time 1; This represents a function for calculating the signal quality vector of a visible light image frame; Indicates a custom comparison threshold; Indicates a custom scale parameter; The signal quality vector of the infrared modal data The calculation is as follows:

[0013] In formula (3): express The signal quality vector of the infrared image frame at time 1; This function represents the calculation of the signal quality vector of an infrared image frame. Indicates thermal contrast, and , Indicates the maximum temperature. Indicates the background temperature; The stage function representing the value of thermal radiation intensity will Values ​​are limited to a preset minimum value of 0 and a maximum value of 1; values ​​outside this range are forcibly set to boundary values. The signal quality vector of the audio modal data The calculation is as follows:

[0014] In equation (4): Indicates the signal-to-noise ratio. This represents the average power of the signal; This represents the average power of the noise; Indicates spectral flatness; Indicates the number of audio frame sampling points; This represents the amplitude of the k-th sampling point after the audio frame signal has undergone Fourier transform; express The signal quality vector of the audio frame segment at a given time; This represents a function that calculates the signal quality vector of an audio frame segment.

[0015] According to a preferred embodiment of the present invention, S2 specifically includes: The signal quality vector of each modality data is mapped using the lightweight neural network MobileViT. to the corresponding raw score :

[0016] In equation (6): The parameter is MobileViT neural network; Represents a three-dimensional real vector space; Corresponding to visible light modal data Infrared modal data and audio modal data ; Alignment and softmax normalization yield the initial weights for each modality. :

[0017] In equation (7): A custom temperature coefficient is used to control the weight distribution; The target weights for each modality of data after smoothing are obtained using the exponential moving average (EMA). :

[0018] In equation (8): Customize the historical memory coefficient.

[0019] According to a preferred embodiment of the present invention, in step S3, the feature vectors of each modality data at the current time are extracted and fused to obtain the fused features at the current time, specifically including: The Video Transformer feature extractor is used to extract spatiotemporal feature vectors from visible light modal data and infrared modal data, respectively, to obtain the feature vectors of visible light modal data at the current time. And the feature vector of the infrared modal data at the current moment. , , Both represent dimensions; Audio feature vectors of audio modal data are extracted from frequencies using the AudioTransformer feature extractor. , Indicates dimension; The feature vectors of data from various modalities are fused using a cross-modal attention approach to obtain the fused features at the current time step. : .

[0020] According to a preferred embodiment of the present invention, in step S3, the fusion feature at the current moment is... Inputting the anomaly detection network will output the overall anomaly confidence score. , Probability vector of anomaly type and spatial positioning Spatial positioning Bounding box information containing visible light modal data and infrared modal data; The comprehensive anomaly confidence level Defined as a weighted combination of the scores from each modality:

[0021] In formula (10): The output of the regression head represents the anomaly confidence score for each mode in the anomaly detection network.

[0022] According to a preferred embodiment of the present invention, in step S4, a fused feature sequence with a given historical window length of L is... The prediction model uses the TimeSformer model, and the prediction function is defined as follows: Among them, the comprehensive anomaly confidence sequence Then we have:

[0023] Training loss of the TimeSformer model It consists of a regression term and an uncertainty term:

[0024] In equation (12): For the regression term, This is an uncertain term; Indicates the time after t time; Indicates the parameters of the TimeSformer model; To balance the weights.

[0025] In another aspect of the present invention, an apparatus is provided for implementing an intelligent diagnosis method for power transmission and transformation equipment anomalies based on adaptive multimodal weights, the apparatus comprising: The data acquisition module is used to acquire multimodal data and preprocess each modal data to obtain the signal quality vector of each modal data; wherein, the multimodal data includes visible light modal data, infrared modal data and audio modal data with consistent time reference; The weight calculation module is used to calculate the weight of each mode data based on the signal quality vector of each mode data. The anomaly detection module is used to extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time, which is used as the input of the anomaly detection network to obtain the anomaly recognition result. The anomaly prediction module is used to obtain a fusion feature sequence of a given historical window length and use it as input to the prediction model to predict the comprehensive anomaly confidence sequence at the next K time points. The comprehensive anomaly confidence at the next time point is compared with a preset threshold. If it is greater than the preset threshold, an early warning is triggered.

[0026] In another aspect of the invention, an electronic device is also provided, comprising: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights as described above.

[0027] In another aspect of the invention, a machine-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform the intelligent diagnostic method for power transmission and transformation equipment anomalies based on adaptive multimodal weights as described above.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention provides an intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights. The method calculates the weight value based on the signal quality of the modal data. It has the characteristics of strong environmental adaptability and can dynamically adjust the weight according to the quality of the real-time modal data, thereby improving the detection accuracy in environments with changes in illumination, weather interference, and noise. At the same time, it will automatically reduce the weight for low-quality or interfered modal data, thereby improving the robustness of the prediction model and reducing the probability of false detection and missed detection.

[0029] (2) The present invention predicts trends based on the fusion feature sequence of historical windows, and can issue an alarm several hours or even earlier before the fault occurs, thus buying time for maintenance arrangements. Brief Description of the Drawings

[0030] Figure 1 is a flowchart of the method for intelligent diagnosis of abnormalities in power transmission and transformation equipment based on adaptive multi-modal weights according to the present invention.

[0031] Figure 2 is a schematic diagram of the adaptive weight dynamic adjustment mechanism in Embodiment 1 of the present invention.

[0032] Figure 3 is a schematic diagram of the cross-modal attention fusion and anomaly detection structure in Embodiment 1 of the present invention.

[0033] Figure 4 is a schematic diagram of the cross-modal attention fusion and anomaly prediction structure in Embodiment 1 of the present invention. Detailed Embodiments

[0034] The present invention will be further described below in conjunction with the drawings and embodiments.

[0035] It should be noted that the following detailed description is exemplary and intended to provide further illustration of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0037] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0038] Embodiment 1 Refer Figure 1 , this embodiment provides a method for intelligent diagnosis of abnormalities in power transmission and transformation equipment based on adaptive multi-modal weights, and the method includes: S1. Obtain multi-modal data and preprocess each modal data therein to obtain a signal quality vector of each modal data; wherein, the multi-modal data includes visible light modal data, infrared modal data, and audio modal data with consistent time bases.

[0039] In this embodiment, the obtaining of multi-modal data is specifically that corresponding acquisition devices are respectively installed on substations, transmission lines, and inspection drones, including visible light cameras for collecting visible light modal data, and the frame rate thereof is An infrared thermal imager used to acquire infrared modal data has a frame rate of And directional or omnidirectional microphones used to acquire audio modal data, with a sampling rate of It also provides location and timestamps using IMU / GPS.

[0040] All acquisition devices use a unified time base 't' for timestamping, such as GPS clocks or NTP protocol time synchronization. If sampling rate differences exist, they are aligned through interpolation / resampling. Specifically, for visible light image frames... Infrared image frames Audio frame fragments ,in, Indicates the acquisition timestamps for visible light image frames and infrared image frames; Represents the timestamp of audio frame segment acquisition; defines the alignment window. , This represents the width of the time alignment window, i.e., the time tolerance or synchronization tolerance range, the maximum cross-modal time deviation allowed by the system, and is used to construct the synchronization frame sequence:

[0041] In formula (1): Indicates a sequence of synchronization frames; This indicates any sampling point within the aligned window after interpolation.

[0042] The interpolation method can be linear / spline interpolation to supplement or drop frames.

[0043] Traditional preprocessing operations are performed on each modal data, and its signal quality vector is calculated simultaneously. These represent visible light, thermal imaging, and audio, respectively, as adaptive weight inputs.

[0044] Preprocessing of visible light modal data mainly includes distortion correction, noise reduction, and color normalization; its signal quality vector The calculation is as follows:

[0045] In formula (2): This represents the average brightness of a visible light image frame; Represents a visible light image frame; Indicates the number of visible light image frames; This represents the luminance variance of a visible light image frame; express The signal quality vector of the visible light image frame at time 1; This represents a function for calculating the signal quality vector of a visible light image frame; Indicates a custom comparison threshold; This indicates a custom scale parameter.

[0046] Preprocessing of infrared modal data mainly includes distortion correction and noise reduction; its signal quality vector The calculation is as follows:

[0047] In formula (3): express The signal quality vector of the infrared image frame at time 1; This function represents the calculation of the signal quality vector of an infrared image frame. Indicates thermal contrast, and , Indicates the maximum temperature. Indicates the background temperature; The stage function representing the value of thermal radiation intensity will Values ​​are limited to a preset minimum value of 0 and a maximum value of 1. Values ​​outside this range are forcibly set to boundary values.

[0048] Preprocessing of audio modal data mainly includes bandpass filtering and noise suppression to highlight acoustic features related to potential hazards in power equipment; its signal quality vector The calculation is as follows:

[0049] In equation (4): Indicates the signal-to-noise ratio. This represents the average power of the signal; This represents the average power of the noise; Indicates spectral flatness; Indicates the number of audio frame sampling points; This represents the amplitude of the k-th sampling point after the audio frame signal has undergone Fourier transform; express The signal quality vector of the audio frame segment at a given time; This represents a function that calculates the signal quality vector of an audio frame segment.

[0050] Therefore, concatenating the signal quality vectors of each modality data point yields the signal quality vector of the multimodal data. :

[0051] In equation (5): This indicates the matrix transpose.

[0052] S2. Based on the signal quality vector of each modal data, calculate the weight of each modal data accordingly.

[0053] In this embodiment, the dynamic weight adjustment mechanism is as follows: Figure 2 As shown, the dynamic weights of the multimodal data are calculated based on the signal quality vector from the previous step and then fused in the subsequent steps.

[0054] Specifically, the weights are calculated based on the signal quality vector corresponding to each modality of data, including: First, the signal quality vector of each modality data is mapped using the lightweight neural network MobileViT. to the corresponding raw score :

[0055] In equation (6): The parameter is MobileViT neural network; It represents a three-dimensional real vector space.

[0056] Next, align and normalize using softmax to obtain the corresponding initial weights. :

[0057] In equation (7): A custom temperature coefficient is used to control the weight distribution.

[0058] Finally, weight smoothing is performed to prevent transient fluctuations. The target weight for each modal data is obtained by using the exponential moving average (EMA) after smoothing. :

[0059] In equation (8): Customize the historical memory coefficient.

[0060] This allows for the suppression of low-quality modes, specifically, the reduction of signal quality vectors in visible light image frames during nighttime or backlighting conditions. The probability weight decreases, therefore its corresponding probability weight... Automatically suppressed, the system relies on both infrared and audio modal data; when encountering wind, rain, or high electromagnetic noise, the signal quality vector of the audio frame segment... The decrease will suppress its corresponding audio weight, avoiding misjudgment of discharge due to environmental noise; when visible light is blurred due to drone vibration, the brightness of the visible light image frame will be used as the metric. Reduce the signal quality vector of visible light image frames Increase the priority of thermal modes.

[0061] S3. Extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time. Use this feature as the input to the anomaly detection network to obtain the anomaly recognition result.

[0062] In this embodiment, both the visible light modal data and the infrared modal data are extracted using the Video Transformer feature extractor to obtain spatiotemporal feature vectors, thus obtaining the feature vectors of the visible light modal data at the current time. And the feature vector of the infrared modal data at the current moment. The audio modal data uses the AudioTransformer feature extractor to extract audio feature vectors from frequencies, i.e. . , , Indicates dimension.

[0063] The feature vectors of three modalities are fused using a cross-modal attention approach to obtain the fused features at the current time step. : .

[0064] This fusion feature This serves as the input to the anomaly detection network. The structure of the anomaly detection network is as follows: Figure 3 As shown, it includes a decoder part and a target head part. The target head part consists of a regression head for outputting anomaly confidence, a classification head for outputting anomaly type, and a detection head for outputting anomaly location.

[0065] Fusion features Inputting this multi-task network will output the overall anomaly confidence score. , Probability vector of anomaly type and spatial positioning Spatial localization includes bounding boxes on the image / heatmap. This allows for anomaly detection, localization, and classification within the same model, reducing system complexity.

[0066] Among them, the overall anomaly confidence level Defined as a weighted combination of the scores from each modality:

[0067] In formula (10): The output of the regression head represents the anomaly confidence score for each mode in the anomaly detection network.

[0068] The anomaly detection rule is: if the overall anomaly confidence level is... If the value exceeds the preset threshold, a "suspected anomaly" message is triggered; otherwise, the record is entered into the normal log.

[0069] S4. Obtain the fusion feature sequence of a given historical window length and use it as the input of the prediction model to predict the comprehensive anomaly confidence sequence for the next K time moments. Compare the comprehensive anomaly confidence at the next time moment with a preset threshold. If it is greater than the preset threshold, trigger an early warning.

[0070] In this embodiment, the network structure of the prediction model is as follows: Figure 4 As shown, it includes a decoder part and a target header part. The decoder part can use the TimeSformer decoder, and the target header is divided into a time-series prediction header for outputting future abnormal trend prediction values. This is achieved by fusing feature sequences with a given historical window length L. To predict the comprehensive anomaly confidence sequence for the next K time points. .

[0071] Specifically, the prediction model uses the TimeSformer model, and the prediction function is defined as follows: ,in Then we have: .

[0072] Training loss of the TimeSformer model It consists of a regression term and an uncertainty term:

[0073] In equation (12): For the regression term, This is an uncertain term; Indicates the time after t time; Indicates the parameters of the TimeSformer model; To balance the weights.

[0074] The early warning judgment rule is: when the comprehensive anomaly confidence level at a predicted future time is... Greater than the preset threshold, and the rate of increase An early warning is triggered when the temperature exceeds a preset threshold. For example, if the transformer bushing temperature is predicted to rise continuously over the next 6 hours and is accompanied by an abnormal increase, the system will trigger a "potential insulation breakdown" warning in advance and suggest a shutdown for maintenance.

[0075] Thus, by setting dual thresholds for confidence level and growth rate to trigger early warnings, potential equipment degradation or failure trends can be identified in advance.

[0076] The effectiveness of the method of the present invention will be verified by specific examples below.

[0077] Take a 500kV substation as an example.

[0078] Equipment deployment: Three types of sensors are deployed in the main transformer area of ​​the substation: a visible light camera (30fps), an infrared thermal imager (25fps), and an omnidirectional microphone (48kHz sampling rate). All devices are connected to the edge computing unit, and time synchronization is achieved using GPS timing with an accuracy better than 1ms.

[0079] Data Acquisition and Synchronization: Continuously collect data for 24 hours, set an alignment window every 5 seconds, use spline interpolation to uniformly resample visible light and infrared frames to 25fps, and segment audio into 50ms segments and interpolate and align them to the same timeline.

[0080] Preprocessing and quality assessment: Daytime images are clear, brightness = 120, variance = 800, calculated as 0.92; Nighttime backlight scene, brightness = 40, variance = 150, = 0.35; Infrared image shows sleeve temperature 68°C, background temperature 35°C, = 33, = 0.88; Audio signal-to-noise ratio = 15dB, spectral flatness = 0.6, = 0.72; Generated signal quality vector = [0.35, 0.88, 0.72].

[0081] Dynamic weight adjustment: Input the MobileViT network and output the original score. After Softmax normalization, the probability weights are obtained [0.18, 0.47, 0.35]. After smoothing with EMA (α=0.9), the final weights are obtained [0.20, 0.46, 0.34], with the infrared mode becoming dominant.

[0082] Feature extraction and fusion: Video Transformer extracts infrared spatiotemporal features, Audio Transformer extracts audio features, and Cross-Attention fuses them before inputting them into a multi-task head.

[0083] Anomaly detection results: The system outputs an overall anomaly confidence level of 0.83 (>threshold 0.6), which is judged as "suspected overheating". The location bounding box covers the top of the bushing, and the type probability shows that "insulation aging" accounts for 72%.

[0084] Trend prediction and early warning: Take the fusion features of historical window length L=12 time steps (5 minutes each) and input them into TimeSformer to predict the confidence sequence of the future K=6 steps (30 minutes). The model predicts that the confidence will continue to rise, reaching 0.95 after 6 minutes, with an increase rate of 0.02 / min (>threshold 0.015), triggering an "early overheat warning". It is recommended to arrange infrared retesting within 2 hours and prepare a power outage plan.

[0085] This example demonstrates that the present invention has good diagnostic capabilities and early warning foresight in a real substation environment.

[0086] Example 2 This embodiment provides an apparatus for implementing an intelligent diagnosis method for power transmission and transformation equipment anomalies based on adaptive multimodal weights. The apparatus includes: The data acquisition module is used to acquire multimodal data and preprocess each modal data to obtain the signal quality vector of each modal data; wherein, the multimodal data includes visible light modal data, infrared modal data and audio modal data with consistent time reference; The weight calculation module is used to calculate the weight of each mode data based on the signal quality vector of each mode data. The anomaly detection module is used to extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time, which is used as the input of the anomaly detection network to obtain the anomaly recognition result. The anomaly prediction module is used to obtain a fusion feature sequence of a given historical window length and use it as input to the prediction model to predict the comprehensive anomaly confidence sequence at the next K time points. The comprehensive anomaly confidence at the next time point is compared with a preset threshold. If it is greater than the preset threshold, an early warning is triggered.

[0087] Example 3 This embodiment also provides an electronic device, including: At least one processor; and The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights as described above.

[0088] In this embodiment, the electronic device may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile computing device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable computing device, consumer electronic device, etc.

[0089] Example 4 This embodiment also provides a machine-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights as described above.

[0090] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0091] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0092] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for intelligent diagnosis of anomalies in power transmission and transformation equipment based on adaptive multimodal weights, characterized in that, The method includes: S1. Acquire multimodal data and preprocess each modal data to obtain the signal quality vector of each modal data; wherein, the multimodal data includes visible light modal data, infrared modal data and audio modal data with consistent time reference; S2. Based on the signal quality vector of each modal data, calculate the weight of each modal data accordingly; S3. Extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time. Use it as the input of the anomaly detection network to obtain the anomaly recognition result. S4. Obtain the fusion feature sequence of a given historical window length and use it as the input of the prediction model to predict the comprehensive anomaly confidence sequence for the next K time moments. Compare the comprehensive anomaly confidence at the next time moment with a preset threshold. If it is greater than the preset threshold, trigger an early warning.

2. The intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights according to claim 1, characterized in that, In step S1, the multimodal data includes visible light modal data, infrared modal data, and audio modal data with a consistent time reference, specifically: For visible light image frames Infrared image frames Audio frame fragments ,in, Indicates the acquisition timestamps of visible light image frames and infrared image frames. Represents the timestamp of audio frame segment acquisition, and defines the alignment window. , Indicate the width of the time alignment window and construct the synchronization frame sequence: In formula (1): Indicates a sequence of synchronization frames; This indicates any sampling point within the aligned window after interpolation.

3. The intelligent diagnosis method for power transmission and transformation equipment anomalies based on adaptive multimodal weights according to claim 2, characterized in that, In S1, the signal quality vector of the visible light modal data The calculation is as follows: In formula (2): This represents the average brightness of a visible light image frame; Represents a visible light image frame; Indicates the number of visible light image frames; This represents the luminance variance of a visible light image frame; express The signal quality vector of the visible light image frame at time 1; This represents a function for calculating the signal quality vector of a visible light image frame; Indicates a custom comparison threshold; Indicates a custom scale parameter; The signal quality vector of the infrared modal data The calculation is as follows: In formula (3): express The signal quality vector of the infrared image frame at time 1; This function represents the calculation of the signal quality vector of an infrared image frame. Indicates thermal contrast, and , Indicates the maximum temperature. Indicates the background temperature; The stage function representing the value of thermal radiation intensity will Values ​​are limited to a preset minimum value of 0 and a maximum value of 1; values ​​outside this range are forcibly set to boundary values. The signal quality vector of the audio modal data The calculation is as follows: In equation (4): Indicates the signal-to-noise ratio. This represents the average power of the signal; This represents the average power of the noise; Indicates spectral flatness; Indicates the number of audio frame sampling points; This represents the amplitude of the k-th sampling point after the audio frame signal has undergone Fourier transform; express The signal quality vector of the audio frame segment at a given time; This represents a function that calculates the signal quality vector of an audio frame segment.

4. The intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights according to claim 1, characterized in that, S2 specifically includes: The signal quality vector of each modality data is mapped using the lightweight neural network MobileViT. to the corresponding raw score : In equation (6): The parameter is MobileViT neural network; Represents a three-dimensional real vector space; This represents the signal quality vector for each mode of data. Corresponding to visible light modal data Infrared modal data and audio modal data ; Alignment and softmax normalization yield the initial weights for each modality. : In equation (7): A custom temperature coefficient is used to control the weight distribution; The target weights for each modality of data after smoothing are obtained using the exponential moving average (EMA). : In equation (8): Customize the historical memory coefficient.

5. The intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights according to claim 4, characterized in that, In step S3, the feature vectors of each modality data at the current time are extracted and fused to obtain the fused features at the current time, specifically including: The Video Transformer feature extractor is used to extract spatiotemporal feature vectors from visible light modal data and infrared modal data, respectively, to obtain the feature vectors of visible light modal data at the current time. And the feature vector of the infrared modal data at the current moment. , , Both represent dimensions; Audio feature vectors of audio modal data are extracted from frequencies using the AudioTransformer feature extractor. , Indicates dimension; The feature vectors of data from various modalities are fused using a cross-modal attention approach to obtain the fused features at the current time step. : 。 6. The intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights according to claim 5, characterized in that, In step S3, the fusion features at the current moment are... Inputting the anomaly detection network will output the overall anomaly confidence score. , Probability vector of anomaly type and spatial positioning Spatial positioning Bounding box information containing visible light modal data and infrared modal data; The comprehensive anomaly confidence level Defined as a weighted combination of the scores from each modality: In formula (10): The output of the regression head represents the anomaly confidence score for each mode in the anomaly detection network.

7. The intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights according to claim 6, characterized in that, In S4, a fusion feature sequence with a historical window length of L is given. The prediction model uses the TimeSformer model, and the prediction function is defined as follows: Among them, the comprehensive anomaly confidence sequence Then we have: Training loss of the TimeSformer model It consists of a regression term and an uncertainty term: In equation (12): For the regression term, This is an uncertain term; Indicates the time after t time; Indicates the parameters of the TimeSformer model; To balance the weights.

8. An apparatus for implementing an intelligent diagnosis method for power transmission and transformation equipment anomalies based on adaptive multimodal weights, characterized in that, The device includes: The data acquisition module is used to acquire multimodal data and preprocess each modal data to obtain the signal quality vector of each modal data; wherein, the multimodal data includes visible light modal data, infrared modal data and audio modal data with consistent time reference; The weight calculation module is used to calculate the weight of each mode data based on the signal quality vector of each mode data. The anomaly detection module is used to extract the feature vector of each modality data at the current time and fuse them to obtain the fused feature at the current time, which is used as the input of the anomaly detection network to obtain the anomaly recognition result. The anomaly prediction module is used to obtain a fusion feature sequence of a given historical window length and use it as input to the prediction model to predict the comprehensive anomaly confidence sequence at the next K time points. The comprehensive anomaly confidence at the next time point is compared with a preset threshold. If it is greater than the preset threshold, an early warning is triggered.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights as described in any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores executable instructions that, when executed, cause the machine to perform the intelligent diagnostic method for power transmission and transformation equipment based on adaptive multimodal weights as described in any one of claims 1 to 7.

Images