Power distribution equipment state detection method and device based on multi-modal data fusion, terminal equipment and storage medium

Abstract

The invention discloses a power distribution equipment state detection method and device based on multi-modal data fusion, terminal equipment and a storage medium, and belongs to the field of power distribution equipment. Performing feature analysis on the acquired multi-modal data by using a time sequence convolutional network, a residual neural network and a Transform network to obtain a local abnormal fluctuation feature, an infrared temperature spatial distribution feature and a video global feature representation; performing feature splicing on the local abnormal fluctuation feature, the infrared temperature spatial distribution feature and the video global feature representation to obtain a fusion feature, and inputting the fusion feature to a long-short term memory network to obtain a historical hidden state sequence; and determining the equipment state of the to-be-detected power distribution equipment according to the historical hidden state sequence, the fusion features and edge equipment deployed in the local network domain of the equipment. By implementing the application, the problem of low accuracy of state analysis of the power distribution equipment caused by only depending on single modal data in the prior art can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power distribution equipment, and in particular to a power distribution equipment state detection method and device based on multi-modal data fusion, a terminal device and a storage medium. BACKGROUND

[0002] With the deepening of the construction of smart grid, the safe and stable operation of power distribution equipment is directly related to the reliability of the power system and the power supply quality. Accurate detection of the state of power distribution equipment and timely detection of potential fault hazards are of great significance to the protection of power supply and the reduction of economic losses.

[0003] In the prior art, power distribution equipment state detection is mostly based on single modal data, such as relying only on real-time running data, infrared images or running video files for analysis. The detection method based on real-time running data can obtain electrical parameters such as current and voltage of the equipment, but is easily disturbed by power grid load fluctuations, environmental factors and other factors, making it difficult to accurately capture local abnormalities of the equipment. Using only infrared images for detection can intuitively reflect the temperature distribution of the equipment, but cannot combine with the timing information of the equipment operation, making it difficult to judge the development trend of temperature abnormalities. Simply relying on running video files for detection has limitations in image feature extraction and behavior analysis, and has insufficient ability to identify minor faults. Due to the limited information dimension of single modal data, the running state of the power distribution equipment cannot be fully reflected, resulting in low accuracy of the detection results, which cannot meet the needs of actual engineering applications. SUMMARY

[0004] The present application provides a power distribution equipment state detection method and device based on multi-modal data fusion, a terminal device and a storage medium, which can solve the problem of low accuracy of the power distribution equipment state analysis in the prior art due to relying only on single modal data.

[0005] An embodiment of the present application provides a power distribution equipment state detection method based on multi-modal data fusion, comprising:

[0006] Obtaining real-time running data, infrared images and running video files of the power distribution equipment to be detected;

[0007] According to the preset timing convolution network and the real-time running data, local abnormal fluctuation features in the timing window sample are extracted;

[0008] Using a preset residual neural network and a preset Transformer network, infrared temperature spatial distribution features corresponding to the infrared images are captured;

[0009] The residual neural network is used for spatial feature extraction of each frame image in the running video file, to obtain a spatial feature representation corresponding to each frame image, and according to a preset three-dimensional convolutional network, the time sequence convolutional network and the spatial feature representation corresponding to each frame image, a video global feature representation containing time sequence dynamic information and spatial semantic information is obtained.

[0010] The local abnormal fluctuation feature, the infrared temperature spatial distribution feature and the video global feature representation are spliced to obtain a fusion feature, and the fusion feature is input into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network.

[0011] According to the historical hidden state sequence, the fusion feature and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution equipment, a device state of the to-be-inspected power distribution equipment is determined.

[0012] Further, the local abnormal fluctuation feature in the time sequence window sample is extracted according to the preset time sequence convolutional network and the real-time running data, including:

[0013] According to the time sequence feature in the real-time running data, the real-time running data is divided into a plurality of time sequence window samples;

[0014] According to the preset time sequence convolutional network, local feature extraction is performed on the time sequence window sample to extract the local abnormal fluctuation feature in the time sequence window sample.

[0015] Further, the infrared temperature spatial distribution feature corresponding to the infrared image is captured by using the preset residual neural network and the preset Transformer network, including:

[0016] The infrared image is subjected to pixel normalization processing to obtain a pixel normalized infrared image;

[0017] The pixel normalized infrared image is input into the preset residual neural network to obtain an initial feature of the infrared image;

[0018] The initial feature of the infrared image is input into the preset Transformer network to obtain the infrared temperature spatial distribution feature captured by the Transformer network.

[0019] Further, the video global feature representation containing time sequence dynamic information and spatial semantic information is obtained according to the preset three-dimensional convolutional network, the time sequence convolutional network and the spatial feature representation corresponding to each frame image, including:

[0020] The spatial feature representation corresponding to each frame image in the running video file is sorted in time sequence to form a feature sequence;

[0021] According to the preset sliding window, the feature sequence is intercepted to obtain a plurality of time sequence feature segments;

[0022] The inter-frame time sequence dynamic feature of each time sequence feature segment is captured by a preset three-dimensional convolutional network to obtain a time sequence dynamic feature.

[0023] The time sequence dynamic feature is input into the time sequence convolutional network to determine a video global feature representation containing time sequence dynamic information and spatial semantic information.

[0024] Further, the device state of the to-be-inspected power distribution equipment is determined according to the historical hidden state sequence, the fusion feature, and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution equipment, and the device state of the to-be-inspected power distribution equipment is determined according to the historical hidden state sequence, the fusion feature, and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution equipment.

[0025] The historical hidden state sequence and the fusion feature are integrated into a final fusion feature, and the final fusion feature is input into an edge device deployed in a local network domain of the to-be-inspected power distribution equipment, so that the edge device outputs the device state of the to-be-inspected power distribution equipment based on network parameters issued by a pre-trained federal learning system.

[0026] Further, before the final fusion feature is input into the edge device deployed in the local network domain of the to-be-inspected power distribution equipment, the method further comprises:

[0027] The final fusion feature is mapped to a node feature corresponding to a preset device topology graph by using a preset graph neural network to obtain an updated device topology graph, wherein the device topology graph comprises a plurality of power distribution equipment nodes and a connection relationship between the power distribution equipment nodes.

[0028] An embodiment of the present application also provides a power distribution equipment state detection device based on multi-modal data fusion, comprising: a data acquisition module, a local abnormal fluctuation feature extraction module, an infrared temperature spatial distribution feature capture module, a video global feature representation determination module, a historical hidden state sequence determination module, and a device state determination module.

[0029] The data acquisition module is configured to acquire real-time running data, an infrared image, and a running video file of a to-be-inspected power distribution equipment.

[0030] The local abnormal fluctuation feature extraction module is configured to extract a local abnormal fluctuation feature in the time sequence window sample according to a preset time sequence convolutional network and the real-time running data.

[0031] The infrared temperature spatial distribution feature capturing module is configured to capture infrared temperature spatial distribution features corresponding to the infrared image by using a preset residual neural network and a preset Transformer network.

[0032] The video global feature representation determining module is configured to extract spatial features of each frame image in a running video file by using the residual neural network to obtain spatial feature representations corresponding to the frame images, and obtain a video global feature representation containing time sequence dynamic information and spatial semantic information according to a preset three-dimensional convolution network, the time sequence convolution network, and the spatial feature representations corresponding to the frame images.

[0033] The historical hidden state sequence determining module is configured to splice the local abnormal fluctuation feature, the infrared temperature spatial distribution feature, and the video global feature representation to obtain a fusion feature, and input the fusion feature into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network.

[0034] The device state determining module is configured to determine a device state of the to-be-inspected power distribution device according to the historical hidden state sequence, the fusion feature, and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution device.

[0035] Further, the power distribution device state detection apparatus based on multi-modal data fusion further includes a device topology graph updating module.

[0036] The device topology graph updating module is configured to map the final fusion feature to a node feature corresponding to a preset device topology graph by using a preset graph neural network to obtain an updated device topology graph; the device topology graph includes a plurality of power distribution device nodes and connection relationships between the power distribution device nodes.

[0037] The application further provides a terminal device, including:

[0038] one or more processors;

[0039] a memory coupled to the processors and configured to store one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the power distribution device state detection method based on multi-modal data fusion as described in the above embodiments.

[0041] The application further provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the power distribution device state detection method based on multi-modal data fusion as described in the above embodiments.

[0042] By implementing the present application, the following advantages are achieved:

[0043] The application provides a power distribution equipment state detection method and device based on multi-modal data fusion, terminal equipment and storage medium. The method comprehensively collects real-time operation data, infrared images and operation video files of the power distribution equipment to be detected, processes the real-time operation data according to a preset time sequence convolution network, and extracts local abnormal fluctuation features in the time sequence window sample. Secondly, a preset residual neural network and a preset Transformer network are used to capture infrared temperature spatial distribution features corresponding to the infrared images. In addition, the residual neural network is used to extract spatial features of each frame of image in the operation video file to obtain spatial feature representations corresponding to each frame of image, and a preset three-dimensional convolution network, the time sequence convolution network and the spatial feature representations corresponding to each frame of image are used to obtain video global feature representations containing time sequence dynamic information and spatial semantic information. Thus, the limitations of single modal data are broken through, real-time operation data, infrared images and operation video files are comprehensively collected, electrical parameters, temperature distribution and dynamic behavior information of the equipment are fused, different modal features are extracted by the time sequence convolution network, the residual neural network, the Transformer network and the three-dimensional convolution network, the advantages of data are complemented, potential abnormalities of the equipment are effectively captured, and the accuracy of power distribution equipment state detection is greatly improved.

[0044] Finally, the local abnormal fluctuation features, the infrared temperature spatial distribution features and the video global feature representations are spliced to obtain fusion features, the fusion features are input into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network, and then the device state of the power distribution equipment to be detected is determined according to the historical hidden state sequence, the fusion features and an edge device deployed in a local network domain of the power distribution equipment to be detected. Thus, the device state is quickly analyzed and determined locally, a large amount of data does not need to be transmitted to a remote server, data transmission delay is reduced, fault response speed is improved, network transmission pressure and data leakage risk are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1is a flowchart of a power distribution equipment state detection method based on multi-modal data fusion provided by an embodiment of the present application;

[0047] Figure 2 is a structural diagram of a power distribution equipment state detection device based on multi-modal data fusion provided by an embodiment of the present application;

[0048] Figure 3 is a structural diagram of a power distribution equipment state detection device based on multi-modal data fusion provided by another embodiment of the present application;

[0049] Figure 4 is a structural diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0052] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0053] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0054] In the description of the embodiments of the present application, the term "and / or" is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0055] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0056] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connection", "connection", "fixing" and the like should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanical connection, or it can be electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For ordinary skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0057] Reference Figure 1 To solve the problem that the state accuracy of power distribution equipment obtained by analyzing only single modal data is low in the prior art, an embodiment of the present application provides a power distribution equipment state detection method based on multi-modal data fusion, which comprises the following steps:

[0058] S1, acquiring real-time running data, infrared image and running video file of the power distribution equipment to be detected;

[0059] Specifically, the real-time running data includes real-time current value of the power distribution equipment to be detected, voltage value at both ends of the power distribution equipment to be detected, temperature, vibration and the like when the power distribution equipment to be detected is running;

[0060] Specifically, the equipment thermal radiation distribution image (i.e. the infrared image) photographed by the infrared thermal imager includes a local temperature abnormal area and a thermal radiation intensity distribution, wherein the local temperature abnormal area includes overheating of a wiring terminal (temperature of a cable joint is increased due to poor contact, which presents a highlight color block in the infrared image), an insulation deterioration area (thermal loss is increased due to aging of an insulator, and the temperature is higher than that of a normal part) and a component failure heat generation (local temperature abnormality when a winding fault occurs in a transformer); the thermal radiation intensity distribution displays a temperature gradient in the form of a pseudo-color image, which is convenient for directly locating the overheating point and the temperature abnormal area;

[0061] Specifically, the running video file includes real-time dynamic video frames of key components of the device, for monitoring mechanical actions and appearance changes, including key component running states (such as opening and closing actions of circuit breaker contacts, surface deformation of insulating sleeves, and liquid level changes of transformer oil level gauges), surface abnormal phenomena (such as component vibration abnormalities, discharge phenomena), and the like.

[0062] S2, extracting local abnormal fluctuation features in the time series window sample according to the preset time series convolutional network and the real-time running data;

[0063] Illustratively, the preset time series convolutional network (Temporal Convolutional Network, TCN) is constructed by a multi-layer convolutional structure with a one-dimensional dilated causal convolution as the core. By sliding and scanning the time series window sample with different size convolution kernels, and by layer-by-layer extracting local fluctuation patterns (such as sudden rise, temporary drop, periodic disturbance, etc.) in the real-time running data of current, voltage, etc. with a dilated factor, the time series signal is converted into a vector representation containing local abnormal fluctuation features.

[0064] In a preferred embodiment, the extracting of the local abnormal fluctuation features in the time series window sample according to the preset time series convolutional network and the real-time running data comprises:

[0065] According to the time series features in the real-time running data, the real-time running data is divided into a plurality of time series window samples;

[0066] According to the preset time series convolutional network, local feature extraction is performed on the time series window sample, and the local abnormal fluctuation features in the time series window sample are extracted;

[0067] Specifically, according to the time series features in the real-time running data, the real-time running data is divided into a plurality of time series window samples, the window size is T1, the step size is S, and the i-th time series window sample is:

[0068] t = (i-1) · S + 1;

[0069] In the formula, x t represents a specific data point of the i-th time series window sample, R d represents d-dimensional real-time running data, which is determined by the type of real-time running data; t represents the data point index corresponding to the window start time;

[0070] Specifically, the time window sample is input to the first layer of the time convolution network, and feature mapping is performed through the convolution kernel built in the time convolution network. The convolution kernel detects high and low frequency components in the time window through a sliding window, and through stacking multiple convolution layers (each layer using a different expansion factor), the features of different time scales are fused to form a hierarchical representation. For example, the first layer extracts short-term fluctuations (such as current surges); the second layer extracts medium-term trends (such as load changes); the third layer extracts long-term patterns (such as device aging), and the final output feature vector contains abnormal fluctuation information of each time step in the time window sample, i.e. local abnormal fluctuation features (such as: current surge is represented by a feature value significantly higher than the threshold at a certain time step; voltage sag is represented by a feature value lower than the threshold for consecutive multiple time steps).

[0071] Specifically, the TCN has higher computational efficiency and can avoid the gradient vanishing problem in long-time dependence. It adopts dilated causal convolution to capture local fluctuation patterns.

[0072]

[0073] In the formula, h tcn represents the local abnormal fluctuation feature vector output after dilated causal convolution calculation; w k represents the weight parameter of the convolution kernel, which determines the contribution of different time offsets to feature extraction; x t-d·k represents the input time window sample; d is the expansion factor; K is the size of the convolution kernel; k represents the position index within the convolution kernel; t represents the time step index currently being processed.

[0074] S3, using a preset residual neural network and a preset Transformer network to capture an infrared temperature spatial distribution feature corresponding to the infrared image;

[0075] In a preferred embodiment, the use of a preset residual neural network and a preset Transformer network to capture an infrared temperature spatial distribution feature corresponding to the infrared image comprises:

[0076] Performing pixel normalization processing on the infrared image to obtain a pixel normalized infrared image;

[0077] Inputting the pixel normalized infrared image into the preset residual neural network to obtain an initial feature of the infrared image;

[0078] Inputting the initial feature of the infrared image into the preset Transformer network to obtain an infrared temperature spatial distribution feature captured by the Transformer network;

[0079] Specifically, the infrared image is subjected to pixel normalization processing An infrared image subjected to pixel normalization processing is obtained

[0080]

[0081] In the formula, H represents the height of the infrared image I h,w,c ; W represents the width of the infrared image I h,w,c ; C represents the channel number of the infrared image I h,w,c ; μ image represents the mean of all collected infrared images; σ image represents the standard deviation of all collected infrared images;

[0082] Specifically, the input layer of the preset residual neural network (ResNet-18) receives the infrared image subjected to normalization processing; the backbone network thereof is constructed by stacking multiple residual blocks, two 3x3 convolution layers are arranged in each residual block, and the depth network gradient disappearance problem is avoided by means of a skip connection mechanism; the output layer relies on global average pooling and a fully connected layer operation to convert feature mapping into a fixed-dimension feature vector, which is used as the initial feature of the infrared image, and the initial feature (edge, temperature gradient, etc.) of the image is extracted by multiple layers of convolution;

[0083] Specifically, the infrared image subjected to pixel normalization processing is input into the preset residual neural network to obtain the initial feature of the infrared image.

[0084] Specifically, the preset Transformer network captures the long-distance dependence relationship between pixels in the infrared image by means of a self-attention mechanism, and converts the initial feature into an infrared temperature spatial distribution feature reflecting the temperature spatial distribution; the structure thereof comprises: the input layer maps the initial feature extracted by the ResNet into a sequence form; the encoder is stacked by multiple Transformer blocks, each block comprising a multi-head self-attention layer and a feedforward neural network, the self-attention layer calculates the correlation weight between pixels by means of a query (Query), key (Key), and value (Value) matrix, and captures spatial relationships such as "temperature gradient between hot spots and surrounding areas" and "heat conduction mode between different device components"; the feature vector generated by the output layer comprises the infrared temperature spatial distribution feature, and highlights the spatial distribution feature of the temperature abnormal area.

[0085] Specifically, when processing the infrared image by means of the Transformer network, the initial feature of the infrared image extracted by the residual neural network is input into the preset Transformer network, which works based on a multi-head self-attention mechanism, and the mechanism can be represented as:

[0086]

[0087] wherein, h trans represents the output infrared temperature spatial distribution feature; Q represents a query vector for locating a key feature area (such as an overheated component) in the infrared image; K represents a key vector encoding a feature space position; T represents a time sequence window size; d k represents the key vector dimension for controlling the attention weight distribution range; V represents a value vector for weighting the feature importance.

[0088] Thus, the infrared temperature spatial distribution feature captured by the Transformer network is obtained.

[0089] S4, using the residual neural network to extract spatial features of each frame image in the running video file, obtaining spatial feature representation corresponding to each frame image, and obtaining video global feature representation containing time sequence dynamic information and spatial semantic information according to the preset three-dimensional convolution network, the time sequence convolution network and the spatial feature representation corresponding to each frame image.

[0090] Illustratively, when extracting spatial features of each frame image in the running video file, the residual neural network (ResNet-18) used is consistent with the residual neural network used in step S3 to extract initial features from the infrared image, and the residual neural network will not be described here.

[0091] Specifically, using the residual neural network (ResNet-18) to extract spatial features, spatial features of each frame image in the running video file are extracted to obtain spatial feature representation corresponding to each frame image.

[0092] In a preferred embodiment, the video global feature representation containing time sequence dynamic information and spatial semantic information is obtained according to the preset three-dimensional convolution network, the time sequence convolution network and the spatial feature representation corresponding to each frame image, comprising:

[0093] The spatial feature representation corresponding to each frame image in the running video file is sorted in time sequence to form a feature sequence.

[0094] According to the preset sliding window, the feature sequence is intercepted to obtain a plurality of time sequence feature segments.

[0095] Through the preset three-dimensional convolution network, inter-frame time sequence dynamic features of each time sequence feature segment are captured to obtain time sequence dynamic features.

[0096] The time sequence dynamic features are input into the time sequence convolution network to determine the video global feature representation containing time sequence dynamic information and spatial semantic information.

[0097] Specifically, the spatial feature representation corresponding to each frame image is sorted according to the time sequence of video playback to construct a feature sequence, which can reflect the preliminary change trajectory of the device state in the time dimension; then, the feature sequence is intercepted according to the pre-set sliding window parameter, to obtain a plurality of time sequence feature segments, wherein each time sequence feature segment focuses on the feature change of the power distribution equipment to be detected within a continuous time period, and then these segments are input into a pre-set three-dimensional convolution network (ResNet3D). The three-dimensional convolution network can effectively mine the time sequence dynamic correlation between frames in the segment, such as the continuous process of device mechanical action and the development trend of surface deformation, and finally output the captured time sequence dynamic features, providing rich time sequence information support for subsequent device state comprehensive perception and fault diagnosis. The specific calculation formula is as follows:

[0098]

[0099] In the formula, F image represents the time sequence dynamic feature obtained after ResNet3D processing; ResNet3D(·) represents a three-dimensional residual neural network; I 1:L represents the time sequence feature segment input into the ResNet3D; T2 represents the time step of the time sequence feature segment; D represents the length of the feature vector corresponding to each time step; represents the output time sequence dynamic feature F image is a matrix in a real number space;

[0100] Specifically, the time sequence dynamic feature is input into the time sequence convolution network, so that the time sequence convolution network outputs a video global feature representation containing time sequence dynamic information and spatial semantic information.

[0101] It should be noted that the time sequence convolution network is the same as the time sequence convolution network mentioned in step S2, and will not be described here.

[0102] S5, the local abnormal fluctuation feature, the infrared temperature spatial distribution feature and the video global feature representation are spliced to obtain a fusion feature, and the fusion feature is input into a pre-set long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network.

[0103] Specifically, the local anomaly fluctuation features, video global feature representations, and temperature distribution features extracted by the temporal convolutional network are concatenated to obtain fused features. The feature concatenation requires alignment according to the feature dimension (or time step) and then concatenation and merging of these three types of features in the same dimension. After integrating the local anomaly fluctuation features, infrared temperature spatial distribution features, and video global feature representations, more comprehensive multimodal inputs can be provided for subsequent Long Short-Term Memory (LSTM) networks.

[0104] Specifically, the pre-defined Long Short-Term Memory (LSTM) network utilizes a collaborative mechanism of forget gates, input gates, and output gates to perform temporal modeling of the fused features. Based on the hidden state of the previous time step and the current input, the forget gate selectively discards redundant information from historical cell states, the input gate filters effective components from the current fused features, and the output gate controls the output intensity of the final feature. After gating and filtering long-term dependent information (such as the trend of fault gradual change signals), features from different time steps are fused in the cell state to generate a historical hidden state sequence. This historical hidden state sequence covers the fused information of the device's temporal features, including the gradual trend of the device's operating state and the cumulative effect of abnormal fluctuations.

[0105] It should be noted that the Long Short-Term Memory network mentioned is an existing network, and will not be described in detail here.

[0106] S6. Determine the device status of the power distribution equipment under inspection based on the historical hidden state sequence, the fusion features, and the edge devices pre-deployed in the local network domain of the power distribution equipment under inspection.

[0107] In a preferred embodiment, determining the device status of the power distribution equipment under inspection based on the historical hidden state sequence, the fusion features, and edge devices pre-deployed within the local network domain of the power distribution equipment under inspection includes:

[0108] The historical hidden state sequence and the fusion feature are integrated into a final fusion feature, and the final fusion feature is input to the edge device deployed in the local network domain of the power distribution equipment under inspection, so that the edge device outputs the equipment status of the power distribution equipment under inspection based on the network parameters issued by the pre-trained federated learning system.

[0109] Specifically, the historical hidden state sequence and the fused features are concatenated along the time step dimension and input into a preset neural network. The neural network comprises two layers, the first layer using parameter W. μ Bias b μ Features after splicing Perform a linear transformation and output the mean. The second layer uses parameter W. σ Bias bσ Similarly, for the stitching feature processing, the output log variance Finally, according to the mean And the log variance (First restored from the log variance ), introduce noise Following the standard normal distribution Generate the final fusion feature Where, through the deep learning framework (such as PyTorch), using tensor operations, automatic differentiation and other functions, the calculation of the final fusion feature can be realized, and the specific calculation formula is as follows:

[0110]

[0111] Illustratively, after obtaining the final fusion feature, the final fusion feature is input into the edge device deployed in the local network domain of the to-be-inspected power distribution equipment, so that the edge device outputs the device state of the to-be-inspected power distribution equipment based on the network parameters issued by the pre-trained federal learning system.

[0112] Specifically, the device state includes normal or abnormal;

[0113] Specifically, the edge device refers to a client participating in federal learning (such as a communication device deployed in the local network domain of the to-be-inspected power distribution equipment), which is divided into three stages when training the federal learning system:

[0114] Local training updates local model parameters by inputting local data sets (including historical final fusion features and corresponding manually annotated fault labels), through gradient descent, the specific formula is as follows:

[0115]

[0116] In the formula, is the local model parameter of the mth edge device at t2; η is the learning rate, used to control the step size of parameter update; is the loss function; D m is the local data set of the mth edge device;

[0117] The cloud of the federal learning system aggregates the parameter updates of all cloud devices using the FedAvg algorithm to generate global model parameters The specific calculation formula is as follows:

[0118]

[0119] In the formula, represents the global model parameter aggregated by the cloud at t2+1; M represents the total number of edge devices participating in federal learning; denotes the updated local model parameter uploaded by the mth edge device at t2+1;

[0120] Finally, the resource allocation is optimized by dynamic scheduling combined with the PPO algorithm:

[0121]

[0122] wherein, π * (a|s) denotes the optimal policy, and denotes the best probability distribution of selecting action a under state s; denotes the expected value under policy π * (a|s); γ is a discount factor, used to weigh the importance of current and future rewards; is the state at t2, and action is performed under state

[0123] Specifically, after optimizing the resource allocation by the PPO algorithm, the edge device uploads the updated parameters to the cloud with local data, the cloud aggregates to generate global model parameters and then issues them, and the iterative process is repeated until the federated learning system is trained. Then the federated learning system sends the parameters to the edge device, and the edge device obtains the network parameters issued by the pre-trained federated learning system, and then calculates the device state of the to-be-inspected power distribution equipment combined with the final fusion features obtained.

[0124] In a preferred embodiment, before inputting the final fusion features into the edge device deployed in the local network domain of the to-be-inspected power distribution equipment, it further comprises:

[0125] The final fusion features are mapped into the node features corresponding to the preset device topology graph by using a preset graph neural network, to obtain an updated device topology graph; wherein, the device topology graph comprises a plurality of power distribution equipment nodes and connection relationships between the power distribution equipment nodes.

[0126] Specifically, the final fusion features of the to-be-inspected power distribution equipment are taken as the initial features of each layer of GNN by using a preset graph neural network (GNN), and then the neighbor node features of each node are aggregated according to the connection relationship between the devices. After the calculation of each layer of GNN, the features of each node are updated, thereby obtaining an updated device topology graph. Specifically, the specific formula for updating the node features is as follows:

[0127]

[0128] wherein, denotes the feature representation of node u1 at the l j +1 layer, i.e., the feature of updated node u1; σ denotes an activation function ReLU; denotes the lj a trainable weight matrix of a layer for linear transformation on the aggregated features; N(u1) represents a neighbor node set of node u1, that is, other power distribution equipment nodes connected with u1; represents the final fusion feature of node u1 in the l j th layer, which is the initial feature of node u1 itself in the l j th layer; u2 represents a neighbor power distribution equipment node in the neighbor node set of node u1; represents the feature representation of neighbor node u2 in the l j th layer; represents the bias term of the l j th layer; |N(u1)| represents the number of neighbor nodes of node u1.

[0129] Referring to Figure 2 , the application provides a power distribution equipment state detection device based on multi-modal data fusion, comprising a data acquisition module, a local abnormal fluctuation feature extraction module, an infrared temperature spatial distribution feature capture module, a video global feature representation determination module, a historical hidden state sequence determination module and an equipment state determination module.

[0130] The data acquisition module is configured to acquire real-time operation data, an infrared image and an operation video file of a power distribution equipment to be detected.

[0131] The local abnormal fluctuation feature extraction module is configured to extract local abnormal fluctuation features in the time series window sample according to a preset time series convolution network and the real-time operation data.

[0132] The infrared temperature spatial distribution feature capture module is configured to capture infrared temperature spatial distribution features corresponding to the infrared image by using a preset residual neural network and a preset Transformer network.

[0133] The video global feature representation determination module is configured to extract spatial features of each frame image in the operation video file by using the residual neural network to obtain spatial feature representations corresponding to the frame images, and obtain video global feature representations containing time series dynamic information and spatial semantic information according to a preset three-dimensional convolution network, the time series convolution network and the spatial feature representations corresponding to the frame images.

[0134] The historical hidden state sequence determination module is configured to splice the local abnormal fluctuation features, the infrared temperature spatial distribution features and the video global feature representations to obtain fusion features, and input the fusion features into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network.

[0135] The device status determination module is used to determine the device status of the power distribution equipment under inspection based on the historical hidden state sequence, the fusion features, and edge devices pre-deployed in the local network domain of the power distribution equipment under inspection.

[0136] See Figure 3 The power distribution equipment status detection device based on multimodal data fusion further includes: an equipment topology map update module;

[0137] The device topology map update module is used to map the final fused features to the node features corresponding to the preset device topology map using a preset graph neural network, so as to obtain an updated device topology map; wherein, the device topology map includes a number of power distribution equipment nodes and the connection relationships between each power distribution equipment node.

[0138] See Figure 4 One embodiment of this application also provides a terminal device, including:

[0139] One or more processors;

[0140] A memory, coupled to the processor, for storing one or more programs;

[0141] When the one or more programs are executed by the one or more processors, the one or more processors implement the power distribution equipment status detection method based on multimodal data fusion as described above.

[0142] The processor controls the overall operation of the terminal device to complete all or part of the steps of the aforementioned power distribution equipment status detection method based on multimodal data fusion. The memory stores various types of data to support the operation of the terminal device. This data may include, for example, instructions for any application or method operating on the terminal device, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0143] In an exemplary embodiment, the terminal device can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements, for executing the power distribution equipment state detection method based on multi-modal data fusion as described in any one of the above embodiments, and achieving the consistent technical effects as the above method.

[0144] In another exemplary embodiment, a computer readable storage medium including a computer program is also provided, which, when executed by a processor, implements the steps of the power distribution equipment state detection method based on multi-modal data fusion as described in any one of the above embodiments. For example, the computer readable storage medium can be the above-mentioned memory including the computer program, which can be executed by the processor of the terminal device to complete the power distribution equipment state detection method based on multi-modal data fusion as described in any one of the above embodiments, and achieve the consistent technical effects as the above method.

[0145] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements are also considered to be within the scope of protection of the present application.

Claims

1. A power distribution equipment state detection method based on multi-modal data fusion, characterized in that, The method comprises the following steps: acquiring real-time operation data, an infrared image and an operation video file of a power distribution device to be inspected; extracting local abnormal fluctuation features in the time sequence window samples according to a preset time sequence convolution network and the real-time operation data; capturing infrared temperature spatial distribution features corresponding to the infrared image by using a preset residual neural network and a preset Transformer network; extracting spatial features of each frame of image in the operation video file by using the residual neural network to obtain spatial feature representations corresponding to each frame of image, and obtaining video global feature representations containing time sequence dynamic information and spatial semantic information according to a preset three-dimensional convolution network, the time sequence convolution network and the spatial feature representations corresponding to each frame of image; concatenating the local abnormal fluctuation features, the infrared temperature spatial distribution features and the video global feature representations to obtain fusion features, and inputting the fusion features into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network; determining a device state of the power distribution device to be inspected according to the historical hidden state sequence, the fusion features and an edge device pre-deployed in a local network domain of the power distribution device to be inspected.

2. The power distribution equipment condition detection method based on multi-modal data fusion as claimed in claim 1, wherein, The method of extracting local abnormal fluctuation features in the time sequence window samples according to a preset time sequence convolution network and the real-time operation data comprises the following steps: dividing the real-time operation data into a plurality of time sequence window samples according to time sequence features in the real-time operation data; extracting local abnormal fluctuation features in the time sequence window samples by using a preset time sequence convolution network to extract local features of the time sequence window samples.

3. The power distribution equipment condition detection method based on multi-modal data fusion as claimed in claim 1, wherein, The method of capturing infrared temperature spatial distribution features corresponding to the infrared image by using a preset residual neural network and a preset Transformer network comprises the following steps: performing pixel normalization processing on the infrared image to obtain an infrared image after pixel normalization processing; inputting the infrared image after pixel normalization processing into the preset residual neural network to obtain initial features of the infrared image; inputting the initial features of the infrared image into the preset Transformer network to obtain infrared temperature spatial distribution features captured by the Transformer network.

4. The power distribution equipment condition detection method based on multi-modal data fusion as claimed in claim 1, wherein, The method of obtaining video global feature representations containing time sequence dynamic information and spatial semantic information according to a preset three-dimensional convolution network, the time sequence convolution network and the spatial feature representations corresponding to each frame of image comprises the following steps: sorting the spatial feature representations corresponding to each frame of image in the operation video file in time sequence to form a feature sequence; extracting a plurality of time sequence feature segments by using a preset sliding window to intercept the feature sequence; capturing time sequence dynamic features by using a preset three-dimensional convolution network to capture inter-frame time sequence dynamic features of each time sequence feature segment; inputting the time sequence dynamic features into the time sequence convolution network to determine video global feature representations containing time sequence dynamic information and spatial semantic information.

5. The power distribution equipment condition detection method based on multi-modal data fusion as claimed in claim 1, wherein, The device state of the to-be-inspected power distribution equipment is determined according to the historical hidden state sequence, the fusion feature, and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution equipment. The historical hidden state sequence and the fusion feature are integrated into a final fusion feature, and the final fusion feature is input into an edge device deployed in a local network domain of the to-be-inspected power distribution equipment, so that the edge device outputs the device state of the to-be-inspected power distribution equipment based on network parameters issued by a pre-trained completed federated learning system.

6. The power distribution equipment condition detection method based on multi-modal data fusion as claimed in claim 5, wherein, Before the final fusion feature is input into the edge device deployed in the local network domain of the to-be-inspected power distribution equipment, the method further includes: The final fusion feature is mapped into node features corresponding to a preset device topology graph by using a preset graph neural network, to obtain an updated device topology graph; the device topology graph includes a plurality of power distribution equipment nodes and connection relationships between the power distribution equipment nodes.

7. A power distribution equipment state detection device based on multi-modal data fusion, characterized in that, It includes: a data acquisition module, a local abnormal fluctuation feature extraction module, an infrared temperature spatial distribution feature capturing module, a video global feature representation determination module, a historical hidden state sequence determination module, and a device state determination module; The data acquisition module is configured to acquire real-time operation data, an infrared image, and an operation video file of the to-be-inspected power distribution equipment. The local abnormal fluctuation feature extraction module is configured to extract local abnormal fluctuation features in the time sequence window sample according to a preset time sequence convolution network and the real-time operation data. The infrared temperature spatial distribution feature capturing module is configured to capture infrared temperature spatial distribution features corresponding to the infrared image by using a preset residual neural network and a preset Transformer network. The video global feature representation determination module is configured to perform spatial feature extraction on each frame of image in the operation video file by using the residual neural network, to obtain spatial feature representations corresponding to the frames of image, and to obtain video global feature representations containing time sequence dynamic information and spatial semantic information according to a preset three-dimensional convolution network, the time sequence convolution network, and the spatial feature representations corresponding to the frames of image. The historical hidden state sequence determination module is configured to splice the local abnormal fluctuation features, the infrared temperature spatial distribution features, and the video global feature representations to obtain fusion features, and to input the fusion features into a preset long short-term memory network to obtain a historical hidden state sequence output by the long short-term memory network. The device state determination module is configured to determine the device state of the to-be-inspected power distribution equipment according to the historical hidden state sequence, the fusion feature, and an edge device pre-deployed in a local network domain of the to-be-inspected power distribution equipment.

8. The power distribution equipment condition detection apparatus based on multi-modal data fusion as claimed in claim 7, wherein, It further includes a device topology graph updating module. The device topology graph updating module is configured to map the final fusion feature into node features corresponding to a preset device topology graph by using a preset graph neural network, to obtain an updated device topology graph; the device topology graph includes a plurality of power distribution equipment nodes and connection relationships between the power distribution equipment nodes.

9. A terminal device, comprising: It includes: one or more processors; a memory coupled with the processors, storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors realize the power distribution equipment state detection method based on multi-modal data fusion as claimed in any one of claims 1-6.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the power distribution equipment state detection method based on multi-modal data fusion as claimed in any one of claims 1-6.

Images