Insulator defect detection method based on multi-modal feature fusion

Abstract

The invention relates to an image processing technology, in particular to an insulator defect detection method based on multi-modal feature fusion. Performing image enhancement on the RGB picture to obtain an event frame corresponding to the RGB picture; performing dual-channel feature extraction, and processing the event features and the RGB features by using an ERFM feature fusion method to obtain fused features; processing the fused features by using a feature attention module to obtain enhanced features; training an insulator defect detection network based on multi-modal feature fusion by using the enhanced features; and inputting the corresponding RGB picture and the event frame into an insulator defect detection network to generate an insulator identification result graph marked with defects. According to the method, data feature fusion of two modes is adopted to cover more comprehensive information, so that the defect detection model generated by training can keep excellent performance in various complex environments, the environment adaptability of the detection model is greatly enhanced, the conditions of false detection and missing detection are reduced, and the insulator defect detection precision is ensured.

Description

Technical Field

[0001] The invention relates to image processing technology, and in particular to an insulator defect detection method based on multi-modal feature fusion. Background Art

[0002] Power scene detection is a hot topic in the field of computer vision. Its core goal is to use various sensors to conduct comprehensive defect detection on power equipment and lines through deep learning technology to ensure the safe and stable operation of the power system. In order to ensure that the power system is in a stable operating state, regular inspections of insulators are essential. One of the main tasks of these inspections is to identify and detect potential defects in insulators, thereby preventing power accidents caused by equipment failures. Therefore, defect detection under insulators is of great significance to ensure the safety and reliability of transmission lines.

[0003] At present, insulator defect detection mostly relies on drones equipped with traditional sensors for inspection. This method has certain advantages, such as wide coverage, high efficiency, and the ability to reach areas that are difficult to reach manually. However, traditional sensors also face many challenges in practical applications. First, due to the limitations of the working principles and technical characteristics of traditional sensors, their performance is often unsatisfactory in complex lighting environments. For example, in an electric power scene with the sky as the background, the lighting conditions change dramatically, and strong exposure and backlighting may occur. In this case, it is difficult for the sensor to capture a clear image of the insulator, resulting in reduced detection accuracy.

[0004] In addition, the high-speed movement of drones is also a factor that cannot be ignored. During the inspection process, drones need to fly at high speed to improve inspection efficiency and coverage area, but this also brings problems of image blur and jitter. Especially when facing complex terrain and environment, the flight posture of drones will constantly change, further causing problems of image blur and jitter. These problems combined make the defect detection model based on traditional sensors lack sufficient maturity and reliability in practical applications, seriously affecting its practical application value. Summary of the invention

[0005] In order to solve the problem of limitations of existing defect detection under insulators, the present invention provides an insulator defect detection method based on multi-modal feature fusion.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] The insulator defect detection method based on multimodal feature fusion is implemented by the following steps:

[0008] Step S1: Perform image enhancement on the RGB image collected by the traditional vision sensor through geometric transformation to obtain a processed RGB image; accumulate and preprocess the events collected by the neuromorphic vision sensor to obtain an event frame corresponding to the RGB image collected by the traditional vision sensor;

[0009] Step S2: performing dual-channel feature extraction based on the processed RGB image and event frame, and performing feature extraction and feature enhancement on the event feature extraction channel and the RGB feature extraction channel respectively through a convolution module to obtain event features and RGB features;

[0010] Step S3: Using the ERFM feature fusion method to process event features and RGB features to obtain fused event and RGB features;

[0011] Step S4: Use the feature attention module to process the fused event and RGB features to obtain enhanced event and RGB features;

[0012] Step S5: constructing a multi-modal feature fusion insulator defect detection network, and using the enhanced event and RGB features to train the multi-modal feature fusion insulator defect detection network;

[0013] Step S6: Input any two corresponding RGB images and event frames that have not participated in the training into the trained multimodal feature fusion insulator defect detection network to generate an insulator recognition result map with marked defects.

[0014] Furthermore, in step S1, it specifically includes:

[0015] Use neuromorphic vision sensors and traditional vision sensors to collect event data and RGB images respectively;

[0016] The RGB images collected by traditional vision sensors are enhanced through geometric transformation, and the event data collected by neuromorphic vision sensors are accumulated to generate event frames, with an accumulation time of 33ms per frame;

[0017] The generated event frames are preprocessed and cropped to event frames of the same size as the RGB images, thereby obtaining event frames corresponding to the RGB images collected by traditional visual sensors.

[0018] Furthermore, in step S2, it specifically includes:

[0019] The preprocessed event frame is used as the input of the event feature extraction channel. The Sobel operator is first used to detect the edges in the event frame by calculating the gradient of the event frame in the horizontal and vertical directions and finding the part with the most drastic brightness change in the event frame.

[0020] Let the input event frame be I, and its pixel value can be expressed as I(x,y), where x and y are the coordinates of the event frame. The Sobel operator uses two 3x3 convolution kernels, one for calculating the horizontal gradient G x , and the other one is used to calculate the vertical gradient G y . Horizontal Sobel convolution kernel Vertical Sobel convolution kernel

[0021]

[0022]

[0023] Secondly, use G x and G y Calculate the gradient magnitude G of each pixel:

[0024]

[0025] The gradient magnitude G(x,y) is stored in a new image matrix to obtain the gradient magnitude image of the event frame. The value of each pixel (x,y) represents the gradient strength at that location. The gradient magnitude image is normalized to obtain the edge enhanced image:

[0026]

[0027] Among them, G max and G min Represents the maximum and minimum values ​​in the gradient magnitude image. Normalizing the gradient magnitude image can better display the edge information of the insulator, making the edge features of the insulator more obvious and easier to analyze.

[0028] Secondly, based on the edge-enhanced image of the event frame, the Conv convolution module is used to extract the initial event features f E .

[0029] f E =Conv(Event) (5)

[0030] For the RGB feature extraction channel, the RGB initial feature f is directly extracted through the Conv convolution module based on the preprocessed RGB image. R .

[0031] f R =Conv(RGB) (6)

[0032] Based on the initial feature f of the event E and RGB initial feature f R , the following neural network module is used to extract the final feature F Eand F R .

[0033] Specifically, firstly based on the feature f E and f R , apply Conv convolution and split into two parts along the channel dimension:

[0034]

[0035]

[0036] Among them, chunk is a function used to split a tensor into multiple sub-tensors. Then and Apply n Bottleneck modules, each module processes the features to obtain a series of new features:

[0037]

[0038]

[0039] Where i represents the output feature of the i-th Bottleneck module. Bottleneck module is a structural module used in deep neural networks, mainly used to increase the depth and complexity of the network while reducing the amount of calculation and the number of parameters.

[0040] Will and Combine the features processed by all Bottleneck modules separately:

[0041]

[0042]

[0043] For the fused feature Y E and Y R Perform 1×1 convolution to obtain the final feature F E and F R .

[0044] Furthermore, in step S3, it specifically includes:

[0045] F E and F R After feature fusion by the Event and RGB Feature Fusion Module (ERFM), the fused feature F is obtained. The fused feature F has better representation ability and richer information content. The formula is as follows:

[0046] F=ERFM(FE ,F R ) (13)

[0047] The specific ERFM method first uses a 1*1 Conv convolutional network to perform feature F E and F R Activate to get the activated feature F Event and F RGB , the feature size is N×D, where N represents the feature sequence length and D represents the dimension of each sequence element.

[0048] F Event =Conv 1*1 (F E ) (14)

[0049] F RGB =Conv 1*1 (F R ) (15)

[0050] By assigning weights to the spatial attention module features FEvent and FRGB, the key areas are highlighted. First, the FEvent and FRGB features are max-pooled and average-pooled in the channel dimension:

[0051]

[0052]

[0053]

[0054]

[0055] Among them, F maxE and F avgE Yes F Event Maximum pooling and average pooling results, F maxR and F avgR Yes F RGB The maximum pooling and average pooling results. The channel aggregation results are then fused and the spatial attention weights are generated through 1×1 convolution:

[0056] S E =σ(Conv 1*1 (concat(F maxE ,F avgE )) (20)

[0057] S R =σ(Conv 1*1 (concat(F maxR ,F avgR )) (twenty one)

[0058] Among them, σ is the activation function.

[0059] Apply the generated spatial attention weights to the features F Event and F RGB The enhanced feature F EventSA and F RGBSA :

[0060] F EventSA =S E ⊙F Event (twenty two)

[0061] F RGBSA =S E ⊙F RGB (twenty three)

[0062] Through the channel attention module, the feature F EventSA and F RGBSA Assign weights to each channel of F to highlight the importance of a specific channel. First, EventSA and F RGBSA Perform global average pooling to obtain a global description of each channel:

[0063]

[0064]

[0065] Among them, F Eavg and F Ravg It is the description vector in the channel dimension.

[0066] The result of global average pooling is passed through a fully connected layer to generate channel attention weights:

[0067] W E =σ(FC2(ReLU(FC1(F Eavg ))) (26)

[0068] W R =σ(FC2(ReLU(FC1(F Ravg ))) (27)

[0069] Among them, FC1 and FC2 are fully connected layers, σ and ReLU are activation functions. The generated channel attention weights are then applied to the input feature F EventSA and F RGBSA , get the weighted feature F AEvent and F ARGB :

[0070] F CAEvent =W E ⊙F EventSA (28)

[0071] F CARGB =W R ⊙F RGBSA (29)

[0072] Step S36: Merge into one feature through Concat operation:

[0073] F = Concat(F CAEvent ,F CARGB ) (30)

[0074] Among them, F is the fused feature, which contains the fused and enhanced insulator feature information, with better characterization ability and richer information content.

[0075] The fused feature F is used as the initial feature f of the event in the subsequent stage E , F R As the initial RGB feature f in the subsequent stage R , iterate formula (7-13) 3 times, and the feature F obtained after 3 iterations is the final fusion feature F final .

[0076] Further, in step S4, based on the fused feature F final , the feature attention module is introduced for further feature enhancement, which mainly includes two parts: compression layer and excitation layer, specifically including:

[0077] The compression layer is used to combine the final fusion features F final Perform global average pooling to obtain the global compressed feature value of the current feature:

[0078]

[0079] Among them, z is the compressed feature vector with a dimension of 1×D.

[0080] The excitation layer uses a two-layer fully connected bottleneck structure to obtain the weight of each channel, and then uses the weighted features as subsequent input, enabling the network to more effectively extract the location information and deep semantic information in the lower convolutional layer.

[0081] The calculation formula of the first fully connected layer is:

[0082] z1=σ(W1z2+b1) (32)

[0083] The calculation formula of the second fully connected layer is:

[0084] s=σ(W2z2+b2) (33)

[0085] Where W is the weight matrix, W1 dimension is The dimension of W2 is Where r is the compression ratio. b is the bias vector, and b1 has the dimension The dimension of b2 is 1 × D. σ is the activation function, and the resulting weight vector s has a dimension of 1 × D.

[0086] The fused feature F final Weighted with the weight vector s to obtain the weighted feature F weighted :

[0087] F weighted (i) = F final (i)⊙s (34)

[0088] Among them, ⊙ represents element-by-element multiplication, F weighted It is the feature obtained after introducing the feature attention module for further feature enhancement.

[0089] Furthermore, in step S5, the insulator defect detection network training step based on multimodal feature fusion specifically includes:

[0090] Use event and RGB dual feature extraction channels to extract event features F E and RGB feature F R ;

[0091] Based on event features E and RGB feature F R , use the ERFM feature fusion module to perform feature fusion to obtain the fused feature F final ;

[0092] Based on the fused feature F final , introduce the feature attention module for further feature enhancement, and then enhance the fused features through the attention module to obtain F weighted ;

[0093] The feature F extracted and enhanced by the detection head weighted Perform final insulator classification and bounding box regression;

[0094] Furthermore, in step S6, it specifically includes:

[0095] Input any two corresponding RGB images and event frames that have not participated in the training;

[0096] Based on the trained insulator defect detection network, feature extraction, fusion and enhancement are performed on the input data;

[0097] Based on the enhanced fusion features, the insulator defect detection network performs the final insulator classification and bounding box regression to generate an insulator recognition result map with marked defects.

[0098] The insulator defect detection method based on multimodal feature fusion described in the present invention uses computer vision and deep neural network models as technical support to achieve a high-precision defect detection method under insulators.

[0099] Beneficial effects of the present invention:

[0100] Compared with the existing defect detection methods, the present invention takes into account the limitation that single modality data cannot cover all the information required for the detection task in special scenarios. This limitation will lead to the problem of low defect recognition accuracy. Therefore, the present invention introduces events collected by neuromorphic visual sensors as one of the data sources, and proposes a multimodal fusion technology using event features and RGB features for defect detection under insulators. The fusion of data features of the two modalities covers more comprehensive information, so that the defect detection model generated by training can maintain excellent performance in various complex environments, greatly enhance the environmental adaptability of the detection model, reduce false detection and missed detection, and ensure the accuracy of insulator defect detection. It has broad application prospects and important practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 It is a schematic diagram of the insulator defect detection structure based on multi-modal feature fusion according to an embodiment of the present invention.

[0102] Figure 2 Schematic diagram of a dual-branch feature extraction structure according to an embodiment of the present invention.

[0103] Figure 3 It is a schematic diagram of the ERFM feature fusion structure of an embodiment of the present invention.

[0104] Figure 4 It is a schematic diagram of the feature attention enhancement structure of an embodiment of the present invention.

[0105] Figure 5 It is a schematic diagram of the recognition effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0106] In order to make the purpose, technical scheme and advantages of the present invention clearer and more complete, the present invention will be further described in detail below in conjunction with the embodiments and drawings. It should be noted that the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely, and to fully convey the scope of the present invention to those skilled in the art.

[0107] In this embodiment, the insulator data set used is from the public data sets CPLID, SFID, and IDID.

[0108] like Figure 1As shown in FIG. 1 , an insulator defect detection method based on multimodal feature fusion is implemented by the following steps:

[0109] Step S1: Perform image enhancement on the RGB image collected by the traditional vision sensor through geometric transformation to obtain a processed RGB image; accumulate and preprocess the events collected by the neuromorphic vision sensor to obtain an event frame corresponding to the RGB image collected by the traditional vision sensor;

[0110] Step S2: performing dual-channel feature extraction based on the processed RGB image and event frame, and performing feature extraction and feature enhancement on the event feature extraction channel and the RGB feature extraction channel respectively through a convolution module to obtain event features and RGB features;

[0111] Step S3: Using the ERFM feature fusion method to process event features and RGB features to obtain fused event and RGB features;

[0112] Step S4: Use the feature attention module to process the fused event and RGB features to obtain enhanced event and RGB features;

[0113] Step S5: constructing a multi-modal feature fusion insulator defect detection network, and using the enhanced event and RGB features to train the multi-modal feature fusion insulator defect detection network;

[0114] Step S6: Input any two corresponding RGB images and event frames that have not participated in the training into the trained multimodal feature fusion insulator defect detection network to generate an insulator recognition result map with marked defects.

[0115] In the step S1, image acquisition and image preprocessing are performed.

[0116] Since neuromorphic visual sensors have only been developed and commercialized in recent years, the currently available power event data is very limited. In addition, since insulators are distributed on overhead transmission lines, they are limited by the distribution range and distribution environment of overhead transmission lines. Using neuromorphic visual sensors to reprint existing RGB images is more convenient and safer than real-life photography. Therefore, the present invention obtains experimental data by reprinting RGB images using neuromorphic visual sensors.

[0117] The RGB images are selected from the public datasets CPLID, SFID, and IDID. The experimental equipment includes a neuromorphic visual sensor, a camera stand, a computer connected to the camera, and an LCD monitor for displaying and playing RGB images. The recording system includes a hardware system and a software system. The core of the hardware system is the neuromorphic visual sensor, which converts moving objects into event streams. The sensor is mounted on a stand and aligned with the vertical center of the display. The recording system is placed in a dark place to suppress ambient stray light.

[0118] After the data collection is completed, the event data is accumulated to generate event frames, and the accumulation time for each frame is 33ms; finally, the generated event frames are enhanced and cropped to be event frames of the same size as the RGB image.

[0119] In step S2, a dual-channel feature extraction method based on events and RGB is proposed. A dual-branch and inconsistent design is used to extract semantic features from the two modalities based on the edge feature information of event modal data and the texture feature characteristics of RGB modal data. The extraction branch structure is as follows:

[0120] Since single modal data cannot cover all the information required for insulator defect detection in special scenarios, the present invention designs an event frame feature extraction branch and an RGB feature extraction branch with reference to the backbone module of YOLOv8, and improves the defect detection accuracy by integrating event and RGB modal data.

[0121] RGB images can effectively display the texture features of insulators under normal lighting conditions, while event frames have strong feature contrast in weak light environments and can clearly show the edge features of insulators, thus making up for the shortcomings of RGB images in identifying insulator defects under weak light conditions.

[0122] In order to effectively obtain the edge characteristics and texture features of the insulator, the present invention proposes a dual-channel feature extraction method based on events and RGB. Figure 2 As shown in the figure, the event feature extraction channel uses the clear contour characteristics of the event image to improve the feature extraction effect through the edge enhancement module; the RGB feature extraction channel uses the rich texture information in the RGB image to extract and enhance features through a specific convolution module.

[0123] For the event feature extraction channel, the edge enhancement module is first introduced to better display the edge information of the insulator, and then the convolution module is used to extract the initial features of the event. Specifically, first, based on the preprocessed event frame, the Sobel operator is used to detect the edges in the event frame by calculating the gradients of the event frame in the horizontal and vertical directions to find the part with the most drastic brightness change in the event frame. The detailed calculation is: let the input event frame be I, and its pixel value can be expressed as I(x,y), where x and y are the coordinates of the event frame. The Sobel operator uses two 3x3 convolution kernels, one for calculating the gradient Gx in the horizontal direction and the other for calculating the gradient Gy in the vertical direction. Horizontal Sobel convolution kernel Vertical Sobel convolution kernel

[0124]

[0125]

[0126] Secondly, use G x and G y Calculate the gradient magnitude G of each pixel:

[0127]

[0128] The gradient magnitude G(x,y) is stored in a new image matrix to obtain the gradient magnitude image of the event frame. The value of each pixel (x,y) represents the gradient strength at that location. The gradient magnitude image is normalized to obtain the edge enhanced image:

[0129]

[0130] Among them, G max and G min Represents the maximum and minimum values ​​in the gradient magnitude image. Normalizing the gradient magnitude image can better display the edge information of the insulator, making the edge features of the insulator more obvious and easier to analyze. Secondly, based on the edge enhanced image of the event frame, the Conv convolution module is used to extract the initial feature f of the event E .

[0131] f E =Conv(Event) (5)

[0132] For the RGB feature extraction channel, the RGB initial feature f is directly extracted through the Conv convolution module based on the preprocessed RGB image. R .

[0133] f R =Conv(RGB) (6)

[0134] Next, based on the initial feature f E and RGB initial feature f R , the following neural network module is used to extract the final feature F E and F R Specifically, firstly based on the feature f E and f R , apply Conv convolution and split into two parts along the channel dimension:

[0135]

[0136]

[0137] Among them, chunk is a function used to split a tensor into multiple sub-tensors. Then and Apply n Bottleneck modules, each module processes the features to obtain a series of new features:

[0138]

[0139]

[0140] Where i represents the output feature of the i-th Bottleneck module. Bottleneck module is a structural module used in deep neural networks, mainly used to increase the depth and complexity of the network while reducing the amount of calculation and the number of parameters.

[0141] Finally, and Combine the features processed by all Bottleneck modules separately:

[0142]

[0143]

[0144] For the fused feature Y E and Y R Perform 1×1 convolution to obtain the final feature F E and F R .

[0145] In step S3, the method for fusing the features of different sensors is as follows:

[0146] Event frames have strong feature contrast characteristics in low-light environments, and RGB images have unique texture characteristics under normal lighting conditions. Therefore, single-modality data cannot cover all feature information in special environments. In traditional research, event and RGB feature fusion methods usually merge multimodal features through simple concatenated convolutions, or use attention mechanisms to focus on channel dimensions. These methods fail to explicitly process important spatial cues in detection tasks and fail to fully utilize the complementary characteristics between RGB and event data.

[0147] Considering this problem, Figure 3 As shown, the present invention proposes an ERFM feature fusion method to comprehensively process features from different sensors, so that the Even frame and RGB image can achieve the best feature complementarity and fusion effect.

[0148] F E and F R The fused feature F is obtained by the event and RGB feature fusion module ERFM. The formula is as follows:

[0149] F=ERFM(F E ,F R ) (13)

[0150] The specific ERFM mainly includes 6 steps: 1) Obtain activation feature F Event and F RGB ; 2) is feature F Event and F RGB Assign spatial attention weights; 3) Apply spatial attention weights to F Event and F RGB The enhanced feature F EventSA and F RGBSA ; 4) F EventSA and F RGBSA Generate channel attention weights; 5) Apply the channel attention weights to the enhanced features F EventSA and F RGBSA The features get the weighted features F AEvent and F ARGB ; 6) Use Concat operation to merge F AEvent and F ARGB Get the fused feature F.

[0151] The specific implementation is:

[0152] First, use a 1*1 Conv convolutional network to train the feature F E and F R Activate to get the activated feature F Event and F RGB, the feature size is N×D, where N represents the feature sequence length and D represents the dimension of each sequence element.

[0153] F Event =Conv 1*1 (F E ) (14)

[0154] F RGB =Conv 1*1 (F R ) (15)

[0155] Next, through the spatial attention module feature F Event and F RGB Assign weights to highlight key areas. First, perform maximum pooling and average pooling on the FEvent and FRGB features in the channel dimension:

[0156]

[0157]

[0158]

[0159]

[0160] Among them, F maxE and F avgE Yes F Event Maximum pooling and average pooling results, F maxR and F avgR Yes F RGB Max pooling and average pooling results. The results of max pooling and average pooling are fused and the spatial attention weights are generated through 1×1 convolution:

[0161] S E =σ(Conv 1*1 (concat(F maxE ,F avgE )) (20)

[0162] S R =σ(Conv 1*1 (concat(F maxR ,F avgR )) (twenty one)

[0163] Where σ is the activation function. The generated spatial attention weights are applied to the features F Event and F RGB The enhanced feature F EventSA and F RGBSA :

[0164] FEventSA =S E ⊙F Event (twenty two)

[0165] F RGBSA =S E ⊙F RGB (twenty three)

[0166] Afterwards, the channel attention module is used to add EventSA and F RGBSA Assign weights to each channel of F to highlight the importance of a specific channel. EventSA and F RGBSA Perform global average pooling to obtain a global description of each channel:

[0167]

[0168]

[0169] Among them, F Eavg and F Ravg is the description vector on the channel dimension. The result of global average pooling is passed through the fully connected layer to generate the channel attention weight:

[0170] W E =σ(FC2(ReLU(FC1(F Eavg ))) (26)

[0171] W R =σ(FC2(ReLU(FC1(F Ravg ))) (27)

[0172] Among them, FC1 and FC2 are fully connected layers, σ and ReLU are activation functions. The generated channel attention weights are applied to the input features F EventSA and F RGBSA , get the weighted feature F AEvent and F ARGB :

[0173] F CAEvent =W E ⊙F EventSA (28)

[0174] F CARGB =W R ⊙F RGBSA (29)

[0175] Finally, the Concat operation is used to merge them into one feature:

[0176] F = Concat(F CAEvent ,F CARGB) (30)

[0177] Among them, F is the fused feature, which contains the fused and enhanced insulator feature information, with better characterization ability and richer information content.

[0178] Finally, the fused feature F is used as the initial feature f of the event in the subsequent stage E , F R As the initial RGB feature f in the subsequent stage R , iterate formula (7-13) 3 times, and the feature F obtained after 3 iterations is the final fusion feature F final .

[0179] In step S4, the feature attention enhancement module applies the feature attention method to the fused features, so that the model can automatically learn which fused features are more critical, thereby enhancing the attention to these key features and effectively improving the performance of the model in complex tasks.

[0180] like Figure 4 As shown, the present invention further introduces a feature attention enhancement module to enhance the fused features. The feature attention enhancement module mainly includes two parts: a compression layer and an excitation layer. The specific implementation is:

[0181] The compression layer is used to combine the final fusion features F final Perform global average pooling to obtain the global compressed feature value of the current feature.

[0182]

[0183] Among them, z is the compressed feature vector with a dimension of 1×D.

[0184] The excitation layer uses a two-layer fully connected bottleneck structure to obtain the weight of each channel, and then uses the weighted features as the input of the next layer, so that the network can more effectively extract the location information and deep semantic information in the lower convolution layer. The first fully connected layer compresses the channel to reduce the amount of calculation, and the second fully connected layer restores the number of channels.

[0185] The calculation formula of the first fully connected layer is:

[0186] z1=σ(W1z2+b1) (32)

[0187] The calculation formula of the second fully connected layer is:

[0188] s=σ(W2z2+b2) (33)

[0189] Where W is the weight matrix, W1 dimension is The dimension of W2 is Where r is the compression ratio. b is the bias vector, and b1 has the dimension The dimension of b2 is 1 × D. σ is the activation function, and the resulting weight vector s has a dimension of 1 × D.

[0190] Finally, the fused feature F final Weighted with the weight vector s to obtain the weighted feature F weighted :

[0191] F weighted (i) = F final (i)⊙s (34)

[0192] Among them, ⊙ represents element-by-element multiplication, F weighted It is the feature obtained after introducing the feature attention module for further feature enhancement.

[0193] In the step S5, the insulator defect detection network training based on multimodal feature fusion is completed to improve the defect recognition accuracy, thereby ensuring the safe and stable operation of the power system.

[0194] At present, insulator defect detection mostly relies on drones equipped with traditional sensors for inspection. This method has certain advantages, such as wide coverage, high efficiency, and the ability to reach areas that are difficult to reach manually. However, due to the drastic changes in light background with the sky and the high-speed movement of the drone itself, it is impossible to achieve reasonable performance in complex lighting environments such as exposure and backlighting. Traditional sensors also face many challenges in practical applications, which makes the defect detection model based on traditional sensors lack sufficient maturity and reliability in practical applications, seriously affecting its practical application value.

[0195] The present invention proposes an insulator defect detection method based on multimodal feature fusion, introduces events collected by neuromorphic vision sensors as one of the data sources, and proposes a multimodal fusion technology using event features and RGB features to detect defects under insulators. Event and RGB dual feature extraction channels are used to extract features of two different data, event and RGB, and the detection accuracy is improved by complementing the RGB and event frame feature information. A new feature fusion module is proposed to integrate feature information from different sensors, perform feature fusion after feature preprocessing, and enable event and RGB images to achieve the best feature complementarity and fusion effect. The fused features are then enhanced through the attention module. Finally, the extracted features are subjected to final insulator classification and bounding box regression through the detection head;

[0196] The training process generates parameters such as accuracy, recall, and mAP to help fully understand the performance of the model. When the loss function in the training process tends to be smooth, the training of the insulator defect detection network based on multimodal feature fusion is completed.

[0197] In step S6, based on the trained insulator defect detection network model based on multimodal feature fusion, any two corresponding RGB images and event frames that have not participated in the training are input to generate an insulator recognition result with marked defects.

[0198] like Figure 5 As shown in the figure, firstly, any two corresponding RGB images and event frames that have not participated in the training are input; then, based on the trained insulator defect detection network, the input data is feature extracted, fused and enhanced; then, based on the enhanced fusion features, the insulator defect detection network performs the final insulator classification and bounding box regression, thereby generating an insulator recognition result map with marked defects.

[0199] This paper introduces events collected by neuromorphic vision sensors as one of the data sources, and proposes a multimodal fusion technology using event features and RGB features to detect defects under insulators. The fusion of data features of the two modalities covers more comprehensive information, so that the defect detection model generated by training can maintain excellent performance in various complex environments. Figure 2 As shown in the results in the red box, the present invention improves the detection accuracy of the insulator detection model, reduces false detections and missed detections, and ensures the accuracy of insulator defect detection.

[0200] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An insulator defect detection method based on multimodal feature fusion, characterized in that: The following steps are involved: Step S1: Perform image enhancement on the RGB image collected by the traditional visual sensor through geometric transformation to obtain a processed RGB image; Accumulate and preprocess the events collected by the neuromorphic vision sensor to obtain event frames corresponding to the RGB images collected by the traditional vision sensor; Step S2: performing dual-channel feature extraction based on the processed RGB image and event frame, and performing feature extraction and feature enhancement on the event feature extraction channel and the RGB feature extraction channel respectively through a convolution module to obtain event features and RGB features; Step S3: Using the ERFM feature fusion method to process event features and RGB features to obtain fused event and RGB features; Step S4: Use the feature attention module to process the fused event and RGB features to obtain enhanced event and RGB features; Step S5: constructing a multi-modal feature fusion insulator defect detection network, and using the enhanced event and RGB features to train the multi-modal feature fusion insulator defect detection network; Step S6: Input any two corresponding RGB images and event frames that have not participated in the training into the trained multimodal feature fusion insulator defect detection network to generate an insulator recognition result map with marked defects.

2. The insulator defect detection method based on multimodal feature fusion according to claim 1 is characterized in that: Step S1 specifically includes: Obtain insulator event data and RGB images collected by neuromorphic vision sensors and traditional vision sensors respectively; The RGB image is enhanced by translation, scaling, mirroring, and rotation to obtain the processed RGB image. At the same time, the event data is accumulated to generate event frames, and the accumulation time for each frame is 33ms. The generated event frames are preprocessed and cropped to event frames of the same size as the processed RGB picture image.

3. The insulator defect detection method based on multimodal feature fusion according to claim 1 is characterized in that: Step S2 specifically includes: The event feature extraction channel uses the Sobel operator to calculate the gradient of the event frame in the horizontal and vertical directions, finds the part with the most dramatic brightness change in the event frame, and detects the edge in the event frame: Let the input event frame be I, and the pixel value can be expressed as I(x,y), where x and y are the coordinates of the event frame; The Sobel operator uses two 3x3 convolution kernels, one for calculating the horizontal gradient G x , and the other one is used to calculate the vertical gradient G y ; Horizontal Sobel convolution kernel Vertical Sobel convolution kernel Using G x (x,y) and G y (x, y) calculates the gradient magnitude G(x, y) of each pixel: The gradient magnitude G(x,y) is stored in the image matrix to obtain the gradient magnitude image of the event frame. The value of each pixel point (x,y) represents the gradient strength at that position. Normalize the gradient magnitude image to obtain the edge enhanced image: Among them, G max and G min Represents the maximum and minimum values ​​in the gradient magnitude image; Based on the edge-enhanced image of the event frame, the Conv convolution module is used to extract the initial event feature f E : f E =Conv(Event) (5) The RGB feature extraction channel directly extracts the RGB initial feature f through the Conv convolution module. R : f R =Conv(RGB) (6)。 4. The insulator defect detection method based on multimodal feature fusion according to claim 2 is characterized in that: Step S2 specifically includes: Based on the initial feature f E and RGB initial feature f R , apply Conv convolution and split into two parts along the channel dimension: Among them, chunk is a function used to split a tensor into multiple sub-tensors; For Y1 2 and Y2 2 Apply n Bottleneck modules respectively, each module processes the features to obtain a series of new features: Where i represents the output feature of the i-th Bottleneck module; Y1 1 and Y2 1 Combine the features processed by all Bottleneck modules separately: For the fused feature Y E and Y R Perform 1×1 convolution to obtain the final event feature F E and RGB feature F R .

5. The insulator defect detection method based on multi-modal feature fusion according to claim 4 is characterized in that: In step S3, the fused event and RGB features are obtained, and the formula is as follows: F=ERFM(F E ,F R ) (13) Use 1*1 Conv convolutional network to analyze event feature F E and RGB feature F R Activate and get the activated feature F Event and F RGB , the feature size is N×D, where N represents the feature sequence length and D represents the dimension of each sequence element: F Event =Conv 1*1 (F E ) (14) F RGB =Conv 1*1 (F R ) (15) Through the spatial attention module, the feature F Event and F RGB Assign weights to feature F Event and F RGB Perform maximum pooling and average pooling in the channel dimension. The specific formula is: Among them, F maxE and F avgE Yes F Event Maximum pooling and average pooling results, F maxR and F avgR Yes F RGB Max pooling and average pooling results; The results of average pooling and maximum pooling are fused and the spatial attention weights are generated through 1×1 convolution: S E =σ(Conv 1*1 (concat(F maxE ,F avgE )) (20) S R =σ(Conv 1*1 (concat(F maxR ,F avgR )) (21) Where σ is the activation function; Apply spatial attention weights to features F Event and F RGB , and get the enhanced feature F EventSA and F RGBSA : F EventSA =S E ⊙F Event (22) F RGBSA =S E ⊙F RGB (23) Through the channel attention module, the feature F EventSA and F RGBSA Assign weights to each channel of feature F EventSA and F RGBSA Perform global average pooling to obtain a global description of each channel: Among them, F Eavg and F Ravg It is the description vector in the channel dimension; The result of global average pooling is passed through a fully connected layer to generate channel attention weights: W E =σ(FC2(ReLU(FC1(F Eavg ))) (26) W R =σ(FC2(ReLU(FC1(F Ravg ))) (27) Among them, FC1 and FC2 are fully connected layers, σ and ReLU are activation functions; Apply channel attention weights to feature F EventSA and F RGBSA , get the weighted feature F AEvent and F ARGB : F CAEvent =W E ⊙F EventSA (28) F CARGB =W R ⊙F RGBSA (29) The weighted features F are fused through the Concat operation AEvent and F ARGB : F=Concat(F CAEvent ,F CARGB ) (30) Among them, F is the fused feature, which contains the fused and enhanced insulator feature information; The fused feature F is used as the initial feature f of the event in the subsequent stage E , F R As the initial RGB feature f in the subsequent stage R , iterate formula (7)-(13) three times, and the feature F obtained after three iterations is the final fusion feature F final .

6. The insulator defect detection method based on multi-modal feature fusion according to claim 5 is characterized in that: In step S4, enhanced event and RGB features are obtained, including: The feature attention module includes a compression layer and an excitation layer; The compression layer is used to combine the final fusion features F final Perform global average pooling to obtain the global compressed feature value of the current feature: Among them, z is the compressed feature vector with a dimension of 1×D; The excitation layer uses a two-layer fully connected bottleneck structure to obtain the weight of each channel, and then uses the weighted features as the input of the next layer; Among them, the first fully connected layer compresses the channel to reduce the amount of calculation. The calculation formula is: z1=σ(W1z2+b1) (32) The second fully connected layer restores the number of channels, and the calculation formula is: s=σ(W2z2+b2) (33) Among them, W is the weight matrix, and W1 dimension is The dimension of W2 is r is the compression ratio; b is the bias vector, and the dimension of b1 is b2 has a dimension of 1×D; σ is the activation function, and the resulting weight vector s has a dimension of 1×D; The fusion feature F final Weighted with the weight vector s to obtain the weighted feature F weighted , which are the enhanced events and RGB features: F weighted (i)=F final (i)⊙s (34) where ⊙ represents element-wise multiplication.

7. The insulator defect detection method based on multi-modal feature fusion according to claim 6 is characterized in that: Step S5 specifically includes: Insulator defect detection network based on multimodal feature fusion for enhanced event and RGB features F weighted Perform insulator classification and bounding box regression; The insulator defect detection network with multimodal feature fusion includes a classification branch and a bounding box regression branch. The classification branch is used to predict the category, and the bounding box regression branch is used to predict the bounding box position. The training process generates accuracy, recall, and mAP parameters for comprehensive evaluation of the model’s performance; During training, a comprehensive loss function is used to optimize the model. The comprehensive loss function includes bounding box regression loss, confidence loss, and classification loss. When the comprehensive loss function in the training process tends to be smooth, the training of the insulator defect detection network with multimodal feature fusion is completed.

8. The insulator defect detection method based on multi-modal feature fusion according to claim 7 is characterized in that: Step S6 specifically includes: Any two corresponding RGB images and event frames that have not participated in the training are input into the trained multimodal feature fusion insulator defect detection network. The multimodal feature fusion insulator defect detection network extracts, fuses and enhances the input data. Based on the enhanced fusion features, the multimodal feature fusion insulator defect detection network performs the final insulator classification and bounding box regression to generate an insulator recognition result map with marked defects.

Images