Generator rotor surface defect detection method and system based on cross-modal fusion

Abstract

The invention relates to a generator rotor surface defect detection method and system based on cross-modal fusion. The method comprises the following steps: acquiring an infrared image and a visible light image of the outer surface of a generator rotor magnetic pole; and respectively inputting the infrared image and the visible light image into a preset double-flow target detection model, and outputting a defect detection result of the outer surface of the generator rotor magnetic pole. The double-flow target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network comprises an infrared branch, a visible light branch and an image fusion module. According to the invention, two parallel branches of the backbone network of the double-flow target detection model are used to carry out feature extraction on the infrared image and the visible light image, and cross-modal feature fusion is carried out on the features of two modals through the image fusion module based on cross attention, so that mutual reference and supplement of information between different modals are realized. The fused image features maintain the advantages of respective modalities and improve the overall defect detection effect at the same time.

Description

Technical Field

[0001] The present invention relates to the field of target calibration technology, and in particular to a generator rotor surface defect detection method and system based on cross-modal fusion. Background Art

[0002] In the field of machine vision, defect detection methods are mainly divided into detection methods based on traditional image processing and detection methods based on deep learning: ① Traditional image processing methods, such as edge detection, select common operators such as Robert and Soble to obtain derivative values, and use derivatives to characterize image changes. The area with the most significant changes is the edge of the target. Traditional visual methods require manual design and manual feature extraction, and need to be tuned for different detection tasks and scenarios. The detection effect depends largely on the results of the tuning. ② Methods based on deep learning, which have developed rapidly and are currently the mainstream in the application of defect detection tasks in the field of machine vision. Deep learning models can extract features from images through networks such as CNN, extract spatial location information at the shallow level of the network, and extract high-level semantic information at the deep level. At the same time, it can provide an "end-to-end" network, which means that the entire image processing process from input to output is completed within a unified framework without the need for manual intervention or feature engineering in the middle. The development of technologies such as cloud computing and GPU acceleration also provides hardware support for this method.

[0003] The influence of illumination on target detection is significant. It directly affects the quality of the image and the effect of feature extraction. Good illumination conditions are the prerequisite for the normal functioning of defect detection methods. Changes in illumination intensity will lead to uneven contrast in the image, affecting the separation of the target and the background. Bright areas may be overexposed and dark areas may be underexposed, making it difficult to extract target features.

[0004] Outdoor space is usually large, which can facilitate the operation, transportation and disassembly of large equipment. Therefore, the maintenance of large generator sets is generally carried out outdoors. However, outdoor maintenance is easily affected by the weather. The lighting conditions of rainy days, foggy days, sunny days, cloudy days and other weather conditions are inconsistent. The change of lighting directly affects the quality of the image and the effect of feature extraction. Therefore, it is difficult for the existing target detection method to complete the defect detection task on the surface of the generator rotor under dynamic lighting changes. This may cause the maintenance task to be unable to be completed on schedule, resulting in economic losses and safety hazards to the power supply. Summary of the invention

[0005] In view of this, it is necessary to provide a generator rotor surface defect detection method based on cross-modal fusion to solve the above-mentioned defects of the prior art.

[0006] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a generator rotor surface defect detection method based on cross-modal fusion, comprising:

[0007] Acquire infrared images and visible light images of the outer surface of the generator rotor magnetic poles;

[0008] The infrared image and the visible light image are respectively input into the preset dual-stream target detection model, and the defect detection results of the outer surface of the generator rotor magnetic pole are output;

[0009] Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0010] Preferably, the step of inputting the infrared image and the visible light image into a preset dual-stream target detection model respectively and outputting the defect detection result of the outer surface of the generator rotor magnetic pole comprises:

[0011] The infrared image and the visible light image are input into two parallel branches of the backbone network respectively, and the infrared branch and the visible light branch are used to extract features of the infrared image and the RGB image respectively;

[0012] The image fusion module of the backbone network introduces a cross-attention mechanism to fuse the features of the infrared image and visible light image across modalities.

[0013] PaFPN neck performs feature fusion of different scales on the fused features to obtain feature maps of different scales;

[0014] The feature maps of different scales are input into the corresponding detection head, and the defect detection results of the outer surface of the generator rotor pole are output.

[0015] Preferably, the infrared branch of the backbone network includes a Conv+C2f combination module and a SPPF module arranged at the end of the infrared branch;

[0016] The Conv+C2f combined module is used to extract features from infrared images;

[0017] The SPPF module set at the end of the infrared branch is used to perform multi-scale pooling on infrared image features;

[0018] Preferably, the Conv+C2f combined module includes a Conv module and a C2f module;

[0019] The Conv module includes three parts: Conv2d, BatchNorm2d and SiLU. Conv2d is used to perform convolution operations on images to extract image features. BatchNorm2d is used to normalize the features after convolution. SiLU is used as a nonlinear activation function to increase the nonlinear expression ability of the network.

[0020] The C2f module is used to segment the image features output by the Conv module in the channel dimension. A part of the features is directly transmitted to the end of the C2f module, and the other part of the features is processed through a series of Block modules. Rich gradient information is obtained through the C2f module.

[0021] Preferably, the visible light branch of the backbone network includes a Conv+C2f-SA combination module and another SPPF module arranged at the end of the visible light branch;

[0022] The Conv+C2f-SA combination module is used to extract features from RGB images; the Conv+C2f-SA combination module includes a Conv module and a C2f-SA module, and the C2f-SA module introduces the Transformer self-attention mechanism based on the C2f module;

[0023] The SPPF module set at the end of the visible light branch is used to perform multi-scale pooling of visible light image features.

[0024] Preferably, the extracting features of the infrared image and the RGB image respectively through the infrared branch and the visible light branch comprises:

[0025] The infrared branch uses the Conv+C2f combination module to extract features from infrared images. The extracted infrared image features are subjected to multi-scale pooling by the SPPF module and then sent to the image fusion module.

[0026] The visible light branch uses the Conv+C2f-SA combination module to extract features from RGB images. The extracted visible light image features are multi-scale pooled by the SPPF module and then sent to the image fusion module Fusion.

[0027] Preferably, the image fusion module Fusion of the backbone network introduces a cross-attention mechanism to perform cross-modal feature fusion on the features of the two modalities of infrared image and visible light image, including:

[0028] The image fusion module Fusion of the backbone network receives the infrared image features and visible light image features output by the infrared branch and the visible light branch;

[0029] Use 1×1 convolution to perform feature mapping on the input infrared image features and visible light image features respectively to generate the infrared modality query Q IR , key K IR Sum value V IR , and the query Q in the visible light modality RGB , key K RGB Sum value V RGB ;

[0030] Perform a cross-attention operation on the feature map to obtain the attention weights between the infrared modality features and the visible light modality features;

[0031] According to the attention weight between the infrared modality features and the visible light modality features, the image features of the infrared modality and the image features of the visible light modality are weightedly calculated respectively, including the output of the two modality features;

[0032] The features of the two modalities obtained after the cross-attention operation are concat- ed to fuse the information of the two modalities.

[0033] In a second aspect, an embodiment of the present invention provides a generator rotor surface defect detection system based on cross-modal fusion, comprising:

[0034] An image acquisition module, used to acquire infrared images and visible light images of the outer surface of the generator rotor magnetic pole;

[0035] A defect detection module is used to input the infrared image and the visible light image into a preset dual-stream target detection model, and output the defect detection result of the outer surface of the generator rotor magnetic pole;

[0036] Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0037] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0038] The memory is used to store programs;

[0039] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the generator rotor surface defect detection method based on cross-modal fusion as described in the embodiment of the first aspect of the present invention.

[0040] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the generator rotor surface defect detection method based on cross-modal fusion as described in the embodiment of the first aspect of the present invention.

[0041] The generator rotor surface defect detection method and system based on cross-modal fusion provided by the present invention respectively extracts features from RGB images and infrared images through two parallel branches of the dual-stream target detection model backbone network, and cross-modal feature fusion is performed on the features of the two modalities of infrared image and visible light image through the image fusion module based on cross-attention, so as to realize mutual reference and supplement of information between different modalities, where infrared image is responsible for providing contour information and RGB image is responsible for supplementing details and texture information; through this complementary mechanism, the fused image features improve the overall defect detection effect while retaining the advantages of each modality. The present invention improves the accuracy and robustness of defect detection, and can be applied in a variety of complex scenarios, such as at night and in severe weather environments, to provide more reliable defect detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a generator rotor surface defect detection method based on cross-modal fusion provided by the present invention;

[0043] Figure 2 A structural diagram of a dual-stream target detection model provided by the present invention;

[0044] Figure 3 A structural diagram of the Conv+C2f-SA combination module provided by the present invention;

[0045] Figure 4 This is a structural diagram of the image fusion module Fusion provided by the present invention;

[0046] Figure 5 A schematic diagram of the fusion of infrared modal features and visible light modal features provided by the present invention;

[0047] Figure 6 A structural block diagram of a generator rotor surface defect detection system based on cross-modal fusion provided by the present invention;

[0048] Figure 7 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] The maintenance of large generator sets is generally carried out outdoors. However, outdoor maintenance is easily affected by the weather. The lighting conditions of rainy days, foggy days, sunny days, cloudy days, etc. are inconsistent. The change of lighting directly affects the quality of the image and the effect of feature extraction. Therefore, it is difficult for the existing target detection method to complete the defect detection task on the surface of the generator rotor under dynamic lighting changes. This may cause the maintenance task to be unable to be completed on schedule, resulting in economic losses and safety hazards to the power supply.

[0052] In view of this, the present invention provides a generator rotor surface defect detection method based on cross-modal fusion, which extracts features from RGB images and infrared images respectively through two parallel branches of the dual-stream target detection model backbone network, and fuses the features of the two modalities of infrared image and visible light image through a cross-attention-based image fusion module, so as to realize mutual reference and supplement of information between different modalities, where infrared images are responsible for providing contour information and RGB images are responsible for supplementing details and texture information; through this complementary mechanism, the fused image features retain the advantages of each modality while improving the overall defect detection effect. The following will be explained and introduced through multiple embodiments.

[0053] Figure 1 The flow chart of the generator rotor surface defect detection method based on cross-modal fusion provided by the present invention. Figure 1 As shown, the generator rotor surface defect detection method based on cross-modal fusion includes the following steps S1-S2, wherein:

[0054] Step S1, acquiring an infrared image and a visible light image of the outer surface of the generator rotor magnetic pole.

[0055] In this embodiment, an infrared camera and a visible light camera are used to collect images on the outer surface of the generator rotor magnetic pole, respectively, to obtain an infrared image and a visible light image. Visible light can be obtained by mixing three colors of RGB in different ratios, so in this article, the RGB image is also referred to as a visible light image.

[0056] Step S2, inputting the infrared image and the visible light image into a preset dual-stream target detection model respectively, and outputting the defect detection result of the outer surface of the generator rotor magnetic pole.

[0057] Figure 2The dual-stream target detection model structure diagram provided by the present invention is shown in FIG. Figure 2 The dual-stream target detection model is composed of a backbone network Backbone, a PaFPN neck (Neck) and a detection head (Detection Head); the backbone network includes an infrared branch, a visible light branch and an image fusion module Fusion. The PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0058] In a preferred embodiment of the present invention, in step S2, the infrared image and the visible light image are respectively input into a preset dual-stream target detection model, and the defect detection result of the outer surface of the generator rotor magnetic pole is output, which may specifically include the following steps S21 to S24:

[0059] In step S21, the infrared image and the visible light image are respectively input into two parallel branches of the backbone network, and features of the infrared image and the RGB image are respectively extracted through the infrared branch and the visible light branch.

[0060] Reference Figure 2 The backbone network Backbone is a two-stream network that can process visible light and infrared images at the same time, and is also responsible for fusing the features of the two modalities of infrared images and visible light images. The backbone network Backbone consists of an infrared branch, a visible light branch, and an image fusion module Fusion. Considering the differences in the features of infrared and visible light images, the two branches are different in structure. Infrared images lack details and texture information, but the local contours are clear. Therefore, the infrared branch is based on CNN and uses a combination of Conv+C2f modules to extract features from infrared images, focusing mainly on the local features of the image. RGB images have rich color and detail information, but CNN has limited ability to extract global features. Therefore, the original C2f module was improved to C2f-SA, in which the Transformer's self-attention mechanism was incorporated. Such improvements enhance the global modeling capabilities of the C2f module, which is conducive to capturing long-distance dependencies.

[0061] In this embodiment, combined with Figure 2 The structures of the infrared branch and the visible light branch in the backbone network are explained.

[0062] The infrared branch of the backbone network includes a Conv+C2f combination module and an SPPF module arranged at the end of the infrared branch. The Conv+C2f combination module is used to extract features from infrared images; the SPPF module arranged at the end of the infrared branch is used to perform multi-scale pooling on infrared image features.

[0063] Among them, the Conv+C2f combination module includes the Conv module and the C2f module. Figure 3This is a structural diagram of the Conv+C2f-SA combination module provided by the present invention. The C2f-SA module introduces the Transformer self-attention mechanism (Self-Attention) based on the C2f module. Figure 3 The Conv module includes three parts: Conv2d, BatchNorm2d and SiLU; Conv2d is used to perform convolution operations on the image and extract image features; BatchNorm2d is used to normalize the features after convolution; SiLU is used as a nonlinear activation function to increase the nonlinear expression ability of the network.

[0064] The C2f module is used to split the image features output by the Conv module in the channel dimension. Some features skip the Block module and directly perform the Concat operation at the end, while the other features pass through the following Block module normally and go through the main branch. At the same time, each time the input passes through a Block, one more branch is derived. In this way, more gradient flow branches are parallelized to form different gradient combinations and obtain richer gradient information.

[0065] Reference Figure 2 ,The visible light branch of the backbone network includes a Conv+C2f-SA combination module and another SPPF module arranged at the end of the visible light branch;

[0066] The Conv+C2f-SA combination module is used to extract features from RGB images; the SPPF module set at the end of the visible light branch is used to perform multi-scale pooling on visible light image features.

[0067] Reference Figure 3 The Conv+C2f-SA combined module includes a Conv module and a C2f-SA module. The C2f-SA module introduces the Transformer's self-attention mechanism (Self-Attention) based on the C2f module.

[0068] The steps of the Conv+C2f-SA combined module to extract visible light image features based on the self-attention mechanism include:

[0069] ① First, the Conv module inputs the RGB image feature F RGB ∈R C×H×W Processing is performed by three 1×1 convolutions to obtain three new feature maps Q(Query), K(Key), and V(Value), where Q∈R c×H×W , K∈R c×H×W , V∈R C ×H×W .

[0070] Q=Conv q 1×1(F RGB )

[0071] K=Conv k 1×1(F RGB )

[0072] V=Conv v 1×1(F RGB )

[0073] ② Use the Flatten module to flatten Q, K, and V for subsequent matrix multiplication. At this time, Q∈R c ×(HW) , K∈R c×(HW) , V∈R C×(HW) , where c is the number of channels, H and W are the height and width of the feature map.

[0074] ③Transpose Q, and the dimension becomes Q T ∈R (HW)×c Then Q T Perform matrix multiplication with K to calculate the similarity between them and obtain an attention weight matrix, Attention Matrix∈R (HW)×(HW) The attention matrix is ​​then normalized using softmax.

[0075] ④ Perform matrix multiplication on the attention matrix and V to obtain the weighted feature V weighted , at this time V weighted ∈R C ×(HW) , and finally restore the dimension to R by reshape C×H×W Finally, the visible light image feature F is obtained after the self-attention mechanism is processed. RGB-SA Among them, F RGB-SA ∈R C×H×W .

[0076] The Conv+C2f-SA combined module can effectively utilize the self-attention mechanism of Transformer to enhance the global modeling ability of CNN, thereby better capturing the global features and long-distance dependencies in visible light images.

[0077] In step S21, the infrared branch uses the Conv+C2f combination module to extract features from the infrared image, and the extracted infrared image features are sent to the image fusion module after multi-scale pooling through the SPPF module. The visible light branch uses the Conv+C2f-SA combination module to extract features from the RGB image, and the extracted visible light image features are sent to the image fusion module after multi-scale pooling through the SPPF module.

[0078] In step S22, the image fusion module Fusion of the backbone network introduces a cross-attention mechanism to perform cross-modal feature fusion on the features of the two modalities of infrared image and visible light image.

[0079] It is understandable that infrared images can clearly show the outline and shape of objects, and can better separate foreground objects even in complex backgrounds. However, the color tone is single, and the texture and detail information are relatively scarce. Visible light images contain rich color information and can provide the color, texture and detail characteristics of objects. It can be seen that infrared images and visible light images are complementary in many aspects. By combining the information of these two modalities, the target detection effect can be significantly improved. Therefore, this patent designs an image fusion module Fusion based on the cross-attention mechanism. Figure 4 This is a structural diagram of the image fusion module Fusion provided by the present invention.

[0080] In step S22, first, the image fusion module Fusion of the backbone network receives the infrared image features F output by the infrared branch and the visible light branch. IR and visible light image feature F RGB ;

[0081] Then, a 1×1 convolution is used to perform feature mapping on the input infrared image features and visible light image features respectively to generate the infrared modality query Q IR , key K IR Sum value V IR , and the query Q in the visible light modality RGB , key K RGB Sum value V RGB ;

[0082] Next, a cross-attention operation is performed on the feature map to obtain the attention weights between the infrared modality features and the visible light modality features. The cross-attention operation is specifically formulated as follows:

[0083]

[0084] Among them, F' IR represents the infrared image features after the cross attention operation, F' RGB represents the visible light image features after the cross-attention operation. × represents matrix multiplication, and σ represents the softmax function. As can be seen from the above formula, during the cross-attention operation, the query Q of the infrared modality is IR With the key K that can be light mode RGB Perform matrix multiplication and the query Q of the optical modality can be RGB Then the key K of the infrared mode IRPerform matrix multiplication. This operation is to measure the degree of match between the two modalities. By calculating the attention weight between the features of one modality and the features of another modality, the correlation between the infrared image and the visible light image is captured, and the weighted fusion of features and effective interaction between modalities are achieved.

[0085] Furthermore, according to the attention weight between the infrared modality features and the visible light modality features, weighted calculations are performed on the image features of the infrared modality and the image features of the visible light modality, respectively, including the output of the two modality features. Figure 5 Schematic diagram of the fusion of infrared modal features and visible light modal features provided by the present invention. Figure 5 In the figure, RGB Modality refers to RGB modality, i.e. visible light modality. Infrared Modality refers to infrared modality. Interraction Modality refers to interactive modality. Figure 5 As shown in the figure, the relationship between the RGB modal features (upper left) and the infrared modal features (lower right) is learned by cross attention to obtain the corresponding attention weight matrix, and the image features are weighted to obtain the output containing the features of the two modalities (lower left, upper right)

[0086] Finally, the features of the two modes obtained after the cross attention operation are concatenated to fuse the information of the two modes and obtain the output F of the image fusion module Fusion. fusion .

[0087] F fusion =Conv1×1(Concat(F' RGB ,F' IR ))

[0088] This embodiment uses a cross-attention-based image fusion module Fusion to perform cross-modal feature fusion on the features of the two modalities, infrared image and visible light image, to achieve mutual reference and complementation of information between different modalities. The infrared image is responsible for providing contour information, and the RGB image is responsible for supplementing details and texture information. Through this complementary mechanism, the fused image features retain the advantages of each modality while improving the overall defect detection effect.

[0089] Step S23, the PaFPN neck performs feature fusion of different scales on the fused features to obtain feature maps of different scales.

[0090] Specifically, the PaFPN neck composed of FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) can fuse the shallow spatial information and deep semantic information of the image, improving the model's object detection ability in complex scenes.

[0091] Step S24, inputting the feature images of different scales into the corresponding detection head, and outputting the defect detection result of the outer surface of the generator rotor pole.

[0092] Specifically, feature maps of different scales are fed into corresponding detection heads. Each detection head is responsible for generating anchor boxes (i.e., preset candidate detection boxes) on feature maps of a specific scale, and adjusting these anchor boxes to better match defects in the image. Multi-scale detection heads can process defects of different sizes at the same time, improving the comprehensiveness and accuracy of detection.

[0093] Figure 6 The structural block diagram of the generator rotor surface defect detection system based on cross-modal fusion provided by the present invention is shown in FIG. Figure 6 , the generator rotor surface defect detection system based on cross-modal fusion includes:

[0094] An image acquisition module 601 is used to acquire an infrared image and a visible light image of the outer surface of the generator rotor magnetic pole;

[0095] The defect detection module 602 is used to input the infrared image and the visible light image into a preset dual-stream target detection model, and output the defect detection result of the outer surface of the generator rotor magnetic pole;

[0096] Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0097] The generator rotor surface defect detection system based on cross-modal fusion provided by the present invention executes the generator rotor surface defect detection method based on cross-modal fusion provided by the above-mentioned embodiments through the above-mentioned modules. The generator rotor surface defect detection method based on cross-modal fusion has been described in detail in the above-mentioned embodiments, and this embodiment will not be repeated here.

[0098] Figure 7 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 7As shown, the present invention also provides an electronic device, and the electronic device 700 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device 700 includes a processor 701 and a memory 702, wherein the memory 702 stores a generator rotor surface defect detection program 703 based on cross-modal fusion.

[0099] In some embodiments, the memory 702 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 702 may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Furthermore, the memory 702 may also include both an internal storage unit of the computer device and an external storage device. The memory 702 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 702 can also be used to temporarily store data that has been output or is to be output. In one embodiment, when the generator rotor surface defect detection program 703 based on cross-modal fusion is executed by the processor 701, the following steps are implemented:

[0100] Acquire infrared images and visible light images of the outer surface of the generator rotor magnetic poles;

[0101] The infrared image and the visible light image are respectively input into the preset dual-stream target detection model, and the defect detection results of the outer surface of the generator rotor magnetic pole are output;

[0102] Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0103] In some embodiments, the processor 701 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 702, such as executing a generator rotor surface defect detection program based on cross-modal fusion.

[0104] This embodiment further provides a computer-readable storage medium, on which a generator rotor surface defect detection program based on cross-modal fusion is stored. When the generator rotor surface defect detection program based on cross-modal fusion is executed by a processor, the following steps are implemented:

[0105] Acquire infrared images and visible light images of the outer surface of the generator rotor magnetic poles;

[0106] The infrared image and the visible light image are respectively input into the preset dual-stream target detection model, and the defect detection results of the outer surface of the generator rotor magnetic pole are output;

[0107] Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

[0108] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generator rotor surface defect detection method based on cross-modal fusion, characterized in that: include: Acquire infrared images and visible light images of the outer surface of the generator rotor magnetic poles; The infrared image and the visible light image are respectively input into the preset dual-stream target detection model, and the defect detection results of the outer surface of the generator rotor magnetic pole are output; Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

2. The generator rotor surface defect detection method based on cross-modal fusion according to claim 1 is characterized in that: The infrared image and the visible light image are respectively input into a preset dual-stream target detection model, and the defect detection result of the outer surface of the generator rotor magnetic pole is output, including: The infrared image and the visible light image are input into two parallel branches of the backbone network respectively, and the infrared branch and the visible light branch are used to extract features of the infrared image and the RGB image respectively; The image fusion module of the backbone network introduces a cross-attention mechanism to fuse the features of the infrared image and visible light image across modalities. PaFPN neck performs feature fusion of different scales on the fused features to obtain feature maps of different scales; The feature maps of different scales are input into the corresponding detection head, and the defect detection results of the outer surface of the generator rotor pole are output.

3. The generator rotor surface defect detection method based on cross-modal fusion according to claim 2 is characterized in that: The infrared branch of the backbone network includes a Conv+C2f combination module and a SPPF module arranged at the end of the infrared branch; The Conv+C2f combined module is used to extract features from infrared images; The SPPF module set at the end of the infrared branch is used to perform multi-scale pooling on infrared image features.

4. The generator rotor surface defect detection method based on cross-modal fusion according to claim 3 is characterized in that: The Conv+C2f combined module includes a Conv module and a C2f module; The Conv module includes three parts: Conv2d, BatchNorm2d and SiLU; Conv2d is used to perform convolution operation on the image and extract image features; BatchNorm2d is used to normalize the features after convolution; SiLU is used as a nonlinear activation function to increase the nonlinear expression ability of the network; The C2f module is used to segment the image features output by the Conv module in the channel dimension. A part of the features is directly transmitted to the end of the C2f module, and the other part of the features is processed through a series of Block modules. Rich gradient information is obtained through the C2f module.

5. The generator rotor surface defect detection method based on cross-modal fusion according to claim 4 is characterized in that: The visible light branch of the backbone network includes a Conv+C2f-SA combination module and another SPPF module arranged at the end of the visible light branch; The Conv+C2f-SA combination module is used to extract features from RGB images; the Conv+C2f-SA combination module includes a Conv module and a C2f-SA module, and the C2f-SA module introduces the Transformer self-attention mechanism based on the C2f module; The SPPF module set at the end of the visible light branch is used to perform multi-scale pooling of visible light image features.

6. The generator rotor surface defect detection method based on cross-modal fusion according to claim 5 is characterized in that: The feature extraction of the infrared image and the RGB image respectively by using the infrared branch and the visible light branch includes: The infrared branch uses the Conv+C2f combination module to extract features from infrared images. The extracted infrared image features are subjected to multi-scale pooling by the SPPF module and then sent to the image fusion module. The visible light branch uses the Conv+C2f-SA combination module to extract features from RGB images. The extracted visible light image features are multi-scale pooled by the SPPF module and then sent to the image fusion module Fusion.

7. The generator rotor surface defect detection method based on cross-modal fusion according to claim 6 is characterized in that: The image fusion module Fusion of the backbone network introduces a cross-attention mechanism to fuse the features of the infrared image and the visible light image across modalities, including: The image fusion module Fusion of the backbone network receives the infrared image features and visible light image features output by the infrared branch and the visible light branch; Use 1×1 convolution to perform feature mapping on the input infrared image features and visible light image features respectively to generate the infrared modality query Q IR , key K IR Sum value V IR , and the query Q in the visible light modality RGB , key K RGB Sum value V RGB ; Perform a cross-attention operation on the feature map to obtain the attention weights between the infrared modality features and the visible light modality features; According to the attention weight between the infrared modality features and the visible light modality features, the image features of the infrared modality and the image features of the visible light modality are weightedly calculated respectively, including the output of the two modality features; The features of the two modalities obtained after the cross-attention operation are concat- ed to fuse the information of the two modalities.

8. A generator rotor surface defect detection system based on cross-modal fusion applied to the generator rotor surface defect detection method based on cross-modal fusion as described in any one of claims 1 to 7, characterized in that: include: An image acquisition module, used to acquire infrared images and visible light images of the outer surface of the generator rotor magnetic pole; A defect detection module is used to input the infrared image and the visible light image into a preset dual-stream target detection model, and output the defect detection result of the outer surface of the generator rotor magnetic pole; Among them, the dual-stream target detection model is composed of a backbone network, a PaFPN neck and a detection head; the backbone network includes an infrared branch, a visible light branch and an image fusion module; the PaFPN neck is composed of a feature pyramid FPN and a path aggregation network PAN.

9. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the generator rotor surface defect detection method based on cross-modal fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the generator rotor surface defect detection method based on cross-modal fusion as described in any one of claims 1 to 7.

Images