Wind turbine blade icing monitoring method and device based on dual-modal image fusion

Abstract

The present invention discloses a method and device for monitoring wind blade icing based on dual-modal image fusion. The method comprises: acquiring an image of an iced wind blade; inputting the image of the iced wind blade into a wind blade icing recognition model to obtain an icing threshold; and performing wind blade icing monitoring based on the icing threshold and the wind blade icing recognition model. The wind blade icing recognition model is obtained by the following steps: acquiring a wind blade icing image dataset, the wind blade icing image dataset comprising visible light images and infrared light images; constructing a dual-modal image fusion model; inputting the wind blade icing image dataset into the dual-modal image fusion model to train the dual-modal image fusion model and obtain the wind blade icing recognition model. The present invention realizes wind blade icing monitoring and improves accuracy. The present invention can be widely applied in the field of image recognition technology.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method and device for monitoring wind turbine blade icing based on dual-modal image fusion. Background Art

[0002] Icing on wind turbine blades can affect the normal operation of wind turbines. Conventional methods for monitoring wind turbine blade icing primarily include visible light monitoring and infrared monitoring. Visible light monitoring captures images with rich texture information, but is susceptible to occlusion or insufficient light. Infrared monitoring can capture thermal radiation from wind turbine blades, but infrared images are single-channel and lack detailed information about the target scene, resulting in low accuracy in ice detection.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of the present invention provide a method and device for monitoring ice coating on wind turbine blades based on dual-modal image fusion, which effectively improves the accuracy of identifying ice coating on wind turbine blades.

[0005] In one aspect, an embodiment of the present invention provides a method for monitoring wind turbine blade icing based on dual-modal image fusion, comprising the following steps:

[0006] Acquire images of ice-covered wind turbine blades;

[0007] Inputting the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold;

[0008] Monitoring wind turbine blade icing according to the icing occurrence threshold and the wind turbine blade icing identification model;

[0009] The wind turbine blade icing identification model is obtained by the following steps:

[0010] Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image;

[0011] Construct a dual-modal image fusion model;

[0012] The wind turbine blade icing image dataset is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

[0013] In some embodiments, the process of constructing the dual-modal image fusion model includes:

[0014] Constructing a visible light input layer and an infrared light input layer respectively;

[0015] After the visible light input layer, a visible light backbone network layer is constructed;

[0016] After the infrared light input layer, an infrared light backbone network layer is constructed;

[0017] After the visible light backbone network layer, a visible light neck network layer is constructed;

[0018] After the infrared light backbone network layer, an infrared light neck network layer is constructed;

[0019] After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed.

[0020] In some embodiments, the process of constructing the visible light backbone network layer includes:

[0021] Constructing a first visible light convolution composite layer, the first visible light convolution composite layer including three standard convolution modules and two C2f modules, the first visible light convolution composite layer being used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature;

[0022] Constructing a second visible light convolution composite layer, the second visible light convolution composite layer including the standard convolution module and the C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature;

[0023] Construct a third visible light convolution composite layer, wherein the third visible light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature.

[0024] In some embodiments, the process of constructing the visible light neck network layer includes:

[0025] Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature;

[0026] Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature;

[0027] Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature;

[0028] Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature;

[0029] Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature;

[0030] Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature;

[0031] Constructing a first standard convolution layer, where the first standard convolution layer is used to perform standard convolution on the second visible light fusion feature to obtain a first standard convolution feature;

[0032] Constructing a third visible light feature connection layer, wherein the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature;

[0033] constructing a third visible light C2f layer, wherein the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature;

[0034] Constructing a second standard convolution layer, where the second standard convolution layer is used to perform standard convolution on the third visible light fusion feature to obtain a second standard convolution feature;

[0035] Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature;

[0036] A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

[0037] In some embodiments, the construction process of the infrared optical backbone network layer includes:

[0038] Constructing a first infrared light convolution composite layer, the first infrared light convolution composite layer including three standard convolution modules and two C2f modules, the first infrared light convolution composite layer being used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature;

[0039] Constructing a second infrared light convolution composite layer, the second infrared light convolution composite layer including the standard convolution module and the C2f module, the second infrared light convolution composite layer being used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature;

[0040] Construct a third infrared light convolution composite layer, wherein the third infrared light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature.

[0041] In some embodiments, the process of constructing the infrared neck network layer includes:

[0042] Constructing a third upsampling layer, wherein the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature;

[0043] Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampling feature and the second infrared light convolution feature to obtain a first infrared light connection feature;

[0044] Constructing a first infrared light C2f layer, where the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature;

[0045] Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature;

[0046] Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampling feature and the first infrared light convolution feature to obtain a second infrared light connection feature;

[0047] constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection feature to obtain a second infrared light fusion feature;

[0048] Constructing a third standard convolution layer, wherein the third standard convolution layer is used to perform standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature;

[0049] Constructing a third infrared light feature connection layer, wherein the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature;

[0050] constructing a third infrared light C2f layer, wherein the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature;

[0051] Constructing a fourth standard convolution layer, wherein the fourth standard convolution layer is used to perform standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature;

[0052] Constructing a fourth infrared light feature connection layer, wherein the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature;

[0053] A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

[0054] In some embodiments, the process of constructing the header network layer includes:

[0055] Constructing a first fusion module, wherein the first fusion module is configured to perform bimodal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first bimodal fusion feature;

[0056] Constructing a second fusion module, the second fusion module being configured to perform bimodal fusion processing on the third visible light fusion feature and the third infrared light fusion feature to obtain the second bimodal fusion feature;

[0057] Constructing a third fusion module, wherein the third fusion module is used to perform bimodal fusion processing on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third bimodal fusion feature;

[0058] A detection head layer is constructed, where the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, where the prediction result includes a target bounding box, a category, and a confidence level.

[0059] In some embodiments, performing bimodal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first bimodal fusion feature includes:

[0060] splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature;

[0061] performing compression and activation processing on the first splicing feature to obtain a compressed and activated feature;

[0062] According to a preset point convolution frequency, the compression and activation features are point-convolved using a point convolution layer to obtain a global feature;

[0063] Calculating a target weight according to a preset activation function and the global feature, the target weight including a visible light weight and an infrared light weight;

[0064] Multiplying the visible light weight and the second visible light fusion feature to obtain a visible light multiplication feature;

[0065] multiplying the infrared light weight and the second infrared light fusion feature to obtain an infrared light multiplication feature;

[0066] Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first addition feature;

[0067] Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature;

[0068] Splicing the first added feature and the second added feature to obtain a second spliced ​​feature;

[0069] Perform point convolution on the second splicing feature to obtain the first bimodal fusion feature.

[0070] In some embodiments, compressing and activating the first splicing feature to obtain the compressed and activated feature includes:

[0071] Performing feature conversion on the first splicing feature to obtain a conversion feature;

[0072] Performing global average pooling on the converted features to obtain compressed features;

[0073] According to the ReLU activation function, the compressed features are input into the first fully connected layer to obtain the excitation features;

[0074] According to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight;

[0075] The channel weight and the conversion feature are multiplied to obtain the compression and activation feature.

[0076] On the other hand, an embodiment of the present invention provides a wind turbine blade icing monitoring device based on dual-modal image fusion, comprising:

[0077] The first module is used to obtain images of ice-covered wind turbine blades;

[0078] The second module is configured to input the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold;

[0079] A third module is used to monitor wind turbine blade icing based on the icing occurrence threshold and the wind turbine blade icing identification model;

[0080] The wind turbine blade icing identification model is obtained by the following steps:

[0081] Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image;

[0082] Construct a dual-modal image fusion model;

[0083] The wind turbine blade icing image dataset is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

[0084] The beneficial effects of the present invention are as follows:

[0085] This embodiment of the present invention first acquires an image of an iced wind turbine blade, then inputs the image into a wind blade icing recognition model to determine an ice occurrence threshold. Finally, wind blade icing monitoring is performed based on the ice occurrence threshold and the wind blade icing recognition model. This allows wind blade icing monitoring to be performed through model recognition and threshold comparison, thereby improving accuracy. The wind blade icing recognition model is constructed by first acquiring a dataset of iced wind turbine blade images, then constructing a dual-modal image fusion model, and finally inputting the dataset into the dual-modal image fusion model to train the dual-modal image fusion model, thereby improving model accuracy.

[0086] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0088] Figure 1 This is a flow chart of a method for monitoring wind turbine blade icing based on dual-modal image fusion according to an embodiment of the present invention;

[0089] Figure 2 This is a flow chart of an embodiment of the present invention for determining an ice accumulation threshold for a wind turbine blade;

[0090] Figure 3 This is a flow chart of an embodiment of the present invention for obtaining a wind turbine blade icing identification model;

[0091] Figure 4 This is a flowchart of building a dual-modal image fusion model according to an embodiment of the present invention;

[0092] Figure 5 A schematic diagram of a YOLOv8 model structure according to an embodiment of the present invention;

[0093] Figure 6 A schematic diagram of a dual-modal image fusion model structure according to an embodiment of the present invention;

[0094] Figure 7 A schematic diagram of a fusion module structure according to an embodiment of the present invention;

[0095] Figure 8 This is a schematic diagram of a SENet attention module structure according to an embodiment of the present invention;

[0096] Figure 9 A schematic diagram of the overall process of model training and ice cover monitoring according to an embodiment of the present invention;

[0097] Figure 10 The figure is a schematic structural diagram of a wind turbine blade icing monitoring device based on dual-modal image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0099] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0100] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0101] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0102] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0103] C2f module: It is an important component in YOLOv8, mainly used for feature extraction and fusion.

[0104] Standard convolution module: named Conv in the source code, it is composed of a two-dimensional convolution layer (Conv2D), a batch normalization layer (BatchNorm2D), and an activation layer (Activation) assembled in series.

[0105] Concat component: It is an important component in YOLOv8. It is mainly used to connect feature maps from different layers in the channel dimension to generate richer feature maps to improve the accuracy of object detection. In the network, Concat module usually appears in the structure of cross-layer connection.

[0106] In the related art, in current research, the methods for wind turbine blade ice image recognition mainly include infrared monitoring methods and visible light monitoring methods. The images collected by the visible light monitoring method have rich texture information and can enhance the detailed features of the wind turbine image, but they are easily affected by occlusion and insufficient light. The infrared monitoring method can capture the thermal radiation information of the wind turbine blades and has strong anti-interference ability. It can be used for wind turbine blade ice monitoring under complex backgrounds such as weak light, dark light, and blur. However, infrared images are single-channel images and it is difficult to provide detailed information of the target scene. In practical applications, its monitoring performance is often difficult to meet the needs of practical applications, and the accuracy of ice recognition is low. Based on the complementary characteristics of infrared monitoring methods and visible light monitoring methods, the two are fused to obtain a fused image that combines the advantages of visible light and infrared light, thereby enhancing the accuracy of wind turbine blade ice monitoring, which is of great significance for improving the all-day monitoring capability of wind turbine blade ice.

[0107] In view of this, the embodiment of the present invention constructs a YOLOv8-TID wind turbine blade icing recognition model based on a dual-modal image fusion based on the YOLOv8 algorithm network. YOLOv8 is a new SOTA (State Of The Art) model proposed by Ultralytics, the R&D company of YOLOv5, in 2023. It has the advantages of fast speed, high accuracy, and strong robustness. At present, there is a large gap in the research on the use of the YOLOv8 model for wind turbine blade icing monitoring. The embodiment of the present invention combines the advantages of infrared monitoring methods and visible light monitoring methods to improve the accuracy of wind turbine blade icing image recognition.

[0108] The wind turbine blade icing monitoring method based on dual-modal image fusion provided in the embodiment of the present application relates to the field of image recognition technology. The wind turbine blade icing monitoring method based on dual-modal image fusion provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the wind turbine blade icing monitoring method based on dual-modal image fusion, etc., but is not limited to the above forms.

[0109] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0110] The following is a detailed explanation of the embodiments of the present application with reference to the accompanying drawings:

[0111] Figure 1 This is an optional flow chart of a wind turbine blade icing monitoring method based on dual-modal image fusion provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.

[0112] Step S101, acquiring an image of an ice-covered wind turbine blade;

[0113] Step S102: inputting the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold;

[0114] Step S103: monitoring the icing of the wind turbine blades according to the icing occurrence threshold and the wind turbine blade icing identification model.

[0115] In some embodiments, in steps S101 to S103, an image of an iced fan blade may be first obtained as a standard image of ice, and then the image of the iced fan blade may be input into a fan blade ice recognition model to obtain an ice occurrence threshold. Finally, based on the ice occurrence threshold and the fan blade ice recognition model, fan blade ice monitoring may be performed. For example, the process of determining the fan blade ice occurrence threshold is as follows: Figure 2 As shown, this is usually conducted in a constant temperature and humidity laboratory environment, using artificial icing experiments to simulate real-world conditions. In the experiment, the confidence score of the wind turbine blade icing recognition model is used as the monitoring target. When significant ice accumulation is observed on the wind turbine blade surface, the prediction box confidence score output by the YOLOv8 model is recorded and used as the icing threshold for image monitoring. Furthermore, in actual operation, when the confidence score identified by the wind turbine blade icing recognition model exceeds the icing threshold, icing can be diagnosed. This allows wind turbine blade icing monitoring to be achieved through model recognition and threshold comparison, thereby improving accuracy.

[0116] In some embodiments, as Figure 3 As shown, obtaining the wind turbine blade icing identification model may include but is not limited to steps S201 to S203.

[0117] Step S201: Acquire a wind turbine blade ice-covered image dataset, where the wind turbine blade ice-covered image dataset includes visible light images and infrared light images;

[0118] Step S202: constructing a dual-modal image fusion model;

[0119] Step S203: inputting the wind turbine blade icing image dataset into the dual-modal image fusion model so as to train the dual-modal image fusion model and obtain a wind turbine blade icing recognition model.

[0120] In steps S201 to S203 shown in the embodiment of the present application, a wind turbine blade icing identification model is obtained, thereby improving the model accuracy.

[0121] In some embodiments, in steps S201 to S203, a wind blade ice-covered image dataset can be obtained through a wind blade image database. The wind blade ice-covered image dataset can also be obtained through other means, but is not limited thereto. For example, images of wind blade ice-covered images can be obtained from the Internet or images of wind blade ice-covered images taken in a laboratory. The wind blade ice-covered images are labeled using the open source software X-AnyLabeling to obtain a wind blade ice-covered image dataset. The dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1:2. Then, a dual-modal image fusion model is constructed, and finally, the wind blade ice-covered image dataset is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain a wind blade ice-covered recognition model. For example, the dual-modal image fusion model can be trained based on the wind blade ice-covered image dataset, and through training iteration and optimization, it can be enabled to accurately monitor and identify the ice coverage on the wind blades. Furthermore, during the model training process, the prepared wind blade ice-covered image dataset can be used for model training. First, configure the training parameters: 300 epochs, 8 workers, 8 batch sizes, 640 input image dimensions (imgsz), and an initial learning rate (lr0) of 0.01. Then, load the YOLOv8 model and download the pretrained weights for the YOLOv8 model as the model's initial state. During training, the model will learn to identify ice-covered areas on the blades. Monitor the training progress and adjust parameters based on the performance of the validation set to prevent overfitting. Once satisfactory performance is achieved, save the trained YOLOv8 model weights for subsequent experiments.

[0122] In some embodiments, in step S202, Figure 4 As shown, the process of constructing the dual-modal image fusion model may include but is not limited to steps S301 to S306.

[0123] Step S301: constructing a visible light input layer and an infrared light input layer respectively;

[0124] Step S302: After the visible light input layer, construct a visible light backbone network layer;

[0125] Step S303: after the infrared light input layer, construct an infrared light backbone network layer;

[0126] Step S304: After the visible light backbone network layer, a visible light neck network layer is constructed;

[0127] Step S305: construct an infrared light neck network layer after the infrared light backbone network layer;

[0128] Step S306: After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed.

[0129] In some embodiments, a conventional YOLOv8 model such as Figure 5 As shown in the figure, this is a single-stage target detection model based on candidate regions. The algorithm consists of an input terminal (Input), a backbone network (Backbone), a neck network (Neck), and a detection head (Detect). Its main idea is to predict the location and category of the target object through grid division and anchor box. Among them, the Backbone layer mainly uses a multi-layer convolutional neural network to gradually extract the feature information of the image. As the convolution deepens, the values ​​of H and W gradually decrease, from the initial 640×640 to 20×20. The reduced size means that the image feature map contains deeper feature information, which helps to detect and recognize large targets. The Neck layer is mainly responsible for fusing the information passed by the Backbone, so that the generated feature map combines deep semantic information and shallow detailed texture information. The Detect layer mainly predicts the feature map passed by the Neck layer and outputs information such as the detected target bounding box, category, and confidence. The YOLOv8-TID model (i.e., dual-modal image fusion model) proposed in this embodiment is an improvement on the traditional YOLOv8 model. The bimodal image fusion model aims to improve image processing accuracy by fusing features from different layers. It consists of a dual input layer, a dual backbone network layer, a dual neck network layer, and a monitoring layer, with the dual neck network layer performing the fusion operation.

[0130] In some embodiments, in step S301, the dual-modal image fusion model structure is as follows: Figure 6 As shown in the figure, a visible light input layer and an infrared light input layer can be constructed separately so that the features of visible light and infrared light can be extracted separately. Furthermore, in the subsequent feature extraction process, the dual-path backbone network layer (visible light backbone network layer and infrared light backbone network layer) can be used to extract features of visible light and infrared light images respectively, and output them to the dual-path neck network layer (visible light neck network layer and infrared light neck network layer), and enhanced by the upsampling fusion process. Given that the dual-path neck network layer has three output links, the YOLOv8-TID model (dual-modal image fusion model) embeds a fusion module at each output position of the dual-path neck network layer to fuse the enhanced features.

[0131] In some embodiments, in step S302, the process of constructing the visible light backbone network layer includes:

[0132] Constructing a first visible light convolution composite layer, the first visible light convolution composite layer includes three standard convolution modules and two C2f modules, and the first visible light convolution composite layer is used to extract the first visible light feature of the visible light image to obtain the first visible light convolution feature;

[0133] Constructing a second visible light convolution composite layer, the second visible light convolution composite layer including a standard convolution module and a C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature;

[0134] A third visible light convolution composite layer is constructed. The third visible light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. The third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature.

[0135] In some embodiments, a first visible light convolution composite layer can be constructed first, then a second visible light convolution composite layer can be constructed, and finally a third visible light convolution composite layer can be constructed. The first visible light convolution composite layer includes 3 standard convolution modules and 2 C2f modules, the second visible light convolution composite layer includes a standard convolution module and a C2f module, and the third visible light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. In the feature extraction process, the first visible light convolution composite layer is used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature. The second visible light convolution composite layer is used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature. The third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature. It can be understood that the composition structures of different visible light convolution composite layers are different, and therefore different visible light feature extraction processes are also different. Exemplarily, the standard convolution module may include a Conv module, which is composed of a two-dimensional convolution layer (Conv2D), a batch normalization layer (BatchNorm2D), and an activation layer (SiLu) assembled in series. The C2f module consists of two BottleNeck layers, a separation layer (Split), a feature connection layer (Concat), and two standard convolution modules (Conv). The Bottleneck layer is an important component introduced in the deep residual network (ResNet) to reduce the computational complexity of the model and improve the feature extraction capability. The fast spatial pyramid pooling module may include an SPPF (Spatial Pyramid Pooling Fast) module. The SPPF module is an important component in the YOLOv5 model, mainly used for feature extraction and enhancement. SPPF is a fast implementation version of the SPP (Spatial Pyramid Pooling) module. It captures feature information of different scales by using multiple pooling kernels of different sizes, thereby enhancing the generalization ability of the model.

[0136] In some embodiments, in step S303, the process of constructing the infrared optical backbone network layer includes:

[0137] Constructing a first infrared light convolution composite layer, the first infrared light convolution composite layer including three standard convolution modules and two C2f modules, the first infrared light convolution composite layer being used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature;

[0138] Constructing a second infrared light convolution composite layer, the second infrared light convolution composite layer including a standard convolution module and a C2f module, the second infrared light convolution composite layer being used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature;

[0139] A third infrared light convolution composite layer is constructed. The third infrared light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. The third infrared light convolution composite layer is used to extract the third infrared light feature from the second infrared light convolution feature to obtain the third infrared light convolution feature.

[0140] In some embodiments, a first infrared light convolution composite layer can be constructed first, then a second infrared light convolution composite layer can be constructed, and finally a third infrared light convolution composite layer can be constructed. The first infrared light convolution composite layer includes three standard convolution modules and two C2f modules, the second infrared light convolution composite layer includes a standard convolution module and a C2f module, and the third infrared light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. During the feature extraction process, the first infrared light convolution composite layer is used to perform a first infrared light feature extraction on the infrared light image to obtain a first infrared light convolution feature. The second infrared light convolution composite layer is used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature. The third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature. It can be understood that the infrared light backbone network layer and the visible light backbone network layer are similar in structure and feature extraction process.

[0141] In some embodiments, in step S304, the process of constructing the visible light neck network layer includes:

[0142] Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature;

[0143] Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature;

[0144] Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature;

[0145] Constructing a second upsampling layer, the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature;

[0146] Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature;

[0147] Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature;

[0148] Constructing a first standard convolution layer, the first standard convolution layer is used to perform standard convolution on the second visible light fusion feature to obtain a first standard convolution feature;

[0149] Constructing a third visible light feature connection layer, where the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature;

[0150] Constructing a third visible light C2f layer, where the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature;

[0151] Constructing a second standard convolution layer, which is used to perform standard convolution on the third visible light fusion feature to obtain a second standard convolution feature;

[0152] Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature;

[0153] A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

[0154] In some embodiments, a first upsampling layer, a first visible light feature connection layer, a first visible light C2f layer, a second upsampling layer, a second visible light feature connection layer, a second visible light C2f layer, a first standard convolution layer, a third visible light feature connection layer, a third visible light C2f layer, a second standard convolution layer, a fourth visible light feature connection layer, and a fourth visible light C2f layer can be constructed in sequence. During the feature extraction process, the third visible light convolution feature can be upsampled by the first upsampling layer to obtain a first upsampling feature; the first upsampling feature and the second visible light convolution feature can be feature-connected by the first visible light feature connection layer to obtain a first visible light connection feature; the first visible light connection feature can be feature-fused by the first visible light C2f layer to obtain a first visible light fusion feature; the first visible light fusion feature can be upsampled by the second upsampling layer to obtain a second upsampling feature; the second upsampling feature and the first visible light convolution feature can be feature-connected by the second visible light feature connection layer to obtain a second visible light connection feature; the second visible light connection feature can be feature-fused by the second visible light C2f layer to obtain a second visible light fusion feature. ; The second visible light fusion feature can be subjected to standard convolution through the first standard convolution layer to obtain the first standard convolution feature; the first standard convolution feature and the first visible light fusion feature can be subjected to feature connection through the third visible light feature connection layer to obtain the third visible light connection feature; the third visible light connection feature can be subjected to feature fusion through the third visible light C2f layer to obtain the third visible light fusion feature; the third visible light fusion feature can be subjected to standard convolution through the second standard convolution layer to obtain the second standard convolution feature; the second standard convolution feature and the third visible light convolution feature can be subjected to feature connection through the fourth visible light feature connection layer to obtain the fourth visible light connection feature; the fourth visible light connection feature can be subjected to feature fusion through the fourth visible light C2f layer to obtain the fourth visible light fusion feature. It can be understood that, Figure 6 The Concat in stands for the feature connection layer.

[0155] In some embodiments, in step S305, the process of constructing the infrared neck network layer includes:

[0156] Constructing a third upsampling layer, the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature;

[0157] Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampled feature and the second infrared light convolution feature to obtain a first infrared light connection feature;

[0158] Constructing a first infrared light C2f layer, where the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature;

[0159] Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature;

[0160] Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampled feature and the first infrared light convolution feature to obtain a second infrared light connection feature;

[0161] Constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection features to obtain a second infrared light fusion feature;

[0162] Constructing a third standard convolution layer, which is used to perform standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature;

[0163] Constructing a third infrared light feature connection layer, where the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature;

[0164] Constructing a third infrared light C2f layer, where the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature;

[0165] Constructing a fourth standard convolution layer, which is used to perform standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature;

[0166] Constructing a fourth infrared light feature connection layer, where the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature;

[0167] A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

[0168] In some embodiments, the third upsampling layer, the first infrared light feature connection layer, the first infrared light C2f layer, the fourth upsampling layer, the second infrared light feature connection layer, the second infrared light C2f layer, the third standard convolution layer, the third infrared light feature connection layer, the third infrared light C2f layer, the fourth standard convolution layer, the fourth infrared light feature connection layer, and the fourth infrared light C2f layer can be constructed in sequence. During the feature extraction process, the third infrared light convolution feature can be upsampled by the third upsampling layer to obtain the third upsampling feature; the third upsampling feature and the second infrared light convolution feature can be feature-connected by the first infrared light feature connection layer to obtain the first infrared light connection feature; the first infrared light connection feature can be feature-fused by the first infrared light C2f layer to obtain the first infrared light fusion feature; the first infrared light fusion feature can be upsampled by the fourth upsampling layer to obtain the fourth upsampling feature; the fourth upsampling feature and the first infrared light convolution feature can be feature-connected by the second infrared light feature connection layer to obtain the second infrared light connection feature; the second infrared light connection feature can be feature-fused by the second infrared light C2f layer to obtain the second infrared light fusion feature. ; The first bimodal fusion feature can be subjected to standard convolution through the third standard convolution layer to obtain a third standard convolution feature; the third standard convolution feature and the first infrared fusion feature can be feature-connected through the third infrared light feature connection layer to obtain a third infrared light connection feature; the third infrared light connection feature can be feature-fused through the third infrared light C2f layer to obtain a third infrared light fusion feature; the second bimodal fusion feature can be subjected to standard convolution through the fourth standard convolution layer to obtain a fourth standard convolution feature; the fourth standard convolution feature and the third infrared light convolution feature can be feature-connected through the fourth infrared light feature connection layer to obtain a fourth infrared light connection feature; the fourth infrared light connection feature can be feature-fused through the fourth infrared light C2f layer to obtain a fourth infrared light fusion feature. It can be understood that, unlike the visible light neck network layer, the infrared light neck network layer uses the output of the fusion module as the input of the third standard convolution layer and the fourth standard convolution layer, so that the features of visible light and infrared light are further fused.

[0169] In some embodiments, in step S306, the process of constructing the header network layer includes:

[0170] Constructing a first fusion module, the first fusion module is used to perform a bimodal fusion process on the second visible light fusion feature and the second infrared light fusion feature to obtain a first bimodal fusion feature;

[0171] Constructing a second fusion module, the second fusion module is used to perform bimodal fusion processing on the third visible light fusion feature and the third infrared light fusion feature to obtain a second bimodal fusion feature;

[0172] Constructing a third fusion module, the third fusion module is used to perform bimodal fusion processing on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third bimodal fusion feature;

[0173] Construct a detection head layer, which is used to predict the third bimodal fusion feature and obtain the prediction result, which includes the target bounding box, category and confidence.

[0174] In some embodiments, during the fusion phase, YOLOv8-TID incorporates shallow infrared feature information, and the fusion process is a progressive, integrated process. Specifically, YOLOv8-TID first uses a fusion module to perform a preliminary fusion of infrared image features with visible light image features. It then fuses the visible light and infrared features using the Concat function. This process continues until the final fusion output. The resulting fused image not only retains rich detailed texture information but also incorporates a large amount of infrared information, providing a more comprehensive picture. Finally, the bimodal fused features are input into the monitoring layer for monitoring. A first fusion module can be constructed, wherein the first fusion module is used to perform bimodal fusion processing on the second visible light fusion feature and the second infrared fusion feature to obtain a first bimodal fusion feature. Then, a second fusion module is constructed, wherein the second fusion module is used to perform bimodal fusion processing on the third visible light fusion feature and the third infrared fusion feature to obtain a second bimodal fusion feature. Finally, a third fusion module is constructed, wherein the third fusion module is used to perform bimodal fusion processing on the fourth visible light fusion feature and the fourth infrared fusion feature to obtain a third bimodal fusion feature. Finally, a detection head layer is constructed, wherein the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, which includes the target bounding box, category, and confidence. Furthermore, a detection head layer can be constructed after the first fusion module and the second fusion module respectively to perform multi-angle prediction. It can be understood that the fusion module is embedded in the output position of the dual-path neck network layer (visible light neck network layer and infrared light neck network layer) of the YOLOv8-TID model. The main function of the fusion module is to fuse the input dual-path feature information (dual-modal features) to enhance the expression performance of the fused image.

[0175] In some embodiments, performing bimodal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain a first bimodal fusion feature includes:

[0176] splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature;

[0177] Performing compression and activation processing on the first splicing feature to obtain a compressed and activated feature;

[0178] According to the preset point convolution frequency, the point convolution layer is used to perform point convolution on the compression and activation features to obtain the global features;

[0179] Calculate the target weight according to the preset activation function and global features. The target weight includes visible light weight and infrared light weight.

[0180] Multiply the visible light weight and the second visible light fusion feature to obtain the visible light multiplication feature;

[0181] Multiply the infrared light weight and the second infrared light fusion feature to obtain the infrared light multiplication feature;

[0182] Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first added feature;

[0183] Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature;

[0184] Splicing the first added feature and the second added feature to obtain a second spliced ​​feature;

[0185] Perform point convolution on the second splicing feature to obtain the first bimodal fusion feature.

[0186] In some embodiments, the fusion module structure is as follows Figure 7 As shown, the second visible light fusion feature and the second infrared light fusion feature can be spliced ​​together to obtain a first spliced ​​feature, and the first spliced ​​feature can be compressed and activated to obtain a compressed and activated feature for adjusting the channel weight of the feature map. Subsequently, convolution smoothing can be used to generate more accurate information. For example, the SENet attention module can be used for compression and activation processing. Then, according to the preset point convolution frequency, the point convolution layer is used to perform point convolution on the compressed and activated features to obtain a global feature. The mathematical expression of splicing, compression and activation processing, and point convolution is: Z r =PW_Conv 2 (SE(C(I vi +I ir ))), where Z r is the processed global feature, PW_Conv k (·) is the point convolution layer, k is the preset point convolution frequency, SE(·) is the compression and activation function, C is the Concat function for feature splicing, I vi is the second visible light fusion feature, I iris the second infrared light fusion feature. Then, according to the preset activation function and the global feature, the target weight is calculated, wherein the target weight includes the visible light weight and the infrared light weight. Exemplarily, the preset activation function may include a Sigmoid function, and the Sigmoid function may be used in the channel dimension to generate the visible light weight and the infrared light weight. Since visible light and infrared light images have complementary features, the visible light weight can be set to w, and the infrared light weight can be set to 1-w. The calculation formula of the visible light weight is: w=σ(F i (Z r )), where w is the visible light weight, σ is the Sigmoid function, and F i is a set of results for batch processing. The visible light weight and the second visible light fusion feature are then multiplied to obtain a visible light multiplication feature, and the infrared light weight and the second infrared light fusion feature are multiplied to obtain an infrared light multiplication feature. It is understandable that the w value can be dynamically adjusted according to the different weight values ​​in the fusion feature to generate a multiplication feature and obtain the local features of the image. The visible light multiplication feature and the second infrared light fusion feature are then added to obtain a first addition feature, and the infrared light multiplication feature and the second visible light fusion feature are added to obtain a second addition feature. It is understandable that the multiplication feature is added to the fusion feature and convolution smoothing is performed so that the fusion feature has both global features and local features. The mathematical expressions for multiplication and addition are: Where, F vi is the second additive feature, F ir is the first added feature, is the element-wise product, is the element-wise sum. Finally, the first added feature and the second added feature are concatenated to obtain the second concatenated feature, and the second concatenated feature is point convolved to obtain the first bimodal fusion feature. The mathematical expression of concatenation and point convolution is: FF r =PW_Conv(C(F vi , F ir )), where FF r is the first bimodal fusion feature, and PW_Conv(·) is the point convolution. It can be understood that the first bimodal fusion feature is obtained from the second visible light fusion feature and the second infrared light fusion feature, the second bimodal fusion feature is obtained from the third visible light fusion feature and the third infrared light fusion feature, and the third bimodal fusion feature is obtained from the fourth visible light fusion feature and the fourth infrared light fusion feature. Their bimodal fusion processing processes are similar.

[0187] In some embodiments, the first splicing feature is compressed and activated to obtain the compressed and activated feature, including:

[0188] Performing feature transformation on the first splicing feature to obtain a transformed feature;

[0189] Perform global average pooling on the transformed features to obtain compressed features;

[0190] According to the ReLU activation function, the compressed features are input into the first fully connected layer to obtain the excitation features;

[0191] According to the Sigmoid activation function, the excitation features are input into the second fully connected layer to obtain the channel weights;

[0192] Multiply the channel weight and the conversion feature to obtain the compression and activation features.

[0193] In some embodiments, the SENet attention module structure is as follows Figure 8 As shown, the first spliced ​​feature can be first transformed to obtain a transformed feature, and then the transformed feature can be globally averaged pooled to obtain a compressed feature. For example, the size of the input first spliced ​​feature can be set to H×W×C, which respectively represents the height H, width W and number of channels C of the feature map. The size remains unchanged after the feature transformation, and a global average pooling operation is performed on the transformed feature. This step is the compression operation. This process can aggregate the spatial dimension of the input first spliced ​​feature to generate a one-dimensional vector, namely the channel descriptor. The mathematical expression of the compression operation is: Where Zc is the weight of the cth channel, H is the height of the conversion feature, W is the width of the conversion feature, and u c(i，j) is the value of the cth channel at position (i, j) in the input conversion feature u. It can be understood that the feature map with C channels can be transformed into a 1×1×C feature map through the compression operation to obtain the global receptive field. Then, according to the ReLU activation function, the compressed feature is input into the first fully connected layer to obtain the excitation feature, and according to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight. Exemplarily, after the compression operation, the obtained global information (i.e., the compressed feature) is excited. The excitation operation adopts a simple gating mechanism, and its process needs to pass through two fully connected layers (i.e., the first fully connected layer and the second fully connected layer) and activation functions (i.e., the ReLU activation function and the Sigmoid activation function). The mathematical expression of the excitation operation is: s = σ(W2δ(W1z)), where s is the channel weight, σ is the Sigmoid activation function, W2 is the second fully connected layer, used to adjust the number of channels to the original number, δ is the ReLU activation function, W1 is the first fully connected layer, used to adjust the number of channels to the shrinkage rate 1 / r, and z is the compression feature. Finally, the channel weight and the conversion feature are multiplied to obtain the compression and activation features. For example, the obtained channel weight s is multiplied with the conversion feature u cMultiply the corresponding channels to obtain the weighted feature sequence (i.e., compression and activation features). This feature sequence is the output of the SENet attention mechanism, and its mathematical expression is: c =s c u c , where x c is the compression and activation feature, S c is the channel weight of the cth channel, u c is the conversion feature. Compared with the input local feature sequence (first spliced ​​feature), the feature sequence (compression and activation feature) output by the SENet attention mechanism contains global context information. It can be understood that the compression and activation function SENet (Squeeze-and-Excitation Networks) in the fusion module is a channel attention mechanism. This is a lightweight gated attention mechanism that can explicitly model the interdependence between convolutional feature channels. The core idea of ​​the SENet attention mechanism is to enhance the representation ability of the model through two key operations: compression (Squeeze) and then excitation (Excitation).

[0194] In some embodiments, the overall process of model training and ice cover monitoring is as follows: Figure 9 As shown, a dataset of wind turbine blade icing images can be constructed. A bimodal fusion module is then introduced to build the YOL0v8-TID wind blade icing recognition model. The dataset is then fed into the YOL0v8-TID model, and model parameters are adjusted based on training results. The trained model is then applied to wind turbine icing images to determine the icing threshold. When icing occurs, the model's prediction result is recorded as the icing threshold. Finally, the model is used to monitor wind blade icing and obtain monitoring results.

[0195] The beneficial effects of implementing embodiments of the present invention include: first, acquiring an image of an iced wind turbine blade, then inputting the image of the iced wind turbine blade into a wind blade icing recognition model to obtain an ice occurrence threshold, and finally, performing wind blade icing monitoring based on the ice occurrence threshold and the wind blade icing recognition model. This enables wind blade icing monitoring through model recognition and threshold comparison, thereby improving accuracy. The wind blade icing recognition model is obtained by first acquiring a dataset of wind blade icing images, then constructing a dual-modal image fusion model, and finally inputting the dataset of wind blade icing images into the dual-modal image fusion model to train the dual-modal image fusion model, thereby improving model accuracy.

[0196] like Figure 10 As shown, an embodiment of the present invention further provides a wind turbine blade icing monitoring device based on dual-modal image fusion, comprising:

[0197] The first module 801 is used to obtain an image of an ice-covered wind turbine blade;

[0198] The second module 802 is configured to input the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold;

[0199] The third module 803 is used to monitor wind turbine blade icing based on an icing threshold and a wind turbine blade icing identification model;

[0200] The wind turbine blade icing identification model is obtained through the following steps:

[0201] Acquire a wind turbine blade ice-covered image dataset, where the wind turbine blade ice-covered image dataset includes visible light images and infrared light images;

[0202] Construct a dual-modal image fusion model;

[0203] The wind blade icing image dataset is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain a wind blade icing recognition model.

[0204] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0205] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A wind turbine blade icing monitoring method based on dual-modal image fusion is characterized by: The following steps are involved: Acquire images of ice-covered wind turbine blades; Inputting the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold; Monitoring wind turbine blade icing according to the icing occurrence threshold and the wind turbine blade icing identification model; The wind turbine blade icing identification model is obtained by the following steps: Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image; Construct a dual-modal image fusion model; Inputting the wind turbine blade icing image dataset into the dual-modal image fusion model to train the dual-modal image fusion model and obtain the wind turbine blade icing recognition model; The construction process of the dual-modal image fusion model includes: Constructing a visible light input layer and an infrared light input layer respectively; After the visible light input layer, a visible light backbone network layer is constructed; After the infrared light input layer, an infrared light backbone network layer is constructed; After the visible light backbone network layer, a visible light neck network layer is constructed; After the infrared light backbone network layer, an infrared light neck network layer is constructed; After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed; The construction process of the visible light backbone network layer includes: Constructing a first visible light convolution composite layer, the first visible light convolution composite layer including three standard convolution modules and two C2f modules, the first visible light convolution composite layer being used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature; Constructing a second visible light convolution composite layer, the second visible light convolution composite layer including the standard convolution module and the C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature; Constructing a third visible light convolution composite layer, the third visible light convolution composite layer including the standard convolution module, the C2f module and the fast spatial pyramid pooling module, the third visible light convolution composite layer being used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature; The construction process of the visible light neck network layer includes: Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature; Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature; Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature; Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature; Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature; Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature; Constructing a first standard convolution layer, where the first standard convolution layer is used to perform standard convolution on the second visible light fusion feature to obtain a first standard convolution feature; Constructing a third visible light feature connection layer, wherein the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature; constructing a third visible light C2f layer, wherein the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature; Constructing a second standard convolution layer, where the second standard convolution layer is used to perform standard convolution on the third visible light fusion feature to obtain a second standard convolution feature; Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature; A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

2. The method according to claim 1, characterized in that The construction process of the infrared optical backbone network layer includes: Constructing a first infrared light convolution composite layer, the first infrared light convolution composite layer including three standard convolution modules and two C2f modules, the first infrared light convolution composite layer being used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature; Constructing a second infrared light convolution composite layer, the second infrared light convolution composite layer including the standard convolution module and the C2f module, the second infrared light convolution composite layer being used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature; Construct a third infrared light convolution composite layer, wherein the third infrared light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature.

3. The method according to claim 2, characterized in that The construction process of the infrared neck network layer includes: Constructing a third upsampling layer, wherein the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature; Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampling feature and the second infrared light convolution feature to obtain a first infrared light connection feature; constructing a first infrared light C2f layer, where the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature; Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature; Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampling feature and the first infrared light convolution feature to obtain a second infrared light connection feature; constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection feature to obtain a second infrared light fusion feature; Constructing a third standard convolution layer, wherein the third standard convolution layer is used to perform standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature; Constructing a third infrared light feature connection layer, wherein the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature; constructing a third infrared light C2f layer, wherein the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature; Constructing a fourth standard convolution layer, wherein the fourth standard convolution layer is used to perform standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature; Constructing a fourth infrared light feature connection layer, wherein the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature; A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

4. The method according to claim 3, characterized in that The construction process of the head network layer includes: Constructing a first fusion module, wherein the first fusion module is configured to perform bimodal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first bimodal fusion feature; Constructing a second fusion module, wherein the second fusion module is used to perform bimodal fusion processing on the third visible light fusion feature and the third infrared light fusion feature to obtain the second bimodal fusion feature; Constructing a third fusion module, wherein the third fusion module is used to perform bimodal fusion processing on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third bimodal fusion feature; A detection head layer is constructed, where the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, where the prediction result includes a target bounding box, a category, and a confidence level.

5. The method according to claim 4, characterized in that The performing bimodal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first bimodal fusion feature includes: splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature; performing compression and activation processing on the first splicing feature to obtain a compressed and activated feature; According to a preset point convolution frequency, the compression and activation features are point-convolved using a point convolution layer to obtain a global feature; Calculating a target weight according to a preset activation function and the global feature, the target weight including a visible light weight and an infrared light weight; Multiplying the visible light weight and the second visible light fusion feature to obtain a visible light multiplication feature; multiplying the infrared light weight and the second infrared light fusion feature to obtain an infrared light multiplication feature; Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first addition feature; Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature; Splicing the first added feature and the second added feature to obtain a second spliced ​​feature; Perform point convolution on the second splicing feature to obtain the first bimodal fusion feature.

6. The method according to claim 5, characterized in that The compressing and activating the first splicing feature to obtain the compressed and activated feature includes: Performing feature conversion on the first splicing feature to obtain a conversion feature; Performing global average pooling on the transformed features to obtain compressed features; According to the ReLU activation function, the compressed features are input into the first fully connected layer to obtain the excitation features; According to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight; The channel weight and the conversion feature are multiplied to obtain the compression and activation feature.

7. A wind turbine blade icing monitoring device based on dual-modal image fusion is characterized by: include: The first module is used to obtain images of ice-covered wind turbine blades; The second module is configured to input the ice-covered wind turbine blade image into a wind turbine blade ice recognition model to obtain an ice occurrence threshold; A third module is used to monitor wind turbine blade icing based on the icing occurrence threshold and the wind turbine blade icing identification model; The wind turbine blade icing identification model is obtained by the following steps: Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image; Construct a dual-modal image fusion model; Inputting the wind turbine blade icing image dataset into the dual-modal image fusion model to train the dual-modal image fusion model and obtain the wind turbine blade icing recognition model; The construction process of the dual-modal image fusion model includes: Constructing a visible light input layer and an infrared light input layer respectively; After the visible light input layer, a visible light backbone network layer is constructed; After the infrared light input layer, an infrared light backbone network layer is constructed; After the visible light backbone network layer, a visible light neck network layer is constructed; After the infrared light backbone network layer, an infrared light neck network layer is constructed; After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed; The construction process of the visible light backbone network layer includes: Constructing a first visible light convolution composite layer, the first visible light convolution composite layer including three standard convolution modules and two C2f modules, the first visible light convolution composite layer being used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature; Constructing a second visible light convolution composite layer, the second visible light convolution composite layer including the standard convolution module and the C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature; Constructing a third visible light convolution composite layer, the third visible light convolution composite layer including the standard convolution module, the C2f module and the fast spatial pyramid pooling module, the third visible light convolution composite layer being used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature; The construction process of the visible light neck network layer includes: Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature; Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature; Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature; Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature; Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature; Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature; Constructing a first standard convolution layer, where the first standard convolution layer is used to perform standard convolution on the second visible light fusion feature to obtain a first standard convolution feature; Constructing a third visible light feature connection layer, wherein the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature; constructing a third visible light C2f layer, wherein the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature; Constructing a second standard convolution layer, where the second standard convolution layer is used to perform standard convolution on the third visible light fusion feature to obtain a second standard convolution feature; Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature; A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

Images