Fan blade icing monitoring method and device based on bimodal image fusion

Abstract

The invention discloses a fan blade icing monitoring method and device based on bimodal image fusion. The method comprises the following steps: acquiring an iced fan blade image; inputting the image of the iced fan blade into a fan blade icing identification model to obtain an icing occurrence threshold value; and according to the icing occurrence threshold value and the fan blade icing identification model, carrying out fan blade icing monitoring. Wherein the fan blade icing identification model is obtained through the following steps that a fan blade icing image data set is obtained, and the fan blade icing image data set comprises a visible light image and an infrared light image; constructing a bimodal image fusion model; and inputting the fan blade icing image data set into the bimodal image fusion model, so that the bimodal image fusion model is trained to obtain the fan blade icing identification model. According to the invention, the icing monitoring of the fan blade is realized, and the accuracy is improved. The method can be widely applied to the technical field of image recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for monitoring ice coating on fan blades based on dual-modal image fusion. Background Art

[0002] Ice on wind turbine blades can affect the normal operation of wind turbines. In the prior art, traditional wind turbine blade ice monitoring methods mainly include visible light monitoring methods and infrared monitoring methods. The images collected by the visible light monitoring method have rich texture information, but are easily affected by occlusion or insufficient light. The infrared monitoring method can capture the thermal radiation information of wind turbine blades, but the infrared image is a single-channel image, which is difficult to provide detailed information of the target scene, and the accuracy of ice recognition is low.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for monitoring ice coating on wind turbine blades based on dual-modal image fusion, which effectively improves the accuracy of identifying ice coating on wind turbine blades.

[0005] On the one hand, an embodiment of the present invention provides a wind turbine blade icing monitoring method based on dual-modal image fusion, comprising the following steps:

[0006] Acquire images of ice-covered wind turbine blades;

[0007] Inputting the ice-covered wind turbine blade image into a wind turbine blade ice-covering recognition model to obtain an ice-covering occurrence threshold;

[0008] Monitoring wind blade icing according to the icing occurrence threshold and the wind blade icing identification model;

[0009] The wind turbine blade icing identification model is obtained by the following steps:

[0010] Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image;

[0011] Construct a dual-modal image fusion model;

[0012] The wind turbine blade icing image data set is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

[0013] In some embodiments, the process of constructing the dual-modal image fusion model includes:

[0014] Constructing a visible light input layer and an infrared light input layer respectively;

[0015] After the visible light input layer, a visible light backbone network layer is constructed;

[0016] After the infrared light input layer, an infrared light backbone network layer is constructed;

[0017] After the visible light backbone network layer, a visible light neck network layer is constructed;

[0018] After the infrared light backbone network layer, an infrared light neck network layer is constructed;

[0019] After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed.

[0020] In some embodiments, the construction process of the visible light backbone network layer includes:

[0021] Constructing a first visible light convolution composite layer, the first visible light convolution composite layer comprising three standard convolution modules and two C2f modules, the first visible light convolution composite layer being used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature;

[0022] Constructing a second visible light convolution composite layer, the second visible light convolution composite layer comprising the standard convolution module and the C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature;

[0023] A third visible light convolution composite layer is constructed, wherein the third visible light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature.

[0024] In some embodiments, the process of constructing the visible light neck network layer includes:

[0025] Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature;

[0026] Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature;

[0027] Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature;

[0028] Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature;

[0029] Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature;

[0030] Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature;

[0031] Constructing a first standard convolution layer, where the first standard convolution layer is used to perform a standard convolution on the second visible light fusion feature to obtain a first standard convolution feature;

[0032] Constructing a third visible light feature connection layer, where the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature;

[0033] Constructing a third visible light C2f layer, wherein the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature;

[0034] Constructing a second standard convolution layer, where the second standard convolution layer is used to perform a standard convolution on the third visible light fusion feature to obtain a second standard convolution feature;

[0035] Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature;

[0036] A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

[0037] In some embodiments, the construction process of the infrared optical backbone network layer includes:

[0038] Constructing a first infrared light convolution composite layer, wherein the first infrared light convolution composite layer includes three standard convolution modules and two C2f modules, and the first infrared light convolution composite layer is used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature;

[0039] Constructing a second infrared light convolution composite layer, wherein the second infrared light convolution composite layer includes the standard convolution module and the C2f module, and the second infrared light convolution composite layer is used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature;

[0040] A third infrared light convolution composite layer is constructed, wherein the third infrared light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature.

[0041] In some embodiments, the construction process of the infrared light neck network layer includes:

[0042] Constructing a third upsampling layer, wherein the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature;

[0043] Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampling feature and the second infrared light convolution feature to obtain a first infrared light connection feature;

[0044] Constructing a first infrared light C2f layer, wherein the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature;

[0045] Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature;

[0046] Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampling feature and the first infrared light convolution feature to obtain a second infrared light connection feature;

[0047] Constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection feature to obtain a second infrared light fusion feature;

[0048] Constructing a third standard convolution layer, wherein the third standard convolution layer is used to perform a standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature;

[0049] Constructing a third infrared light feature connection layer, wherein the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature;

[0050] Constructing a third infrared light C2f layer, wherein the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature;

[0051] Constructing a fourth standard convolution layer, wherein the fourth standard convolution layer is used to perform a standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature;

[0052] Constructing a fourth infrared light feature connection layer, wherein the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature;

[0053] A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

[0054] In some embodiments, the process of constructing the header network layer includes:

[0055] Constructing a first fusion module, wherein the first fusion module is used to perform a dual-modal fusion process on the second visible light fusion feature and the second infrared light fusion feature to obtain the first dual-modal fusion feature;

[0056] Constructing a second fusion module, wherein the second fusion module is used to perform a dual-modal fusion process on the third visible light fusion feature and the third infrared light fusion feature to obtain the second dual-modal fusion feature;

[0057] Constructing a third fusion module, wherein the third fusion module is used to perform a dual-modal fusion process on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third dual-modal fusion feature;

[0058] A detection head layer is constructed, where the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, where the prediction result includes a target bounding box, a category, and a confidence level.

[0059] In some embodiments, performing dual-modal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first dual-modal fusion feature includes:

[0060] splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature;

[0061] Performing compression and activation processing on the first splicing feature to obtain a compression and activation feature;

[0062] According to a preset point convolution frequency, using a point convolution layer to perform point convolution on the compression and activation features to obtain a global feature;

[0063] Calculate a target weight according to a preset activation function and the global feature, wherein the target weight includes a visible light weight and an infrared light weight;

[0064] Multiplying the visible light weight and the second visible light fusion feature to obtain a visible light multiplication feature;

[0065] Multiplying the infrared light weight and the second infrared light fusion feature to obtain an infrared light multiplication feature;

[0066] Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first addition feature;

[0067] Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature;

[0068] Concatenate the first added feature and the second added feature to obtain a second concatenated feature;

[0069] Perform point convolution on the second splicing feature to obtain the first bimodal fusion feature.

[0070] In some embodiments, compressing and activating the first splicing feature to obtain the compressed and activated feature includes:

[0071] Performing feature conversion on the first splicing feature to obtain a conversion feature;

[0072] Performing global average pooling on the conversion features to obtain compressed features;

[0073] According to the ReLU activation function, the compression feature is input into the first fully connected layer to obtain an excitation feature;

[0074] According to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight;

[0075] The channel weight and the conversion feature are multiplied to obtain the compression and activation feature.

[0076] On the other hand, an embodiment of the present invention provides a wind turbine blade icing monitoring device based on dual-modal image fusion, comprising:

[0077] The first module is used to obtain images of ice-covered wind turbine blades;

[0078] The second module is used to input the ice-covered wind turbine blade image into a wind turbine blade ice-covering recognition model to obtain an ice-covering threshold;

[0079] A third module is used to monitor the icing of the fan blades according to the icing occurrence threshold and the fan blade icing identification model;

[0080] The wind turbine blade icing identification model is obtained by the following steps:

[0081] Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image;

[0082] Construct a dual-modal image fusion model;

[0083] The wind turbine blade icing image data set is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

[0084] The beneficial effects of the present invention are as follows:

[0085] The embodiment of the present invention first obtains an image of an iced fan blade, then inputs the image of the iced fan blade into a fan blade ice recognition model to obtain an ice occurrence threshold, and finally monitors the ice on the fan blade according to the ice occurrence threshold and the fan blade ice recognition model, thereby realizing the fan blade ice monitoring through model recognition and threshold comparison, thereby improving the accuracy. The fan blade ice recognition model first obtains a fan blade ice image dataset, then constructs a dual-modal image fusion model, and finally inputs the fan blade ice image dataset into the dual-modal image fusion model so that the dual-modal image fusion model is trained, thereby improving the model accuracy.

[0086] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1 This is a flow chart of a method for monitoring wind turbine blade icing based on dual-modal image fusion according to an embodiment of the present invention;

[0089] Figure 2 This is a flow chart of determining an ice occurrence threshold of a wind turbine blade according to an embodiment of the present invention;

[0090] Figure 3 This is a flow chart of obtaining a wind turbine blade icing identification model according to an embodiment of the present invention;

[0091] Figure 4 A flowchart of building a dual-modal image fusion model according to an embodiment of the present invention;

[0092] Figure 5 A schematic diagram of a YOLOv8 model structure according to an embodiment of the present invention;

[0093] Figure 6 A schematic diagram of a dual-modal image fusion model structure according to an embodiment of the present invention;

[0094] Figure 7 A schematic diagram of a fusion module structure according to an embodiment of the present invention;

[0095] Figure 8 A schematic diagram of a SENet attention module structure according to an embodiment of the present invention;

[0096] Fig. 9 A schematic diagram of the overall process of model training and ice coverage monitoring according to an embodiment of the present invention;

[0097] Fig.10 The present invention is a schematic structural diagram of a wind turbine blade icing monitoring device based on dual-modal image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0099] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0100] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0101] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0102] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0103] C2f module: It is an important component in YOLOv8, mainly used for feature extraction and fusion.

[0104] Standard convolution module: Named Conv in the source code, it is composed of a two-dimensional convolution layer (Conv2D), a batch normalization layer (BatchNorm2D), and an activation layer (Activation) in series.

[0105] Concat component: It is an important component in YOLOv8, which is mainly used to connect feature maps from different layers in the channel dimension to generate richer feature maps to improve the accuracy of target detection. In the network, the Concat module usually appears in the structure of cross-layer connection.

[0106] In the related technology, in the current research, the methods for wind turbine blade ice image recognition mainly include infrared monitoring method and visible light monitoring method. The images collected by the visible light monitoring method have rich texture information and can enhance the detailed features of the wind turbine image, but they are easily affected by occlusion and insufficient light. The infrared monitoring method can capture the thermal radiation information of the wind turbine blades and has strong anti-interference ability. It can be used for wind turbine blade ice monitoring under complex backgrounds such as weak light, dark light, and blur. However, infrared images are single-channel images and it is difficult to provide detailed information of the target scene. In practical applications, their monitoring performance is often difficult to meet the needs of practical applications, and the accuracy of ice recognition is low. Based on the complementary characteristics of infrared monitoring method and visible light monitoring method, the two are fused to obtain a fused image with the advantages of visible light and infrared light, which enhances the accuracy of wind turbine blade ice monitoring, and is of great significance for improving the ability to monitor wind turbine blade ice all day.

[0107] In view of this, the embodiment of the present invention constructs a YOLOv8-TID wind turbine blade icing recognition model based on a dual-modal image fusion based on the YOLOv8 algorithm network. YOLOv8 is a new SOTA (State Of The Art) model proposed by Ultralytics, the R&D company of YOLOv5, in 2023. It has the advantages of fast speed, high accuracy, and strong robustness. At present, there is a large gap in the research on using the YOLOv8 model to monitor icing on wind turbine blades. The embodiment of the present invention combines the advantages of infrared monitoring methods and visible light monitoring methods to improve the accuracy of icing image recognition on wind turbine blades.

[0108] The wind turbine blade icing monitoring method based on dual-modal image fusion provided in the embodiment of the present application relates to the field of image recognition technology. The wind turbine blade icing monitoring method based on dual-modal image fusion provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a wind turbine blade icing monitoring method based on dual-modal image fusion, etc., but is not limited to the above forms.

[0109] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0110] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0111] Figure 1 is an optional flow chart of a wind turbine blade icing monitoring method based on dual-modal image fusion provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.

[0112] Step S101, acquiring an image of an ice-covered wind turbine blade;

[0113] Step S102, inputting the ice-covered wind turbine blade image into a wind turbine blade ice-covering recognition model to obtain an ice-covering threshold;

[0114] Step S103: monitoring the icing of the fan blades according to the icing occurrence threshold and the fan blade icing identification model.

[0115] In some embodiments, in step S101-step S103, an image of an iced fan blade may be first obtained as a standard image of ice, and then the image of the iced fan blade may be input into a fan blade ice recognition model to obtain an ice occurrence threshold, and finally, fan blade ice monitoring may be performed based on the ice occurrence threshold and the fan blade ice recognition model. Exemplarily, the process of determining the fan blade ice occurrence threshold is as follows: Figure 2 As shown, it is usually carried out in a constant temperature and humidity laboratory environment, and an artificial icing experiment is used to simulate the real situation. In the experiment, the confidence score of the fan blade icing recognition model is used as the monitoring target. When obvious ice accumulation is observed on the surface of the fan blade, the prediction box confidence score output by the YOLOv8 model is recorded at this time, and it is used as the icing occurrence threshold for image monitoring. Moreover, in the actual operation process, when the confidence score identified by the fan blade icing recognition model exceeds the icing occurrence threshold, it can be diagnosed as icing occurrence, so that the fan blade icing monitoring can be realized through model recognition and threshold comparison, thereby improving the accuracy.

[0116] In some embodiments, Figure 3 As shown, obtaining a wind turbine blade icing identification model may include but is not limited to steps S201 to S203.

[0117] Step S201, obtaining a wind turbine blade ice-covered image dataset, where the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image;

[0118] Step S202: constructing a dual-modal image fusion model;

[0119] Step S203: inputting the wind blade icing image data set into the dual-modal image fusion model so as to train the dual-modal image fusion model and obtain a wind blade icing recognition model.

[0120] Steps S201 to S203 shown in the embodiment of the present application obtain a wind turbine blade icing identification model, thereby improving the model accuracy.

[0121] In some embodiments, in step S201-step S203, a fan blade ice image dataset can be obtained through a fan blade image database. The fan blade ice image dataset can also be obtained by other means, but is not limited thereto. For example, images of fan blade ice can be obtained from the Internet or images of fan blade ice taken in a laboratory. The fan blade ice images are labeled using the open source software X-AnyLabeling to obtain a fan blade ice image dataset. The dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1:2. Then a bimodal image fusion model is constructed, and finally the fan blade ice image dataset is input into the bimodal image fusion model so that the bimodal image fusion model is trained to obtain a fan blade ice recognition model. For example, the bimodal image fusion model can be trained based on the fan blade ice image dataset, and through training iteration and optimization, it can accurately monitor and identify the ice condition on the fan blade. Furthermore, in the model training process, the prepared fan blade ice image dataset can be used for model training. First, configure the training parameters, set the training cycle (epoch) to 300, threads (workers) to 8, batch size (batchsize) to 8, input photo size (imgsz) to 640, and the initial learning rate (lr0) to 0.01. Then load the YOLOv8 model and download the pre-trained weights of the YOLOv8 model as the initial state of the model. During the training process, the model will learn to identify ice-covered areas on the blades. Monitor the training process and adjust the parameters according to the performance of the validation set to prevent overfitting. After achieving satisfactory performance, save the trained YOLOv8 model weights for subsequent experiments.

[0122] In some embodiments, in step S202, Figure 4 As shown, the construction process of the dual-modal image fusion model may include but is not limited to steps S301 to S306.

[0123] Step S301, constructing a visible light input layer and an infrared light input layer respectively;

[0124] Step S302: after the visible light input layer, construct a visible light backbone network layer;

[0125] Step S303: after the infrared light input layer, construct an infrared light backbone network layer;

[0126] Step S304: construct a visible light neck network layer after the visible light backbone network layer;

[0127] Step S305, constructing an infrared light neck network layer after the infrared light backbone network layer;

[0128] Step S306: construct a head network layer after the visible light neck network layer and the infrared light neck network layer.

[0129] In some embodiments, the traditional YOLOv8 model such as Figure 5 As shown, it is a single-stage target detection model based on candidate areas. The algorithm consists of an input end (Input), a backbone network (Backbone), a neck network (Neck) and a detection head (Detect). The main idea is to predict the position and category of the target object through grid division and anchor box. Among them, the Backbone layer mainly uses a multi-layer convolutional neural network to gradually extract the feature information of the image. As the convolution deepens, the values ​​of H and W gradually decrease, from the initial 640×640 to 20×20. The size reduction means that the image feature map contains deeper feature information, which is helpful for the detection and recognition of large targets. The Neck layer is mainly responsible for fusing the information passed by the Backbone, so that the generated feature map has both deep semantic information and shallow detail texture information. The Detect layer mainly predicts the feature map passed by the Neck layer, and outputs the detected target bounding box, category and confidence information. The YOLOv8-TID model (i.e., the dual-modal image fusion model) proposed in this embodiment is an improvement on the traditional YOLOv8 model. Among them, the dual-modal image fusion model aims to improve the accuracy of image processing by fusing features at different levels. The dual-modal image fusion model consists of a dual input layer, a dual-path backbone network layer, a dual-path neck network layer, and a monitoring layer, in which the dual-path neck network layer performs fusion operations.

[0130] In some embodiments, in step S301, the dual-modal image fusion model structure is as follows: Figure 6 As shown, the visible light input layer and the infrared light input layer can be constructed separately so as to extract the features of visible light and infrared light respectively. Furthermore, in the subsequent feature extraction process, the dual-path backbone network layer (visible light backbone network layer and infrared light backbone network layer) can be used to extract the features of visible light and infrared light images respectively, and output them to the dual-path neck network layer (visible light neck network layer and infrared light neck network layer), and enhanced by the upsampling fusion process. Given that the dual-path neck network layer has three output links, the YOLOv8-TID model (dual-modal image fusion model) embeds a fusion module at each output position of the dual-path neck network layer to fuse the enhanced features.

[0131] In some embodiments, in step S302, the process of constructing the visible light backbone network layer includes:

[0132] Constructing a first visible light convolution composite layer, the first visible light convolution composite layer includes 3 standard convolution modules and 2 C2f modules, and the first visible light convolution composite layer is used to extract the first visible light feature of the visible light image to obtain the first visible light convolution feature;

[0133] Constructing a second visible light convolution composite layer, the second visible light convolution composite layer includes a standard convolution module and a C2f module, and the second visible light convolution composite layer is used to extract a second visible light feature from the first visible light convolution feature to obtain a second visible light convolution feature;

[0134] A third visible light convolution composite layer is constructed, the third visible light convolution composite layer includes a standard convolution module, a C2f module and a fast spatial pyramid pooling module, and the third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature.

[0135] In some embodiments, a first visible light convolution composite layer may be constructed first, then a second visible light convolution composite layer may be constructed, and finally a third visible light convolution composite layer may be constructed. Among them, the first visible light convolution composite layer includes 3 standard convolution modules and 2 C2f modules, the second visible light convolution composite layer includes a standard convolution module and a C2f module, and the third visible light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. In the feature extraction process, the first visible light convolution composite layer is used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature. The second visible light convolution composite layer is used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature. The third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature. It can be understood that the composition structures of different visible light convolution composite layers are different, so different visible light feature extraction processes are also different. Exemplarily, the standard convolution module may include a Conv module, which is assembled in series by a two-dimensional convolution layer (Conv2D), a batch normalization layer (BatchNorm2D), and an activation layer (SiLu). The C2f module consists of two BottleNeck layers, a separation layer (Split), a feature connection layer (Concat), and two standard convolution modules (Conv). The Bottleneck layer is an important component introduced in the deep residual network (ResNet) to reduce the computational complexity of the model and improve the feature extraction capability. The fast spatial pyramid pooling module may include an SPPF (Spatial Pyramid Pooling Fast) module, which is an important component in the YOLOv5 model, mainly used for feature extraction and enhancement. SPPF is a fast implementation version of the SPP (Spatial Pyramid Pooling) module, which captures feature information of different scales by using multiple pooling kernels of different sizes, thereby enhancing the generalization ability of the model.

[0136] In some embodiments, in step S303, the construction process of the infrared optical backbone network layer includes:

[0137] Constructing a first infrared light convolution composite layer, the first infrared light convolution composite layer includes 3 standard convolution modules and 2 C2f modules, and the first infrared light convolution composite layer is used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature;

[0138] Constructing a second infrared light convolution composite layer, the second infrared light convolution composite layer comprising a standard convolution module and a C2f module, the second infrared light convolution composite layer being used to extract a second infrared light feature from the first infrared light convolution feature to obtain a second infrared light convolution feature;

[0139] A third infrared light convolution composite layer is constructed, the third infrared light convolution composite layer includes a standard convolution module, a C2f module and a fast spatial pyramid pooling module, and the third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature.

[0140] In some embodiments, a first infrared light convolution composite layer may be constructed first, then a second infrared light convolution composite layer may be constructed, and finally a third infrared light convolution composite layer may be constructed. The first infrared light convolution composite layer includes three standard convolution modules and two C2f modules, the second infrared light convolution composite layer includes a standard convolution module and a C2f module, and the third infrared light convolution composite layer includes a standard convolution module, a C2f module, and a fast spatial pyramid pooling module. In the feature extraction process, the first infrared light convolution composite layer is used to perform a first infrared light feature extraction on the infrared light image to obtain a first infrared light convolution feature. The second infrared light convolution composite layer is used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature. The third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature. It can be understood that the infrared light backbone network layer and the visible light backbone network layer are similar in structure and feature extraction process.

[0141] In some embodiments, in step S304, the process of constructing the visible light neck network layer includes:

[0142] Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature;

[0143] Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature;

[0144] Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature;

[0145] Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature;

[0146] Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature;

[0147] Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature;

[0148] Constructing a first standard convolution layer, where the first standard convolution layer is used to perform a standard convolution on the second visible light fusion feature to obtain a first standard convolution feature;

[0149] Constructing a third visible light feature connection layer, where the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature;

[0150] Constructing a third visible light C2f layer, where the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature;

[0151] Constructing a second standard convolution layer, the second standard convolution layer is used to perform a standard convolution on the third visible light fusion feature to obtain a second standard convolution feature;

[0152] Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature;

[0153] A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

[0154] In some embodiments, a first upsampling layer, a first visible light feature connection layer, a first visible light C2f layer, a second upsampling layer, a second visible light feature connection layer, a second visible light C2f layer, a first standard convolution layer, a third visible light feature connection layer, a third visible light C2f layer, a second standard convolution layer, a fourth visible light feature connection layer, and a fourth visible light C2f layer may be constructed in sequence. In the feature extraction process, the third visible light convolution feature may be upsampled by the first upsampling layer to obtain a first upsampled feature; the first upsampled feature and the second visible light convolution feature may be feature connected by the first visible light feature connection layer to obtain a first visible light connection feature; the first visible light connection feature may be feature fused by the first visible light C2f layer to obtain a first visible light fusion feature; the first visible light fusion feature may be upsampled by the second upsampling layer to obtain a second upsampled feature; the second upsampled feature and the first visible light convolution feature may be feature connected by the second visible light feature connection layer to obtain a second visible light connection feature; the second visible light connection feature may be feature fused by the second visible light C2f layer to obtain a second visible light fusion feature. ; The second visible light fusion feature can be subjected to standard convolution through the first standard convolution layer to obtain the first standard convolution feature; the first standard convolution feature and the first visible light fusion feature can be subjected to feature connection through the third visible light feature connection layer to obtain the third visible light connection feature; the third visible light connection feature can be subjected to feature fusion through the third visible light C2f layer to obtain the third visible light fusion feature; the third visible light fusion feature can be subjected to standard convolution through the second standard convolution layer to obtain the second standard convolution feature; the second standard convolution feature and the third visible light convolution feature can be subjected to feature connection through the fourth visible light feature connection layer to obtain the fourth visible light connection feature; the fourth visible light connection feature can be subjected to feature fusion through the fourth visible light C2f layer to obtain the fourth visible light fusion feature. It can be understood that, Figure 6 The Concat in represents the feature connection layer.

[0155] In some embodiments, in step S305, the construction process of the infrared light neck network layer includes:

[0156] Constructing a third upsampling layer, where the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature;

[0157] Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampling feature and the second infrared light convolution feature to obtain a first infrared light connection feature;

[0158] Constructing a first infrared light C2f layer, where the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature;

[0159] Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature;

[0160] Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampling feature and the first infrared light convolution feature to obtain a second infrared light connection feature;

[0161] Constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection feature to obtain a second infrared light fusion feature;

[0162] Constructing a third standard convolution layer, the third standard convolution layer is used to perform standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature;

[0163] Constructing a third infrared light feature connection layer, where the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature;

[0164] Constructing a third infrared light C2f layer, where the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature;

[0165] Constructing a fourth standard convolution layer, the fourth standard convolution layer is used to perform standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature;

[0166] Constructing a fourth infrared light feature connection layer, where the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature;

[0167] A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

[0168] In some embodiments, a third upsampling layer, a first infrared light feature connection layer, a first infrared light C2f layer, a fourth upsampling layer, a second infrared light feature connection layer, a second infrared light C2f layer, a third standard convolution layer, a third infrared light feature connection layer, a third infrared light C2f layer, a fourth standard convolution layer, a fourth infrared light feature connection layer, and a fourth infrared light C2f layer may be constructed in sequence. In the feature extraction process, the third infrared light convolution feature may be upsampled by the third upsampling layer to obtain a third upsampled feature; the third upsampled feature and the second infrared light convolution feature may be feature connected by the first infrared light feature connection layer to obtain a first infrared light connection feature; the first infrared light connection feature may be feature fused by the first infrared light C2f layer to obtain a first infrared light fusion feature; the first infrared light fusion feature may be upsampled by the fourth upsampling layer to obtain a fourth upsampled feature; the fourth upsampled feature and the first infrared light convolution feature may be feature connected by the second infrared light feature connection layer to obtain a second infrared light connection feature; the second infrared light connection feature may be feature fused by the second infrared light C2f layer to obtain a second infrared light fusion feature. ; The first bimodal fusion feature can be subjected to standard convolution through the third standard convolution layer to obtain a third standard convolution feature; the third standard convolution feature and the first infrared light fusion feature can be feature-connected through the third infrared light feature connection layer to obtain a third infrared light connection feature; the third infrared light connection feature can be feature-fused through the third infrared light C2f layer to obtain a third infrared light fusion feature; the second bimodal fusion feature can be subjected to standard convolution through the fourth standard convolution layer to obtain a fourth standard convolution feature; the fourth standard convolution feature and the third infrared light convolution feature can be feature-connected through the fourth infrared light feature connection layer to obtain a fourth infrared light connection feature; the fourth infrared light connection feature can be feature-fused through the fourth infrared light C2f layer to obtain a fourth infrared light fusion feature. It can be understood that, unlike the visible light neck network layer, the infrared light neck network layer uses the output of the fusion module as the input of the third standard convolution layer and the fourth standard convolution layer, so that the visible light feature and the infrared light feature are further fused.

[0169] In some embodiments, in step S306, the process of constructing the header network layer includes:

[0170] Constructing a first fusion module, the first fusion module is used to perform a dual-modal fusion process on the second visible light fusion feature and the second infrared light fusion feature to obtain a first dual-modal fusion feature;

[0171] Constructing a second fusion module, the second fusion module is used to perform a dual-modal fusion process on the third visible light fusion feature and the third infrared light fusion feature to obtain a second dual-modal fusion feature;

[0172] Constructing a third fusion module, the third fusion module is used to perform dual-modal fusion processing on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third dual-modal fusion feature;

[0173] A detection head layer is constructed, and the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, which includes a target bounding box, a category, and a confidence level.

[0174] In some embodiments, in the fusion stage, YOLOv8-TID introduces shallow infrared feature information, and its fusion process is a progressive and continuously integrated process. Specifically, YOLOv8-TID first uses the fusion module to preliminarily fuse the infrared image features with the visible light image features, and then fuses the visible light and infrared light features through the Concat function. This process continues until the last fusion output. The fused image finally generated not only retains rich detail texture information, but also incorporates a large amount of infrared information, providing more comprehensive content, and finally inputs the dual-modal fusion features into the monitoring layer for monitoring. A first fusion module can be constructed first, wherein the first fusion module is used to perform dual-modal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first dual-modal fusion feature. Then a second fusion module is constructed, wherein the second fusion module is used to perform dual-modal fusion processing on the third visible light fusion feature and the third infrared light fusion feature to obtain the second dual-modal fusion feature. Then a third fusion module is constructed, wherein the third fusion module is used to perform dual-modal fusion processing on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain the third dual-modal fusion feature. Finally, a detection head layer is constructed, wherein the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, which includes a target bounding box, a category, and a confidence level. Furthermore, a detection head layer can be constructed after the first fusion module and the second fusion module, respectively, to perform multi-angle predictions. It can be understood that the fusion module is embedded in the output position of the dual-path neck network layer (visible light neck network layer and infrared light neck network layer) of the YOLOv8-TID model. The main function of the fusion module is to fuse the input dual-path feature information (bimodal features) to enhance the expression performance of the fused image.

[0175] In some embodiments, performing dual-modal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain a first dual-modal fusion feature includes:

[0176] splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature;

[0177] Performing compression and activation processing on the first splicing feature to obtain a compressed and activated feature;

[0178] According to the preset point convolution frequency, the point convolution layer is used to perform point convolution on the compression and activation features to obtain the global features;

[0179] According to the preset activation function and global features, the target weight is calculated, and the target weight includes visible light weight and infrared light weight;

[0180] Multiply the visible light weight and the second visible light fusion feature to obtain a visible light multiplication feature;

[0181] Multiply the infrared light weight and the second infrared light fusion feature to obtain the infrared light multiplication feature;

[0182] Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first addition feature;

[0183] Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature;

[0184] Concatenate the first added feature and the second added feature to obtain a second concatenated feature;

[0185] Perform point convolution on the second concatenated feature to obtain the first bimodal fusion feature.

[0186] In some embodiments, the fusion module structure is as follows Figure 7 As shown, the second visible light fusion feature and the second infrared light fusion feature can be spliced ​​to obtain the first spliced ​​feature, and the first spliced ​​feature can be compressed and activated to obtain the compressed and activated feature, which is used to adjust the channel weight of the feature map. Convolution smoothing can be used to generate more accurate information. Exemplarily, the SENet attention module can be used for compression and activation processing. Then, according to the preset point convolution frequency, the point convolution layer is used to perform point convolution on the compressed and activated features to obtain the global feature. The mathematical expression of splicing, compression and activation processing, and point convolution is: Z r =PW_Conv 2 (SE(C(I vi +I ir ))), where Z r is the processed global feature, PW_Conv k (·) is the point convolution layer, k is the preset point convolution frequency, SE(·) is the compression and activation function, C is the Concat function for feature concatenation, and I vi is the second visible light fusion feature, I iris the second infrared light fusion feature. Then, according to the preset activation function and the global feature, the target weight is calculated, wherein the target weight includes the visible light weight and the infrared light weight. Exemplarily, the preset activation function may include a Sigmoid function, and the Sigmoid function may be used to generate the visible light weight and the infrared light weight in the channel dimension. Since visible light and infrared light images have complementary features, the visible light weight may be set to w, and the infrared light weight may be set to 1-w. The calculation formula of the visible light weight is: w=σ(F i (Z r )), where w is the visible light weight, σ is the Sigmoid function, and F i is the result set of batch processing. Then the visible light weight and the second visible light fusion feature are multiplied to obtain the visible light multiplication feature, and the infrared light weight and the second infrared light fusion feature are multiplied to obtain the infrared light multiplication feature. It can be understood that the w value can be dynamically adjusted according to the different weight values ​​in the fusion feature to generate the multiplication feature and obtain the local features of the image. Then the visible light multiplication feature and the second infrared light fusion feature are added to obtain the first addition feature, and the infrared light multiplication feature and the second visible light fusion feature are added to obtain the second addition feature. It can be understood that the multiplication feature is added to the fusion feature and convolution smoothing is performed so that the fusion feature has both global features and local features. The mathematical expression of multiplication and addition is: In the formula, F vi is the second additive feature, F ir is the first added feature, is the element-wise product, is the element-wise sum. Finally, the first added feature and the second added feature are concatenated to obtain the second concatenated feature, and the second concatenated feature is point convolved to obtain the first bimodal fusion feature. The mathematical expression of concatenation and point convolution is: FF r =PW_Conv(C(F vi , F ir )), where FF r is the first dual-modal fusion feature, and PW_Conv(·) is point convolution. It can be understood that the first dual-modal fusion feature is obtained by the second visible light fusion feature and the second infrared light fusion feature, the second dual-modal fusion feature is obtained by the third visible light fusion feature and the third infrared light fusion feature, and the third dual-modal fusion feature is obtained by the fourth visible light fusion feature and the fourth infrared light fusion feature, and their dual-modal fusion processing processes are similar.

[0187] In some embodiments, the first splicing feature is compressed and activated to obtain the compressed and activated feature, including:

[0188] Performing feature transformation on the first splicing feature to obtain a transformation feature;

[0189] Perform global average pooling on the transformed features to obtain compressed features;

[0190] According to the ReLU activation function, the compressed features are input into the first fully connected layer to obtain the excitation features;

[0191] According to the Sigmoid activation function, the excitation features are input into the second fully connected layer to obtain the channel weights;

[0192] Multiply the channel weight and the conversion feature to obtain the compression and activation features.

[0193] In some embodiments, the SENet attention module structure is as follows Figure 8 As shown, the first spliced ​​feature can be first transformed to obtain a transformed feature, and then the transformed feature can be globally averaged pooled to obtain a compressed feature. By way of example, the size of the input first spliced ​​feature can be set to H×W×C, which respectively represents the height H, width W, and number of channels C of the feature map. The size remains unchanged after the feature transformation, and a global average pooling operation is performed on the transformed feature. This step is the compression operation, which can aggregate the spatial dimension of the input first spliced ​​feature to generate a one-dimensional vector, namely the channel descriptor. The mathematical expression of the compression operation is: Where Zc is the weight of the cth channel, H is the height of the conversion feature, W is the width of the conversion feature, and u c(i，j) is the value of the cth channel at position (i, j) in the input conversion feature u. It can be understood that the feature map with C channels can be transformed into a 1×1×C feature map through compression operation to obtain the global receptive field. Then, according to the ReLU activation function, the compressed feature is input into the first fully connected layer to obtain the excitation feature, and according to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight. Exemplarily, after the compression operation, the obtained global information (ie, the compressed feature) is subjected to an excitation operation. The excitation operation adopts a simple gating mechanism, and the process needs to pass through two fully connected layers (ie, the first fully connected layer and the second fully connected layer) and activation functions (ie, the ReLU activation function and the Sigmoid activation function). The mathematical expression of the excitation operation is: s = σ(W2δ(W1z)), where s is the channel weight, σ is the Sigmoid activation function, W2 is the second fully connected layer, used to adjust the number of channels to the original number, δ is the ReLU activation function, W1 is the first fully connected layer, used to adjust the number of channels to the contraction rate 1 / r, and z is the compression feature. Finally, the channel weight and the conversion feature are multiplied to obtain the compression and activation features. For example, the obtained channel weight s is multiplied by the conversion feature u cMultiply the corresponding channels to obtain the weighted feature sequence (i.e., compression and activation features). This feature sequence is the output of the SENet attention mechanism, and its mathematical expression is: c =s c u c , where x c is the compression and activation feature, S c is the channel weight of the cth channel, u c is the conversion feature. Compared with the input local feature sequence (first concatenated feature), the feature sequence (compression and activation feature) output by the SENet attention mechanism contains global context information. It can be understood that the compression and activation function SENet (Squeeze-and-Excitation Networks) in the fusion module is a channel attention mechanism. This is a lightweight gated attention mechanism that can explicitly model the interdependence between convolutional feature channels. The core idea of ​​the SENet attention mechanism is to enhance the representation ability of the model through two key operations: compression (Squeeze) and then excitation (Excitation).

[0194] In some embodiments, the overall process of model training and ice coverage monitoring is as follows: Fig. 9 As shown in the figure, we can first build a wind turbine blade icing image dataset, then introduce a dual-modal fusion module, build a YOL0v8-TID wind turbine blade icing recognition model, and then input the wind turbine blade icing image dataset into the YOL0v8-TID wind turbine blade icing recognition model, and adjust the model parameters according to the training effect, and then apply the trained model to the wind turbine icing image to determine the icing threshold. When icing occurs, the model prediction result at this time is recorded as the icing threshold. Finally, the model is used to monitor the icing of the wind turbine blades and obtain the monitoring results.

[0195] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain an image of an iced fan blade, then input the image of the iced fan blade into a fan blade ice recognition model to obtain an ice occurrence threshold, and finally monitor the ice on the fan blade according to the ice occurrence threshold and the fan blade ice recognition model, so that the fan blade ice monitoring can be realized through model recognition and threshold comparison, thereby improving the accuracy. Among them, the fan blade ice recognition model is obtained by first obtaining a fan blade ice image dataset, then constructing a dual-modal image fusion model, and finally inputting the fan blade ice image dataset into the dual-modal image fusion model so that the dual-modal image fusion model is trained, thereby improving the model accuracy.

[0196] like Fig.10 As shown, the embodiment of the present invention further provides a wind turbine blade icing monitoring device based on dual-modal image fusion, comprising:

[0197] The first module 801 is used to obtain an image of a wind turbine blade covered with ice;

[0198] The second module 802 is used to input the ice-covered wind turbine blade image into the wind turbine blade ice-covered recognition model to obtain an ice-covered threshold value;

[0199] The third module 803 is used to monitor the icing of the fan blades according to the icing occurrence threshold and the fan blade icing identification model;

[0200] Among them, the wind turbine blade icing identification model is obtained through the following steps:

[0201] Acquire a wind turbine blade ice-covered image dataset, where the wind turbine blade ice-covered image dataset includes visible light images and infrared light images;

[0202] Construct a dual-modal image fusion model;

[0203] The wind blade icing image dataset is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain a wind blade icing recognition model.

[0204] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0205] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A wind turbine blade icing monitoring method based on dual-modal image fusion, characterized in that: The following steps are involved: Acquire images of ice-covered wind turbine blades; Inputting the ice-covered wind turbine blade image into a wind turbine blade ice-covering recognition model to obtain an ice-covering occurrence threshold; Monitoring wind blade icing according to the icing occurrence threshold and the wind blade icing identification model; The wind turbine blade icing identification model is obtained by the following steps: Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image; Construct a dual-modal image fusion model; The wind turbine blade icing image data set is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

2. The method according to claim 1, characterized in that The construction process of the dual-modal image fusion model includes: Constructing a visible light input layer and an infrared light input layer respectively; After the visible light input layer, a visible light backbone network layer is constructed; After the infrared light input layer, an infrared light backbone network layer is constructed; After the visible light backbone network layer, a visible light neck network layer is constructed; After the infrared light backbone network layer, an infrared light neck network layer is constructed; After the visible light neck network layer and the infrared light neck network layer, a head network layer is constructed.

3. The method according to claim 2, characterized in that The construction process of the visible light backbone network layer includes: Constructing a first visible light convolution composite layer, the first visible light convolution composite layer comprising three standard convolution modules and two C2f modules, the first visible light convolution composite layer being used to perform a first visible light feature extraction on the visible light image to obtain a first visible light convolution feature; Constructing a second visible light convolution composite layer, the second visible light convolution composite layer comprising the standard convolution module and the C2f module, the second visible light convolution composite layer being used to perform a second visible light feature extraction on the first visible light convolution feature to obtain a second visible light convolution feature; A third visible light convolution composite layer is constructed, wherein the third visible light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third visible light convolution composite layer is used to perform a third visible light feature extraction on the second visible light convolution feature to obtain a third visible light convolution feature.

4. The method according to claim 3, characterized in that The construction process of the visible light neck network layer includes: Constructing a first upsampling layer, where the first upsampling layer is used to upsample the third visible light convolution feature to obtain a first upsampling feature; Constructing a first visible light feature connection layer, where the first visible light feature connection layer is used to perform feature connection on the first up-sampled feature and the second visible light convolution feature to obtain a first visible light connection feature; Constructing a first visible light C2f layer, where the first visible light C2f layer is used to perform feature fusion on the first visible light connection feature to obtain a first visible light fusion feature; Constructing a second upsampling layer, where the second upsampling layer is used to upsample the first visible light fusion feature to obtain a second upsampling feature; Constructing a second visible light feature connection layer, where the second visible light feature connection layer is used to perform feature connection on the second up-sampled feature and the first visible light convolution feature to obtain a second visible light connection feature; Constructing a second visible light C2f layer, where the second visible light C2f layer is used to perform feature fusion on the second visible light connection feature to obtain a second visible light fusion feature; Constructing a first standard convolution layer, where the first standard convolution layer is used to perform a standard convolution on the second visible light fusion feature to obtain a first standard convolution feature; Constructing a third visible light feature connection layer, where the third visible light feature connection layer is used to perform feature connection on the first standard convolution feature and the first visible light fusion feature to obtain a third visible light connection feature; Constructing a third visible light C2f layer, wherein the third visible light C2f layer is used to perform feature fusion on the third visible light connection feature to obtain a third visible light fusion feature; Constructing a second standard convolution layer, where the second standard convolution layer is used to perform a standard convolution on the third visible light fusion feature to obtain a second standard convolution feature; Constructing a fourth visible light feature connection layer, where the fourth visible light feature connection layer is used to perform feature connection on the second standard convolution feature and the third visible light convolution feature to obtain a fourth visible light connection feature; A fourth visible light C2f layer is constructed, and the fourth visible light C2f layer is used to perform feature fusion on the fourth visible light connection feature to obtain a fourth visible light fusion feature.

5. The method according to claim 4, characterized in that The construction process of the infrared optical backbone network layer includes: Constructing a first infrared light convolution composite layer, wherein the first infrared light convolution composite layer includes three standard convolution modules and two C2f modules, and the first infrared light convolution composite layer is used to extract a first infrared light feature from the infrared light image to obtain a first infrared light convolution feature; Constructing a second infrared light convolution composite layer, wherein the second infrared light convolution composite layer includes the standard convolution module and the C2f module, and the second infrared light convolution composite layer is used to perform a second infrared light feature extraction on the first infrared light convolution feature to obtain a second infrared light convolution feature; A third infrared light convolution composite layer is constructed, wherein the third infrared light convolution composite layer includes the standard convolution module, the C2f module and the fast spatial pyramid pooling module, and the third infrared light convolution composite layer is used to perform a third infrared light feature extraction on the second infrared light convolution feature to obtain a third infrared light convolution feature.

6. The method according to claim 5, characterized in that The construction process of the infrared light neck network layer includes: Constructing a third upsampling layer, wherein the third upsampling layer is used to upsample the third infrared light convolution feature to obtain a third upsampling feature; Constructing a first infrared light feature connection layer, where the first infrared light feature connection layer is used to perform feature connection on the third up-sampling feature and the second infrared light convolution feature to obtain a first infrared light connection feature; Constructing a first infrared light C2f layer, wherein the first infrared light C2f layer is used to perform feature fusion on the first infrared light connection feature to obtain a first infrared light fusion feature; Constructing a fourth upsampling layer, where the fourth upsampling layer is used to upsample the first infrared light fusion feature to obtain a fourth upsampling feature; Constructing a second infrared light feature connection layer, where the second infrared light feature connection layer is used to perform feature connection on the fourth up-sampling feature and the first infrared light convolution feature to obtain a second infrared light connection feature; Constructing a second infrared light C2f layer, where the second infrared light C2f layer is used to perform feature fusion on the second infrared light connection feature to obtain a second infrared light fusion feature; Constructing a third standard convolution layer, wherein the third standard convolution layer is used to perform a standard convolution on the first bimodal fusion feature to obtain a third standard convolution feature; Constructing a third infrared light feature connection layer, wherein the third infrared light feature connection layer is used to perform feature connection on the third standard convolution feature and the first infrared light fusion feature to obtain a third infrared light connection feature; Constructing a third infrared light C2f layer, wherein the third infrared light C2f layer is used to perform feature fusion on the third infrared light connection feature to obtain a third infrared light fusion feature; Constructing a fourth standard convolution layer, wherein the fourth standard convolution layer is used to perform a standard convolution on the second bimodal fusion feature to obtain a fourth standard convolution feature; Constructing a fourth infrared light feature connection layer, wherein the fourth infrared light feature connection layer is used to perform feature connection on the fourth standard convolution feature and the third infrared light convolution feature to obtain a fourth infrared light connection feature; A fourth infrared light C2f layer is constructed, and the fourth infrared light C2f layer is used to perform feature fusion on the fourth infrared light connection feature to obtain a fourth infrared light fusion feature.

7. The method according to claim 6, characterized in that The construction process of the head network layer includes: Constructing a first fusion module, wherein the first fusion module is used to perform a dual-modal fusion process on the second visible light fusion feature and the second infrared light fusion feature to obtain the first dual-modal fusion feature; Constructing a second fusion module, wherein the second fusion module is used to perform a dual-modal fusion process on the third visible light fusion feature and the third infrared light fusion feature to obtain the second dual-modal fusion feature; Constructing a third fusion module, wherein the third fusion module is used to perform a dual-modal fusion process on the fourth visible light fusion feature and the fourth infrared light fusion feature to obtain a third dual-modal fusion feature; A detection head layer is constructed, where the detection head layer is used to predict the third bimodal fusion feature to obtain a prediction result, where the prediction result includes a target bounding box, a category, and a confidence level.

8. The method according to claim 7, characterized in that The performing dual-modal fusion processing on the second visible light fusion feature and the second infrared light fusion feature to obtain the first dual-modal fusion feature includes: splicing the second visible light fusion feature and the second infrared light fusion feature to obtain a first splicing feature; Performing compression and activation processing on the first splicing feature to obtain a compression and activation feature; According to a preset point convolution frequency, using a point convolution layer to perform point convolution on the compression and activation features to obtain a global feature; Calculate a target weight according to a preset activation function and the global feature, wherein the target weight includes a visible light weight and an infrared light weight; Multiplying the visible light weight and the second visible light fusion feature to obtain a visible light multiplication feature; Multiplying the infrared light weight and the second infrared light fusion feature to obtain an infrared light multiplication feature; Adding the visible light multiplication feature and the second infrared light fusion feature to obtain a first addition feature; Adding the infrared light multiplication feature and the second visible light fusion feature to obtain a second added feature; Concatenate the first added feature and the second added feature to obtain a second concatenated feature; Perform point convolution on the second splicing feature to obtain the first bimodal fusion feature.

9. The method according to claim 8, characterized in that The compressing and activating the first splicing feature to obtain the compressed and activated feature includes: Performing feature conversion on the first splicing feature to obtain a conversion feature; Performing global average pooling on the conversion features to obtain compressed features; According to the ReLU activation function, the compression feature is input into the first fully connected layer to obtain an excitation feature; According to the Sigmoid activation function, the excitation feature is input into the second fully connected layer to obtain the channel weight; The channel weight and the conversion feature are multiplied to obtain the compression and activation feature.

10. A wind turbine blade icing monitoring device based on dual-modal image fusion, characterized in that: include: The first module is used to obtain images of ice-covered wind turbine blades; The second module is used to input the ice-covered wind turbine blade image into a wind turbine blade ice-covering recognition model to obtain an ice-covering threshold; A third module is used to monitor the icing of the fan blades according to the icing occurrence threshold and the fan blade icing identification model; The wind turbine blade icing identification model is obtained by the following steps: Acquire a wind turbine blade ice-covered image dataset, wherein the wind turbine blade ice-covered image dataset includes a visible light image and an infrared light image; Construct a dual-modal image fusion model; The wind turbine blade icing image data set is input into the dual-modal image fusion model so that the dual-modal image fusion model is trained to obtain the wind turbine blade icing recognition model.

Images