Wind turbine nacelle fire early warning method based on multimodal large model fusion of multispectral images

Abstract

The present application discloses a wind turbine cabin fire warning method that fuses multi-spectral images using a multi-modal large model, and relates to the field of fire warning. The method uses visible light images and infrared images for image fusion to obtain sample fusion images of the wind turbine cabin in different working states. The sample fusion images fuse image features under different spectral modes to complement each other's advantages, and can better characterize the image features of the wind turbine cabin in the current working state. The sample fusion images are then annotated and described in text to construct a fire common sense graphic and text dataset, and the parameters of the pre-trained VILA multi-modal large model are adjusted so that the constructed wind turbine cabin fire warning model can understand the basic common sense of wind turbine cabin fire warning, so that the search can be dynamically adjusted according to the target, which is beneficial to improving the recognition and positioning accuracy of abnormal targets in the wind turbine cabin in the fire warning state, and can improve the accuracy of fire warning.

Description

Technical Field

[0001] The present application relates to the field of fire warning, and in particular to a wind turbine cabin fire warning method that fuses a multi-modal large model with multi-spectral images. Background Art

[0002] Wind power generation is a highly promising and important branch of renewable energy generation, receiving significant national attention and attention. Wind turbine systems are complex, and the equipment within the wind turbine nacelle is subject to prolonged high-speed rotation and high-vibration conditions. This can easily lead to problems such as lubricant leaks, gearbox overheating, and reduced cable insulation due to overcurrent. These issues can easily generate smoke and even flames, causing equipment fires and threatening the safe and stable operation of wind farms. Therefore, monitoring these issues within the wind turbine nacelle can help identify fire characteristics early in the process, enabling timely warnings. This can assist power plant operators and maintenance personnel in ensuring equipment fire safety and minimize the damage caused by fires to the wind turbine system.

[0003] With the convergence and development of computer vision and natural language processing technologies, multimodal large models capable of natural language understanding and image recognition have begun to emerge and have achieved initial application in wind farm fire monitoring systems. However, due to the complex environment and high background noise in wind turbine nacelles, existing methods using multimodal large models to infer and identify fire characteristics are prone to hallucinations, resulting in less than ideal detection and warning accuracy. Summary of the Invention

[0004] In response to the above-mentioned problems and technical needs, this application proposes a wind turbine cabin fire early warning method that integrates a multimodal large model with multispectral images. The technical solution of this application is as follows:

[0005] A wind turbine cabin fire early warning method based on a multi-modal large model fused with multispectral images, the wind turbine cabin fire early warning method comprising:

[0006] A dual-spectral camera is used to obtain sample images of the wind turbine nacelle in different working states. The working states of the wind turbine nacelle include normal working state and different fire warning states. The sample images of the wind turbine nacelle in each working state include sample visible light images and sample infrared images.

[0007] An image fusion module is used to fuse sample visible light images and sample infrared images in a set of sample images of the wind turbine cabin in each working state to obtain corresponding sample fused images. The image fusion module performs multi-scale fuzzy transformation on the sample visible light images using a Gaussian kernel function to obtain a visible light base image and a visible light detail image. The image fusion module also performs multi-scale fuzzy transformation on the sample infrared images using a Gaussian kernel function to obtain an infrared base image and an infrared detail image. The visible light base image and the infrared base image are weighted averaged to obtain a fused base image. The visible light detail image and the infrared detail image are image fused to obtain a fused detail image. The fused base image and the fused detail image are reconstructed to obtain a sample fused image of the wind turbine cabin in the current working state.

[0008] Image annotation is performed on abnormal targets used to identify fire warning states in sample fusion images of the wind turbine nacelle in each working state, and fire level description text used to describe the working state of the wind turbine nacelle is added to the sample fusion images to construct a fire common sense image and text dataset;

[0009] The fire common sense image and text dataset was used to train the VILA multimodal large model and adjust the model parameters. The trained base model was then used to build a wind turbine cabin fire warning model.

[0010] A dual-spectral camera is used to capture visible light and infrared images of the wind turbine nacelle to be monitored at the same moment during operation. The images are then fused using an image fusion module to obtain a fused image to be monitored. The fused image to be monitored and a character string indicating the identification requirement are input into a wind turbine nacelle fire warning model to obtain the identification and positioning results of abnormal targets in the wind turbine nacelle to be monitored.

[0011] Based on the recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored, a fire warning result of the wind turbine cabin to be monitored is obtained.

[0012] A further technical solution is that the visible light detail image and the infrared detail image are fused to obtain a fused detail image, which includes:

[0013] The pre-trained residual network model is used to obtain the four-layer depth features of the visible light detail image, and the pre-trained residual network model is used to obtain the four-layer depth features of the infrared detail image. A multi-layer fusion strategy is used to fuse the four-layer depth features of the visible light detail image and the four-layer depth features of the infrared detail image to obtain a fused detail image.

[0014] A further technical solution is that the fire warning state of the wind turbine nacelle includes at least one of an equipment overheating state, an oil leakage state, and a fire state;

[0015] The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine nacelle in each working state is the overheating target. The overheating target is the equipment component in the wind turbine nacelle whose infrared hot spot temperature reaches the temperature threshold;

[0016] The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine nacelle in each working state is the oil leakage target, which is the equipment component in the wind turbine nacelle where the oil leakage occurs;

[0017] The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

[0018] A further technical solution is to add a fire level description text for describing the working status of the wind turbine cabin to the sample fusion image, including:

[0019] Generate corresponding fire level description text according to predetermined text rules based on the status parameters of each device component in the wind turbine nacelle, the status parameters of the smoke target in the wind turbine nacelle, and the fire level displayed by the sample fusion image;

[0020] The status parameters of each equipment component include the name, installation location, and operating status of the equipment component. The operating status of the equipment component is used to indicate whether the equipment component is in a normal operating state or an abnormal operating state. The abnormal operating state of each equipment component includes at least one of the infrared hotspot temperature reaching a temperature threshold and the occurrence of oil leakage. The status parameters of the equipment component whose infrared hotspot temperature reaches the temperature threshold also include the infrared hotspot temperature. The status parameters of the equipment component whose oil leakage occurs also include the location of the oil leakage.

[0021] The state parameters of the smoke target in the wind turbine nacelle include the position and size of the smoke target.

[0022] The further technical solution is to use the fire common sense image and text dataset to train the VILA multimodal large model and adjust the model parameters, including:

[0023] The low-rank adaptive decomposition method in the deepspeed zero2 large model fine-tuning acceleration framework is used to train the VILA multimodal large model and adjust the model parameters. The trained LORA model parameters are merged with the initial parameters of the VILA multimodal large model as the model parameters of the trained wind turbine cabin fire warning model.

[0024] Its further technical solution is that when using the low-rank adaptive decomposition method to train the VILA multimodal large model and adjust the model parameters, the rank of lora is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is 2e-5, and the basic learning rate is set to 2e-4.

[0025] Its further technical solution is that the wind turbine cabin fire warning model constructed includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a trained base model and a target position decoder;

[0026] The fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module; the output of the image feature preprocessing module and the output of the text feature preprocessing module are respectively connected to the feature fusion module; the output of the feature fusion module is connected to the base model; the output of the base model is connected to the target position decoder via a linear layer; the output of the image feature preprocessing module is also connected to the target position decoder; wherein:

[0027] The image feature preprocessing module consists of a cascade of visual encoders, linear layers, and resampling layers. The visual encoder uses the ViT-L / 14 visual encoder used in the DINOv2 open-world object detection model and the convolutional neural network ConvNeXt with an attention mechanism to capture spatial context and local image information, respectively.

[0028] The text feature preprocessing module includes a cascaded word segmentation and encoding module and a vectorized representation module. The word segmentation and encoding module uses the language tagger used in natural language processing applications, and the vectorized representation module uses word2vec technology.

[0029] The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features;

[0030] The base model outputs a positioning feature identifier that combines image context semantics and world knowledge based on multimodal fusion features; the positioning feature identifier is formed into a target position feature vector through a projection matrix through a linear layer and input into the target position decoder;

[0031] The target position decoder adopts the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. The target position decoder outputs the abnormal target positioning frame based on the input target position feature vector and image features. The abnormal target positioning frame is used to indicate the position, type, size and confidence of the abnormal target in the monitored fusion image. The type of abnormal target is overheated target, oil leakage target or smoke target.

[0032] Its further technical solution is that the constructed wind turbine cabin fire warning model also includes a visual clue decoder, the output of the base model is further connected to the visual clue decoder through a linear layer, the positioning feature identifier output by the base model is formed into a visual clue feature vector through a projection matrix through a linear layer and input into the visual clue decoder, the output of the image feature preprocessing module and the output of the feature fusion module are further connected to the visual clue decoder respectively, the visual clue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture, the visual clue decoder outputs an abnormal target search prompt heat map based on the input visual clue feature vector, image features and multimodal fusion features, and the identification and positioning results of the abnormal target in the monitored wind turbine cabin also include:

[0033] Obtain the abnormal target search prompt heat map output by the wind turbine cabin fire warning model. The abnormal target search prompt heat map indicates the confidence level of the presence of abnormal targets at different locations in the monitored fusion image.

[0034] When the confidence level of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches the confidence threshold, the recognition and positioning result including the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output;

[0035] When the confidence of the abnormal target positioning frame does not reach the confidence threshold, the fused image to be monitored is divided into multiple sub-regions, and the confidence of the abnormal target contained in each sub-region is determined according to the abnormal target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of the abnormal target contained. For each traversed sub-region: the image of the fused image to be monitored in the sub-region is input into the wind turbine cabin fire warning model to obtain the abnormal target positioning frame for the sub-region. When the confidence of the abnormal target positioning frame reaches the confidence threshold, the identification and positioning result of the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output. Otherwise, the next sub-region is traversed until all sub-regions are traversed and it is determined that the fused image to be monitored does not contain abnormal targets.

[0036] The beneficial technical effects of this application are:

[0037] The present application discloses a wind turbine cabin fire warning method that fuses multi-spectral images using a multi-modal large model. The method uses visible light images and infrared images for image fusion to obtain sample fusion images of the wind turbine cabin in different working states. The sample fusion images fuse image features under different spectral modes, which can achieve complementary advantages and thus better characterize the image features of the wind turbine cabin in the current working state. The sample fusion images under different working states are then used for image annotation and text description to construct a fire common sense graphic and text dataset. The fire common sense graphic and text dataset is then used to adjust the parameters of the pre-trained VILA multi-modal large model, so that the constructed wind turbine cabin fire warning model can understand the basic common sense of wind turbine cabin fire warning, and thus can dynamically adjust the search according to the target, which is conducive to improving the accuracy of identifying and locating abnormal targets, and can improve the accuracy of fire warning.

[0038] The wind turbine cabin fire warning model constructed by this method can also output an abnormal target search prompt heat map. The abnormal target search prompt heat map is used to introduce an active search mechanism, visual working memory, and a dynamic adjustment strategy. By simulating the human recognition mechanism of abnormal targets, it can fully utilize the world knowledge and context understanding ability of the large model, further improve the accuracy of identifying and locating abnormal targets, and thus further improve the accuracy of wind turbine cabin fire warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a method flow chart of a wind turbine cabin fire warning method according to an embodiment of the present application.

[0040] Figure 2 This is a flow chart of a method for using an image fusion module to fuse a sample visible light image and a sample infrared image in a group of sample images to obtain a corresponding sample fused image in one embodiment of the present application.

[0041] Figure 3 This is a model structure diagram of a wind turbine cabin fire warning model constructed in one embodiment of the present application.

[0042] Figure 4 yes Figure 3 The block diagram of the target position decoder in .

[0043] Figure 5 yes Figure 3 The structure diagram of the visual cue decoder in

[15] . DETAILED DESCRIPTION

[0044] The specific implementation of this application will be further described below with reference to the accompanying drawings.

[0045] This application discloses a wind turbine cabin fire warning method that integrates multi-modal large-scale models with multi-spectral images. Figure 1 The method includes the following steps:

[0046] Step 1: Use a dual-spectral camera to obtain sample images of the wind turbine nacelle in different working states. A set of sample images of the wind turbine nacelle in each working state includes a sample visible light image and a sample infrared image at the same time.

[0047] The operating status of a wind turbine nacelle includes normal operating status and different fire warning statuses. The fire warning status is an abnormal state in which the wind turbine nacelle is in a fire risk state. Fire warning status includes at least one of the following: equipment overheating, oil leakage, and fire. Equipment overheating occurs when a component in the wind turbine nacelle reaches the infrared hotspot temperature threshold and is operating abnormally. Oil leakage occurs when a component in the wind turbine nacelle leaks oil and is operating abnormally. Fire status occurs when a fire has occurred in the wind turbine nacelle and smoke is generated.

[0048] When acquiring sample images of a wind turbine nacelle, one approach is to obtain sample images of the nacelle under different operating conditions during its historical operation, such as directly acquiring images captured by surveillance cameras during the nacelle's historical operation. Another approach is to build a wind turbine nacelle simulation test platform, then simulate different operating conditions of the nacelle on the platform and acquire corresponding sample images. For example, by varying the power of a gearbox in the nacelle to raise the infrared hotspot temperature to a threshold to simulate an overheated device. Another example is to apply lubricant to the gearbox to simulate an oil leak. Another example is to ignite a smoke cake inside the nacelle to generate smoke to simulate a fire.

[0049] Regardless of the method used to acquire sample images of the wind turbine nacelle, the actual image directly captured by the dual-spectral camera is a video stream. First, video image frames are extracted from the video stream at regular intervals. Video image frames in different operating states are then preprocessed, such as by denoising, dehazing, and deraining, to serve as sample images. Furthermore, the sample visible light and sample infrared images within the same set of sample images often require complete image registration, which is not detailed in this embodiment.

[0050] Step 2: Using an image fusion module, image fusion is performed on a set of sample images of the wind turbine nacelle in each working state, including a sample visible light image and a sample infrared image, to obtain a corresponding sample fused image.

[0051] Please refer to Figure 2The image fusion module shown is a schematic diagram of image fusion of a sample visible light image and a sample infrared image. The image fusion module first performs a multi-scale fuzzy transformation on the sample visible light image in the group of sample images using a Gaussian kernel function to obtain a visible light base image and a visible light detail image, and also performs a multi-scale fuzzy transformation on the sample infrared image in the group of sample images using a Gaussian kernel function to obtain an infrared base image and an infrared detail image.

[0052] The visible light basic image and the infrared basic image are fused at the pixel level by weighted average to eliminate light source interference and obtain a fused basic image.

[0053] A fused detail image is generated by performing pixel-level image fusion on the visible light detail image and the infrared detail image. Specifically, a pre-trained residual network model is used to obtain four layers of deep features for the visible light detail image, and a pre-trained residual network model is used to obtain four layers of deep features for the infrared detail image. A multi-layer fusion strategy is then used to fuse these four layers of deep features of the visible light detail image and the infrared detail image to generate the fused detail image.

[0054] Finally, the fused basic image and the fused detail image are reconstructed to obtain a sample fusion image of the wind turbine cabin in the current working state. The obtained sample fusion image integrates the modal characteristics under different spectral images and can provide more comprehensive and accurate image features.

[0055] Step 3: Annotate the abnormal targets used to identify the fire warning status in the sample fusion images of the wind turbine cabin in each working state, and add fire level description text used to describe the working status of the wind turbine cabin to the sample fusion images to construct a fire common sense image and text dataset.

[0056] The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine cabin in each working state is the overheating target. The overheating target is the equipment component with overheating problem when the infrared hot spot temperature in the wind turbine cabin reaches the temperature threshold.

[0057] The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine nacelle in each working state is the oil leakage target, which is the equipment component with oil leakage in the wind turbine nacelle.

[0058] The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

[0059] It should be noted that the sample fused images of the wind turbine nacelle in each operating state may contain one or more types of abnormal targets, corresponding to one or more fire warning states in the wind turbine nacelle's current operating state. For example, a wind turbine nacelle may be experiencing both equipment overheating and fire, resulting in both overheating targets and smoke targets being present in the sample fused images. In practical applications, it is possible to consider only some key equipment components in the wind turbine nacelle, or all equipment components in the wind turbine nacelle.

[0060] When performing image annotation, use image annotation software such as labelimg to annotate the abnormal targets in the sample fusion image, including using annotation boxes to annotate the position, type and size of the abnormal targets in the sample fusion image. The size of the abnormal target is defined by the proportion of the abnormal target occupying the sample fusion image, such as small size, medium size and large size.

[0061] In addition to image annotation, fire classification text is added to each sample fused image. This text describes the operating status of the wind turbine nacelle in that sample fused image. This text is generated according to predefined text rules based on the status parameters of the wind turbine nacelle components, the status parameters of the smoke target within the nacelle, and the fire classification level. The classification of fire classifications for each operating state of the wind turbine nacelle can be customized.

[0062] The status parameters of each equipment component include the component's name, installation location, and operating status. The operating status of an equipment component indicates whether the component is operating normally or abnormally. Abnormal operating status for each equipment component includes at least one of the infrared hotspot temperature reaching a temperature threshold and oil leakage. For equipment components where the infrared hotspot temperature reaches the temperature threshold, the status parameters of the equipment component also include the infrared hotspot temperature. For equipment components where oil leakage occurs, the status parameters of the equipment component also include the location of the oil leakage.

[0063] The state parameters of the smoke target in the wind turbine nacelle include the position and size of the smoke target. The state parameters of the smoke target may further include at least one of the color characteristics, optical characteristics, texture characteristics and physical and chemical characteristics of the smoke target.

[0064] Taking the gearbox, a key component in a wind turbine nacelle, as an example, the following examples illustrate how to add fire severity description text to sample fused images in different working states. The same principle applies when the wind turbine nacelle also includes other components:

[0065] In one example, the fire level description text added to the sample fused image when the wind turbine cabin is in normal working condition is {gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox is in normal working condition, there is no smoke target, and the fire level is 0}.

[0066] In another example, the fire level description text added to the sample fused image when the wind turbine cabin is in an equipment overheating state is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox has an overheating problem, the infrared hotspot temperature of the gearbox is 120°C, there is no smoke target, and the fire level is 1}.

[0067] In another example, the fire level description text added to the sample fused image of the wind turbine nacelle in the oil leakage state is {gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox has an oil leak, the gearbox oil leak position is [0.66, 0.74, 0.89, 0.82], there is no smoke target, and the fire level is 2}

[0068] In another example, the fire level description text added to the sample fused image when the wind turbine cabin is on fire is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox is in normal operation, the position of the smoke target is [0.23, 0.46, 0.57, 0.69], the smoke target is medium in size, and the fire level is 4}.

[0069] Step 4: Use the fire knowledge image and text dataset to train the VILA multimodal large model and adjust the model parameters to obtain a base model through training. The trained base model is then used to build a wind turbine cabin fire warning model.

[0070] The current conventional practice is to directly download the pre-trained VILA multimodal large model for inference. However, this application does not directly use the open source pre-trained VILA multimodal large model. Instead, it uses the fire common sense image and text dataset of the wind turbine cabin to adjust the model parameters of the VILA multimodal large model, and finally obtains a base model that has the ability to understand the basic common sense of wind turbine cabin fire warning. Moreover, the fire common sense image and text dataset used by this application to adjust the model parameters is a dual-spectrum image feature that integrates visible light images and infrared images. It can fully utilize the complementary advantages of different spectral images so that the base model can dynamically adjust the search according to the target, thereby helping to improve the recognition effect.

[0071] The VILA multimodal large model can understand and process text and images, and realize the image-to-text function of text analysis. The VILA multimodal large model uses alternating text data of images and text for pre-training. This application further uses the fire common sense image and text dataset obtained by dual-spectral image fusion to fine-tune the model parameters. This staged training method enables the model to better learn the joint representation of vision and language, effectively retain the model's world knowledge ability, and alleviate the model's forgetting of pure text capabilities.

[0072] In one embodiment, the low-rank adaptive decomposition method in the DeepSpeed ​​Zero2 large model fine-tuning acceleration framework is used to train and adjust the model parameters of the VILA multimodal large model. The trained LoRa model parameters are merged with the initial parameters of the VILA multimodal large model to serve as the model parameters of the trained wind turbine cabin fire warning model. Specifically, when using the low-rank adaptive decomposition method to train and adjust the model parameters of the VILA multimodal large model, the LoRa rank is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is set to 2e-5, and the base learning rate is set to 2e-4.

[0073] The VILA large multimodal model differs from multimodal models based on internal fusion using cross-attention or custom cross-attention layers in that it fuses features from the image and text modalities at an early stage. This fusion approach effectively captures the multimodal representations of images and text by mapping different modalities into a common semantic space, demonstrating unique advantages in multimodal representation learning, cross-modal generation, and zero-shot downstream transfer tasks.

[0074] This application uses the trained base model to build a wind turbine cabin fire warning model. Figure 3 As shown, the system includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a trained base model, and a target position decoder. The outputs of the image feature preprocessing module and the text feature preprocessing module are each connected to the feature fusion module. The output of the feature fusion module is connected to the base model. The output of the base model is connected to the target position decoder via a linear layer. The output of the image feature preprocessing module is also connected to the target position decoder.

[0075] The image feature preprocessing module consists of a cascade of visual encoders, linear layers, and resampling layers. The visual encoder uses the ViT-L / 14 visual encoder used in the DINOv2 open-world object detection model and the ConvNeXt convolutional neural network with an attention mechanism to capture spatial context and local image information, respectively, thereby extracting visual features that may contain abnormal objects. The linear layer performs a linear transformation on the high-dimensional visual features output by the visual encoder, converting them into features suitable for language model processing. The resampling layer uses a sampler to downsample high-dimensional visual features to a low-dimensional space through a cross-attention mechanism. This not only reduces the number of features but also emphasizes important visual features while ignoring less important ones. The combination of linear and resampling layers enables flexible processing of visual features in different scenarios, balancing computational efficiency and model performance, and more efficiently handling complex visual analysis tasks.

[0076] The text feature preprocessing module consists of a cascaded word segmentation and encoding module and a vectorized representation module. The word segmentation and encoding module uses a language tagger from natural language processing applications to treat each byte as a separate vocabulary unit. Based on statistical analysis, it then identifies the most frequently occurring byte pairs in the text and merges them into a new vocabulary unit. This process is repeated until the predetermined vocabulary size is reached or no more byte pairs can be merged. The vectorized representation module uses word2vec technology to map each subword into a one-dimensional vector embedding, converting it into a feature vector in the embedding space of the larger language model. These feature vectors are then injected into the base model as text information.

[0077] The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features.

[0078] The base model is based on the VILA-2.7b open-source multimodal large model pre-trained on multiple benchmark datasets. Based on multimodal feature fusion, the base model outputs location feature identifiers that combine image context and world knowledge. These location feature identifiers are fed through a linear layer and a projection matrix to form a target location feature vector, which is then input into the target location decoder. The location feature identifiers are then fed through another linear layer and a projection matrix to form a visual cue feature vector, which is then input into the visual cue decoder.

[0079] The target position decoder uses the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. Figure 4 The target position decoder shown in the structural diagram includes an organically combined multi-layer perceptron layer and a cosine similarity calculation module.

[0080] Furthermore, the wind turbine cabin fire warning model constructed also includes a visual cue decoder, the output of the base model is also connected to the visual cue decoder through a linear layer, and the output of the image feature preprocessing module and the output of the feature fusion module are also connected to the visual cue decoder respectively.

[0081] The visual cue decoder uses the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual cue decoder outputs an abnormal target search hint heat map based on the input visual cue feature vector, image features and multimodal fusion features. Please refer to Figure 5 The structure of the visual cue decoder is shown in Figure 2. The visual cue decoder is a lightweight network component. Each decoder layer performs the following steps: using the multimodal fusion feature and the visual cue feature vector as localization features, the self-attention layer performs self-attention on the localization features. Next, the label-image cross-attention layer performs cross-attention on the query localization features and image features. The multi-layer perceptron layer updates each localization feature point by point. Finally, the image-label cross-attention layer performs cross-attention from image features to localization features. To ensure that the visual cue decoder has access to key geometric information, the position encoding is added to the image embedding vector when participating in the attention layer operation, and the entire original localization feature containing position information is also re-appended to the image embedding vector. Through this ingenious design, the visual cue decoder prediction head can dynamically and real-timely output segmentation masks. It is also ambiguous-aware, capable of handling ambiguous cues and generating multiple possible valid masks. It also outputs an anomaly search hint heatmap, which indicates the confidence level of the presence of anomalies at different locations in the monitored fused image.

[0082] Step 5: Use a dual-spectrum camera to collect visible light images and infrared images of the wind turbine cabin to be monitored at the same time during operation, and use an image fusion module to fuse the images to obtain a fused image to be monitored. Input the fused image to be monitored and the character string used to indicate the recognition requirement into the wind turbine cabin fire warning model to obtain the recognition and positioning results of the abnormal target in the wind turbine cabin to be monitored.

[0083] This step is similar to the above-mentioned method of acquiring sample images when collecting visible light images and infrared images of the wind turbine cabin to be monitored at the same time, and also includes acquiring video streams, intercepting image frames, and performing various image preprocessing and image registration operations, which will not be repeated here.

[0084] The method of using the image fusion module to fuse the visible light image and the infrared image at the same time to obtain the monitored fused image can refer to the process of obtaining the sample fused image in step 2 above, which will not be repeated here.

[0085] The character string used to indicate the recognition requirement can be customized, for example, the character string can be "Please locate the smoke target in the image", or other similar character strings that can indicate the recognition requirement can be used.

[0086] When the fused image to be monitored and the character string indicating the recognition requirement are input into the wind turbine cabin fire warning model, the fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module. The wind turbine cabin fire warning model, built based on a multimodal large model, understands and processes the input fused image to be monitored and the character string. The target position decoder outputs an abnormal target positioning frame based on the target position feature vector and image features. The abnormal target positioning frame is used to indicate the location, type, size, and confidence level of abnormal targets in the fused image to be monitored. The abnormal target type can be an overheated target, an oil leak target, or a smoke target.

[0087] One approach is to obtain the identification and location results of abnormal targets in the wind turbine nacelle to be monitored based on the abnormal target location frame of the wind turbine nacelle fire warning model. For the abnormal target location frame output by the wind turbine nacelle fire warning model, if its confidence reaches the confidence threshold, it is considered that the abnormal target location frame contains an abnormal target used to indicate the fire warning status. Otherwise, it is considered that the abnormal target location frame does not contain an abnormal target. Finally, the identification and location results are output, including information such as the location, type, and size of all abnormal targets in the fused image to be monitored.

[0088] However, this approach is not ideal when identifying small-sized targets. For example, there will be a large number of missed detections and false detections when identifying small-sized smoke targets and small-sized oil leakage targets. Therefore, in another embodiment, the constructed wind turbine cabin fire warning model also includes a visual cue decoder. In addition to outputting the abnormal target positioning frame through the target position decoder, the positioning feature identifier output by the base model is input into the visual cue decoder through the linear layer through the projection matrix to form a visual cue feature vector. The visual cue decoder will output an abnormal target search prompt heat map based on the input visual cue feature vector, image features and multimodal fusion features. The abnormal target search prompt heat map is used to indicate the confidence level of the existence of abnormal targets at different positions in the fused image to be monitored. After obtaining the abnormal target positioning frame and the abnormal target search prompt heat map output by the wind turbine cabin fire warning model, the abnormal target positioning frame and the abnormal target search prompt heat map are comprehensively considered to obtain the recognition and positioning results, including:

[0089] When the confidence of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches the confidence threshold, the output is an identification and positioning result including the position, type and size of the abnormal target indicated by the abnormal target positioning frame.

[0090] When the confidence level of the abnormal target localization box does not reach the confidence threshold, the system does not directly determine that the abnormal target does not exist. Instead, the fused image to be monitored is divided into multiple subregions. The confidence level of each subregion containing an abnormal target is determined based on the abnormal target search hint heat map. The subregions are then traversed sequentially in descending order of confidence level. For each traversed subregion, the image of the fused image to be monitored within the subregion is input into the wind turbine nacelle fire warning model to obtain an abnormal target localization box for the subregion. When the confidence level of the abnormal target localization box reaches the confidence threshold, the system outputs the identification and localization result, including the position, type, and size of the abnormal target indicated by the abnormal target localization box. Otherwise, the system continues to traverse the next subregion until all subregions are traversed and the system determines that the fused image to be monitored does not contain an abnormal target. This approach introduces an active search mechanism, visual working memory, and a dynamic adjustment strategy. By simulating the human recognition mechanism of abnormal targets, it can fully utilize the world knowledge and context understanding ability of the multimodal large model, improving the accuracy of visual localization and multimodal collaborative reasoning capabilities.

[0091] Step 6: Obtain a fire warning result for the wind turbine nacelle to be monitored based on the recognition and positioning result of the abnormal target in the wind turbine nacelle to be monitored.

[0092] The identification and positioning results obtained through the above recognition and detection can indicate the location, type, and size of the abnormal target. Based on the content of the identification and positioning results, the corresponding fire level is determined and the corresponding fire warning result is output. The fire level and fire warning results corresponding to different identification and positioning results can be customized.

[0093] For example, focusing solely on the gearbox in a wind turbine nacelle, if the recognition and positioning results indicate that there are no abnormal targets in the monitored fused image and the nacelle is operating normally, no prompt will be given or a prompt will be given that the wind turbine nacelle is operating normally. If the recognition and positioning results indicate the presence of an overheated target, a text message will pop up on the video: "Gearbox position is [0.66, 0.74, 0.73, 0.76], gearbox overheating, infrared hotspot temperature of the gearbox is 120°C, there is no smoke target, and fire level is 1." The image and text will then be sent to the maintenance personnel's WeChat account.

[0094] This application also provides an example to verify the effect, obtaining visible light images and infrared images of the wind turbine cabin in different working conditions at the same time, and performing image fusion to obtain the fused image to be monitored, thereby constructing a wind turbine cabin test set.

[0095] Directly using the open-source pre-trained VILA multimodal large model for testing on visible light images from a wind turbine nacelle test set, the prediction accuracy was 79.23%. Directly using the open-source pre-trained VILA multimodal large model for testing on infrared images from a wind turbine nacelle test set, the prediction accuracy was 80.15%. This indicates that due to the limitations of single-spectral image representation features and the lack of dynamic adjustment of the pre-trained VILA multimodal large model to the search target, the multimodal model suffers from hallucination defects similar to those of large language models, resulting in a maximum prediction accuracy of only 80.15%.

[0096] Then, using the method of this application, the model parameters of the VILA multimodal large model were first adjusted using a fire common sense image and text dataset fused with bispectral images. Then, predictions were made using the fused images to be monitored in the wind turbine cabin test set. At this point, the prediction accuracy was increased to 86.72%. Furthermore, by combining the abnormal target search prompt heat map output by the wind turbine cabin fire warning model for active search, the prediction accuracy was further increased to 95.37%. This effectively solves the problem of false detection caused by the multimodal large model's lack of historical memory, effectively filtering out suspected abnormal targets, and further improving the accuracy.

[0097] The above description is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or imagined by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the scope of protection of the present application.

Claims

1. A wind turbine cabin fire warning method based on multi-modal large model fusion multi-spectral image, characterized by: The wind turbine cabin fire early warning method comprises: A dual-spectral camera is used to obtain sample images of the wind turbine nacelle in different working states. The working states of the wind turbine nacelle include normal working state and different fire warning states. The sample images of the wind turbine nacelle in each working state include sample visible light images and sample infrared images. An image fusion module is used to fuse sample visible light images and sample infrared images in a group of sample images of the wind turbine nacelle in each working state to obtain a corresponding sample fused image. The image fusion module performs multi-scale fuzzy transformation on the sample visible light images using a Gaussian kernel function to obtain a visible light base image and a visible light detail image, and performs multi-scale fuzzy transformation on the sample infrared images using a Gaussian kernel function to obtain an infrared base image and an infrared detail image; weighted averaging is performed on the visible light base image and the infrared base image to obtain a fused base image, and image fusion is performed on the visible light detail image and the infrared detail image to obtain a fused detail image. The fused base image and the fused detail image are reconstructed to obtain a sample fused image of the wind turbine nacelle in the current working state; Image annotation is performed on abnormal targets used to identify fire warning states in sample fusion images of the wind turbine nacelle in each working state, and fire level description text used to describe the working state of the wind turbine nacelle is added to the sample fusion images to construct a fire common sense image and text dataset; The fire common sense graphic and text dataset is used to train the VILA multimodal large model and adjust the model parameters to obtain a base model through training, and a wind turbine cabin fire warning model is constructed based on the trained base model; The constructed wind turbine cabin fire warning model includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, the trained base model and a target position decoder; The fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module; the output of the image feature preprocessing module and the output of the text feature preprocessing module are respectively connected to the feature fusion module; the output of the feature fusion module is connected to the base model; the output of the base model is connected to the target position decoder via a linear layer; the output of the image feature preprocessing module is also connected to the target position decoder; wherein: The image feature preprocessing module consists of a cascade of visual encoders, linear layers, and resampling layers. The visual encoder uses the ViT-L / 14 visual encoder used in the DINOv2 open-world object detection model and the convolutional neural network ConvNeXt with an attention mechanism to capture spatial context and local image information, respectively. The text feature preprocessing module includes a cascaded word segmentation and encoding module and a vectorized representation module. The word segmentation and encoding module uses the language tagger used in natural language processing applications, and the vectorized representation module uses word2vec technology. The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features; The base model outputs a positioning feature identifier that combines image context semantics and world knowledge based on multimodal fusion features; the positioning feature identifier is formed into a target position feature vector through a projection matrix via a linear layer and input into a target position decoder; The target position decoder adopts the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. The target position decoder outputs an abnormal target positioning frame based on the input target position feature vector and image features. The abnormal target positioning frame is used to indicate the position, type, size and confidence of the abnormal target in the fused image to be monitored. The type of abnormal target is an overheated target, an oil leak target or a smoke target; Using a dual-spectral camera to capture visible light images and infrared images of the wind turbine nacelle to be monitored at the same time during operation, and using an image fusion module to fuse the images to obtain a fused image to be monitored, inputting the fused image to be monitored and a character string indicating recognition requirements into the wind turbine nacelle fire warning model to obtain an identification and positioning result of an abnormal target in the wind turbine nacelle to be monitored; A fire warning result for the wind turbine nacelle to be monitored is obtained based on the identification and positioning result of the abnormal target in the wind turbine nacelle to be monitored.

2. The wind turbine nacelle fire early warning method according to claim 1, characterized in that: The performing image fusion on the visible light detail image and the infrared detail image to obtain the fused detail image comprises: A pre-trained residual network model is used to obtain the four-layer depth features of the visible light detail image, and a pre-trained residual network model is used to obtain the four-layer depth features of the infrared detail image. A multi-layer fusion strategy is used to fuse the four-layer depth features of the visible light detail image and the four-layer depth features of the infrared detail image to obtain the fused detail image.

3. The wind turbine cabin fire early warning method according to claim 1, characterized in that: The fire warning status of the wind turbine cabin includes at least one of an equipment overheating state, an oil leakage state, and a fire state; The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine nacelle in each working state is the overheating target. The overheating target is the equipment component in the wind turbine nacelle whose infrared hot spot temperature reaches the temperature threshold; The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine nacelle in each working state is the oil leakage target, which is the equipment component in the wind turbine nacelle where the oil leakage occurs; The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

4. The wind turbine nacelle fire early warning method according to claim 3, characterized in that: Adding fire grade description text for describing the working status of the wind turbine nacelle to the sample fusion image includes: Generate corresponding fire level description text according to predetermined text rules based on the status parameters of each equipment component in the wind turbine nacelle, the status parameters of the smoke target in the wind turbine nacelle, and the fire level displayed by the sample fusion image; The status parameters of each equipment component include the name, installation location, and operating status of the equipment component. The operating status of the equipment component is used to indicate whether the equipment component is in a normal operating state or an abnormal operating state. The abnormal operating state of each equipment component includes at least one of the infrared hotspot temperature reaching a temperature threshold and the occurrence of oil leakage. The status parameters of the equipment component whose infrared hotspot temperature reaches the temperature threshold also include the infrared hotspot temperature. The status parameters of the equipment component whose oil leakage occurs also include the location of the oil leakage. The state parameters of the smoke target in the wind turbine nacelle include the position and size of the smoke target.

5. The wind turbine nacelle fire early warning method according to claim 1, characterized in that: Using the fire common sense image and text dataset to train the VILA multimodal large model and adjust the model parameters includes: The low-rank adaptive decomposition method in the deepspeed zero2 large model fine-tuning acceleration framework is used to train the VILA multimodal large model and adjust the model parameters. The trained LORA model parameters are merged with the initial parameters of the VILA multimodal large model as the model parameters of the trained wind turbine cabin fire warning model.

6. The wind turbine cabin fire early warning method according to claim 5, characterized in that: When using the low-rank adaptive decomposition method to train the VILA multimodal large model and adjust the model parameters, the rank of Lora is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is 2e-5, and the basic learning rate is set to 2e-4.

7. The wind turbine nacelle fire early warning method according to claim 1, characterized in that: The constructed wind turbine cabin fire warning model also includes a visual clue decoder. The output of the base model is also connected to the visual clue decoder through a linear layer. The positioning feature identifier output by the base model is formed into a visual clue feature vector through a projection matrix via a linear layer and input into the visual clue decoder. The output of the image feature preprocessing module and the output of the feature fusion module are also connected to the visual clue decoder respectively. The visual clue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual clue decoder outputs an abnormal target search prompt heat map based on the input visual clue feature vector, image features and multimodal fusion features. The identification and positioning results of the abnormal target in the wind turbine cabin to be monitored also include: Obtaining an abnormal target search prompt heat map output by the wind turbine cabin fire warning model, where the abnormal target search prompt heat map indicates the confidence level of the presence of abnormal targets at different positions in the fused image to be monitored; When the confidence level of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches a confidence threshold, an identification and positioning result including the position, type, and size of the abnormal target indicated by the abnormal target positioning frame is output; When the confidence of the abnormal target positioning frame does not reach the confidence threshold, the fused image to be monitored is divided into multiple sub-regions, and the confidence of the abnormal target contained in each sub-region is determined according to the abnormal target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of the abnormal target contained. For each traversed sub-region: the image of the fused image to be monitored in the sub-region is input into the wind turbine cabin fire warning model to obtain the abnormal target positioning frame for the sub-region. When the confidence of the abnormal target positioning frame reaches the confidence threshold, the identification and positioning result of the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output. Otherwise, the next sub-region is traversed until all sub-regions are traversed and it is determined that the fused image to be monitored does not contain abnormal targets.

Images