Wind power cabin fire early warning method based on multi-modal large model fused with multi-spectral image

Abstract

The invention discloses a wind power cabin fire early warning method for fusing a multi-modal large model with a multispectral image, and relates to the field of fire early warning, and the method comprises the steps: carrying out the image fusion of a visible light image and an infrared image, and obtaining a sample fusion image of a wind power cabin in different working states; the sample fusion image fuses the image features in different spectral modes for advantage complementation, so that the image features of the wind power cabin in the current working state can be better represented, and then the sample fusion image is subjected to image labeling and text description to construct a fire common sense image-text data set; the pre-trained VILA multi-mode large model is subjected to parameter adjustment, so that the established wind power cabin fire early warning model can understand the basic common knowledge of wind power cabin fire early warning, search can be dynamically adjusted according to the target, the recognition and positioning accuracy of the abnormal target of the wind power cabin in the fire early warning state can be improved, and the accuracy of the abnormal target recognition and positioning of the wind power cabin in the fire early warning state can be improved. The accuracy of fire early warning can be improved.

Description

Technical Field

[0001] The present application relates to the field of fire warning, and in particular to a wind turbine cabin fire warning method that fuses a multi-modal large model with a multi-spectral image. Background Art

[0002] Wind power generation is an important and promising branch in the field of renewable energy power generation, and has received great attention and importance from the country. The wind turbine system has a complex structure, and the equipment in the wind turbine cabin is in a state of high-speed rotation and high vibration for a long time. The wind turbine cabin is prone to problems such as lubricant leakage, gearbox overheating, and cable overcurrent insulation reduction. These problems are prone to produce smoke and even open flames, causing equipment fires and threatening the safe and stable operation of wind farms. Therefore, monitoring the above-mentioned problems in the wind turbine cabin is conducive to locating the fire characteristics in the early stage of the fire and issuing early warnings in time. It can assist power station operation and maintenance personnel in carrying out equipment fire safety work and minimize the losses caused by fire to the wind turbine system.

[0003] With the integration and development of computer vision and natural language processing technology, multimodal large models with natural language understanding and image recognition have begun to emerge and have been initially applied in fire monitoring systems of wind farms. However, due to the complex environment of wind turbine cabins and the high background noise, the existing multimodal large model inference and identification of fire characteristics is prone to hallucination defects, and the detection and warning accuracy is not ideal. Summary of the invention

[0004] In response to the above problems and technical requirements, this application proposes a wind turbine cabin fire warning method that integrates a multi-modal large model with multi-spectral images. The technical solution of this application is as follows:

[0005] A wind turbine cabin fire early warning method using a multi-modal large model fused with a multi-spectral image, the wind turbine cabin fire early warning method comprising:

[0006] The dual-spectrum camera is used to obtain sample images of the wind turbine cabin in different working states. The working state of the wind turbine cabin includes a normal working state and different fire warning states. The sample images of the wind turbine cabin in each working state include a sample visible light image and a sample infrared image.

[0007] An image fusion module is used to fuse sample visible light images and sample infrared images in a group of sample images of the wind turbine cabin in each working state to obtain a corresponding sample fused image. The image fusion module performs multi-scale fuzzy transformation on the sample visible light images through a Gaussian kernel function to obtain a visible light basic image and a visible light detail image. The sample infrared images are multi-scale fuzzy transformation is performed through a Gaussian kernel function to obtain an infrared basic image and an infrared detail image. The visible light basic image and the infrared basic image are weighted average fused to obtain a fused basic image. The visible light detail image and the infrared detail image are image fused to obtain a fused detail image. The fused basic image and the fused detail image are reconstructed to obtain a sample fused image of the wind turbine cabin in the current working state.

[0008] The abnormal targets used to identify the fire warning status in the sample fusion images of the wind turbine cabin in each working state are annotated, and the fire level description text used to describe the working status of the wind turbine cabin is added to the sample fusion images to construct a fire common sense image and text dataset;

[0009] The fire common sense graphic data set is used to train the VILA multimodal large model and adjust the model parameters. The base model is trained and then the wind turbine cabin fire warning model is built using the trained base model.

[0010] A dual-spectrum camera is used to collect visible light images and infrared images of the wind turbine cabin to be monitored at the same time during operation, and an image fusion module is used to fuse the images to obtain a fused image to be monitored. The fused image to be monitored and a character string indicating the recognition requirement are input into the wind turbine cabin fire warning model to obtain the recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored.

[0011] Based on the recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored, a fire warning result for the wind turbine cabin to be monitored is obtained.

[0012] A further technical solution is that the visible light detail image and the infrared detail image are fused to obtain a fused detail image, which includes:

[0013] The pre-trained residual network model is used to obtain the 4-layer deep features of the visible light detail image, and the pre-trained residual network model is used to obtain the 4-layer deep features of the infrared detail image. A multi-layer fusion strategy is used to fuse the 4-layer deep features of the visible light detail image and the 4-layer deep features of the infrared detail image to obtain a fused detail image.

[0014] A further technical solution is that the fire warning state of the wind turbine cabin includes at least one of an equipment overheating state, an oil leakage state and a fire state;

[0015] The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine cabin in each working state is the overheating target. The overheating target is the equipment component in the wind turbine cabin where the infrared hot spot temperature reaches the temperature threshold;

[0016] The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine cabin in each working state is the oil leakage target, and the oil leakage target is the equipment component with oil leakage in the wind turbine cabin;

[0017] The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

[0018] A further technical solution is to add a fire level description text for describing the working status of the wind turbine cabin to the sample fusion image, including:

[0019] According to the state parameters of each equipment component in the wind turbine cabin displayed by the sample fusion image, the state parameters of the smoke target in the wind turbine cabin and the fire level, a corresponding fire level description text is generated according to a predetermined text rule;

[0020] The state parameters of each equipment component include the name, installation location and operation status of the equipment component. The operation status of the equipment component is used to indicate whether the equipment component is in a normal operation state or an abnormal operation state. The abnormal operation state of each equipment component includes at least one of the infrared hot spot temperature reaching a temperature threshold and oil leakage. The state parameters of the equipment component whose infrared hot spot temperature reaches the temperature threshold also include the infrared hot spot temperature. The state parameters of the equipment component with oil leakage also include the oil leakage location.

[0021] The state parameters of the smoke target in the wind turbine cabin include the position and size of the smoke target.

[0022] Its further technical solution is to use the fire common sense picture and text dataset to train the VILA multimodal large model and adjust the model parameters including:

[0023] The low-rank adaptive decomposition method in the deepspeed zero2 large model fine-tuning acceleration framework is used to train the VILA multimodal large model and adjust the model parameters. The trained LORA model parameters are merged with the initial parameters of the VILA multimodal large model as the model parameters of the trained wind turbine cabin fire warning model.

[0024] Its further technical solution is that when using the low-rank adaptive decomposition method to train the VILA multimodal large model and adjust the model parameters, the lora rank is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is 2e-5, and the basic learning rate is set to 2e-4.

[0025] Its further technical solution is that the wind turbine cabin fire warning model constructed includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a trained base model and a target position decoder;

[0026] The fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module; the output of the image feature preprocessing module and the output of the text feature preprocessing module are respectively connected to the feature fusion module; the output of the feature fusion module is connected to the base model; the output of the base model is connected to the target position decoder via the linear layer; the output of the image feature preprocessing module is also connected to the target position decoder; wherein:

[0027] The image feature preprocessing module includes a cascaded visual encoder, a linear layer, and a resampling layer. The visual encoder adopts the visual encoder ViT-L / 14 used in the DINOv2 open-world object detection model and the convolutional neural network ConvNeXt with an attention mechanism to capture spatial context information and image local information respectively.

[0028] The text feature preprocessing module includes a word segmentation encoding module and a vectorization representation module which are cascaded in sequence. The word segmentation encoding module uses a language marker in natural language processing applications, and the vectorization representation module uses word2vec technology.

[0029] The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features;

[0030] The base model outputs a positioning feature identifier that combines image context semantics and world knowledge based on multimodal fusion features; the positioning feature identifier is formed into a target position feature vector through a linear layer through a projection matrix and input into a target position decoder;

[0031] The target position decoder adopts the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. The target position decoder outputs an abnormal target positioning frame based on the input target position feature vector and image features. The abnormal target positioning frame is used to indicate the position, type, size and confidence of the abnormal target in the monitored fusion image. The type of abnormal target is overheated target, oil leakage target or smoke target.

[0032] Its further technical solution is that the constructed wind turbine cabin fire warning model also includes a visual clue decoder, the output of the base model is also connected to the visual clue decoder through a linear layer, the positioning feature identifier output by the base model is formed into a visual clue feature vector through a linear layer through a projection matrix and input into the visual clue decoder, the output of the image feature preprocessing module and the output of the feature fusion module are also connected to the visual clue decoder respectively, the visual clue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture, the visual clue decoder outputs an abnormal target search prompt heat map based on the input visual clue feature vector, image features and multimodal fusion features, and the recognition and positioning results of the abnormal targets in the monitored wind turbine cabin also include:

[0033] Obtain an abnormal target search prompt heat map output by the wind turbine cabin fire warning model, where the abnormal target search prompt heat map indicates the confidence level of abnormal targets at different positions in the fused image to be monitored;

[0034] When the confidence of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches the confidence threshold, the recognition and positioning result including the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output;

[0035] When the confidence of the abnormal target positioning frame does not reach the confidence threshold, the fused image to be monitored is divided into multiple sub-regions, and the confidence of each sub-region containing abnormal targets is determined according to the abnormal target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of containing abnormal targets. For each traversed sub-region: the image of the fused image to be monitored in the sub-region is input into the wind turbine cabin fire warning model to obtain the abnormal target positioning frame for the sub-region. When the confidence of the abnormal target positioning frame reaches the confidence threshold, the identification and positioning result of the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output, otherwise, the next sub-region is traversed until all sub-regions are traversed and it is determined that the fused image to be monitored does not contain abnormal targets.

[0036] The beneficial technical effects of this application are:

[0037] The present application discloses a wind turbine cabin fire warning method of fusing multi-spectral images with a multi-modal large model. The method uses visible light images and infrared images for image fusion to obtain sample fusion images of the wind turbine cabin in different working states. The sample fusion images fuse image features under different spectral modes, which can achieve complementary advantages and better characterize the image features of the wind turbine cabin in the current working state. The sample fusion images under different working states are then used to perform image annotation and text description to construct a fire common sense graphic data set. The pre-trained VILA multi-modal large model is then parameterized using the fire common sense graphic data set, so that the constructed wind turbine cabin fire warning model can understand the basic common sense of wind turbine cabin fire warning, so that the search can be dynamically adjusted according to the target, which is conducive to improving the accuracy of identifying and locating abnormal targets, and can improve the accuracy of fire warning.

[0038] The wind turbine cabin fire warning model built by this method can also output an abnormal target search prompt heat map. The abnormal target search prompt heat map is used to introduce an active search mechanism, visual working memory, and a dynamic adjustment strategy. By simulating the human recognition mechanism of abnormal targets, the world knowledge and context understanding ability of the large model can be fully utilized, and the accuracy of the recognition and positioning of abnormal targets can be further improved, thereby further improving the accuracy of wind turbine cabin fire warnings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a method flow chart of a wind turbine cabin fire warning method according to an embodiment of the present application.

[0040] Figure 2 It is a flow chart of a method for obtaining a corresponding sample fused image by using an image fusion module to fuse a sample visible light image and a sample infrared image in a group of sample images in one embodiment of the present application.

[0041] Figure 3 It is a model structure diagram of a wind turbine cabin fire warning model constructed in one embodiment of the present application.

[0042] Figure 4 yes Figure 3 Block diagram of the target position decoder in .

[0043] Figure 5 yes Figure 3 The structure of the visual cue decoder in . DETAILED DESCRIPTION

[0044] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0045] This application discloses a wind turbine cabin fire warning method that integrates multi-modal large model and multi-spectral image. Please refer to Figure 1 The method comprises the following steps:

[0046] Step 1: Use a dual-spectrum camera to obtain sample images of the wind turbine cabin in different working states. A set of sample images of the wind turbine cabin in each working state includes a sample visible light image and a sample infrared image at the same time.

[0047] The working state of the wind turbine cabin includes the normal working state and different fire warning states. The fire warning state is an abnormal state in which the wind turbine cabin has a fire risk. The fire warning state includes at least one of the equipment overheating state, oil leakage state and fire state. The equipment overheating state is a state in which there are equipment components in the wind turbine cabin that have reached the infrared hot spot temperature and reached the temperature threshold and are operating abnormally. The oil leakage state is a state in which there are equipment components in the wind turbine cabin that have oil leakage and are operating abnormally. The fire state is a state in which a fire has occurred in the wind turbine cabin and smoke has been generated.

[0048] When obtaining sample images of a wind turbine cabin, on the one hand, sample images of the wind turbine cabin in different working states during the historical operation process can be obtained, such as directly obtaining images obtained by the surveillance camera during the historical operation of the wind turbine cabin. On the other hand, a wind turbine cabin simulation experiment platform can be built, and then different working states of the wind turbine cabin can be simulated on the wind turbine cabin simulation experiment platform and corresponding sample images can be obtained, such as changing the power of the gearbox in the wind turbine cabin so that the infrared hot spot temperature of the power box rises to the temperature threshold to simulate the overheating state of the equipment. Another example is to apply lubricating oil on the gearbox to simulate the oil leakage state. Another example is to ignite a cigarette cake in the wind turbine cabin to generate smoke to simulate a fire state.

[0049] Regardless of the method used to obtain the sample image of the wind turbine cabin, the actual dual-spectrum camera directly obtains the video stream. First, the video image frames are extracted from the video stream according to a certain time interval, and the video image frames under different working states are extracted and subjected to pre-processing operations such as denoising, defogging, and deraining as sample images. In addition, the sample visible light image and the sample infrared image in the same group of sample images often need to be fully registered, which will not be described in detail in this embodiment.

[0050] Step 2: Use an image fusion module to fuse sample visible light images and sample infrared images in a group of sample images of the wind turbine cabin in each working state to obtain a corresponding sample fused image.

[0051] Please refer to Figure 2The image fusion module shown is a schematic diagram of image fusion of a sample visible light image and a sample infrared image. The image fusion module first performs a multi-scale fuzzy transformation on the sample visible light image in the group of sample images through a Gaussian kernel function to obtain a visible light basic image and a visible light detail image, and performs a multi-scale fuzzy transformation on the sample infrared image in the group of sample images through a Gaussian kernel function to obtain an infrared basic image and an infrared detail image.

[0052] The visible light basic image and the infrared basic image are fused at the pixel level by weighted average to eliminate the interference of the light source and obtain the fused basic image.

[0053] The visible light detail image and the infrared detail image are fused at the pixel level to obtain a fused detail image. Specifically, the 4-layer depth features of the visible light detail image are obtained using a pre-trained residual network model, and the 4-layer depth features of the infrared detail image are obtained using a pre-trained residual network model; then a multi-layer fusion strategy is used to fuse the 4-layer depth features of the visible light detail image and the 4-layer depth features of the infrared detail image to obtain a fused detail image.

[0054] Finally, the fused basic image and the fused detail image are reconstructed to obtain a sample fusion image of the wind turbine cabin in the current working state. The obtained sample fusion image integrates the modal characteristics under different spectral images and can provide more comprehensive and accurate image features.

[0055] Step 3: Annotate the abnormal targets used to identify the fire warning status in the sample fusion images of the wind turbine cabin in each working state, and add fire level description text used to describe the working state of the wind turbine cabin to the sample fusion images to construct a fire common sense graphic and text dataset.

[0056] The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine cabin in each working state is the overheating target. The overheating target is the equipment component whose infrared hot spot temperature in the wind turbine cabin reaches the temperature threshold and the overheating problem occurs.

[0057] The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine cabin in each working state is the oil leakage target, and the oil leakage target is the equipment component with oil leakage in the wind turbine cabin.

[0058] The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

[0059] It should be noted that there may be one or more types of abnormal targets in the sample fusion image of the wind turbine cabin in each working state, corresponding to one or more fire warning states in the wind turbine cabin in the current working state. For example, the wind turbine cabin may have both equipment overheating and fire, then there will be both overheating targets and smoke targets in the sample fusion image. In practical applications, only some key equipment components in the wind turbine cabin can be considered, or all equipment components in the wind turbine cabin can be considered.

[0060] When performing image annotation, use image annotation software such as labelimg to annotate the abnormal targets in the sample fusion image, including using annotation boxes to annotate the position, type and size of the abnormal targets in the sample fusion image. The size of the abnormal target is defined by the proportion of the abnormal target occupying the sample fusion image, such as small size, medium size and large size.

[0061] In addition to image annotation, a fire level description text is added to each sample fusion image. The fire level description text is used to describe the working state of the wind turbine cabin in the sample fusion image, including: according to the state parameters of each equipment component of the wind turbine cabin displayed in the sample fusion image, the state parameters of the smoke target in the wind turbine cabin, and the fire level, the corresponding fire level description text is generated according to the predetermined text rules. The division method of the fire level of the wind turbine cabin in each working state can be customized.

[0062] The state parameters of each equipment component include the name, installation location and operation status of the equipment component. The operation status of the equipment component is used to indicate whether the equipment component is in a normal operation state or an abnormal operation state. The abnormal operation state of each equipment component includes at least one of the infrared hot spot temperature reaching the temperature threshold and the occurrence of oil leakage. For the equipment component whose infrared hot spot temperature reaches the temperature threshold, the state parameter of the equipment component also includes the infrared hot spot temperature. For the equipment component with oil leakage, the state parameter of the equipment component also includes the oil leakage location.

[0063] The state parameters of the smoke target in the wind turbine cabin include the position and size of the smoke target. The state parameters of the smoke target may also include at least one of the color characteristics, optical characteristics, texture characteristics and physical and chemical characteristics of the smoke target.

[0064] Taking the gearbox, a relatively critical equipment component in a wind turbine nacelle, as an example, the following examples show examples of adding fire grade description texts to sample fusion images in different working states. The same is true when the wind turbine nacelle also includes other equipment components:

[0065] In one example, the fire level description text added to the sample fused image when the wind turbine cabin is in normal working condition is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox is in normal working condition, there is no smoke target, and the fire level is 0}.

[0066] In another example, the fire level description text added to the sample fused image when the wind turbine cabin is in an equipment overheating state is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox has an overheating problem, the infrared hot spot temperature of the gearbox is 120°C, there is no smoke target, and the fire level is 1}.

[0067] In another example, the fire level description text added to the sample fused image when the wind turbine cabin is in an oil leakage state is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox has an oil leakage, the gearbox oil leakage position is [0.66, 0.74, 0.89, 0.82], there is no smoke target, and the fire level is 2}

[0068] In another example, the fire level description text added to the sample fused image when the wind turbine cabin is on fire is {the gearbox position is [0.66, 0.74, 0.73, 0.76], the gearbox is in normal operation, the position of the smoke target is [0.23, 0.46, 0.57, 0.69], the smoke target is of medium size, and the fire level is 4}.

[0069] Step 4: Use the fire knowledge graphic data set to train the VILA multimodal large model and adjust the model parameters to obtain the base model through training. Use the trained base model to build a wind turbine cabin fire warning model.

[0070] The current common practice is to directly download the pre-trained VILA multimodal large model for reasoning, but this application does not directly use the open source pre-trained VILA multimodal large model, but uses the fire common sense graphic data set of the wind turbine cabin to adjust the model parameters of the VILA multimodal large model, and finally obtains a base model that has the ability to understand the basic common sense of wind turbine cabin fire warning. Moreover, the fire common sense graphic data set used by this application to adjust the model parameters is a dual-spectrum image feature that integrates visible light images and infrared images, which can make full use of the complementary advantages of different spectral images so that the base model can dynamically adjust the search according to the target, thereby helping to improve the recognition effect.

[0071] The VILA multimodal large model can understand and process text and images, and realize the image-to-text function of text analysis. The VILA multimodal large model uses alternating text data with images for pre-training. This application further uses the fire common sense image and text data set obtained by dual-spectral image fusion and matching to fine-tune the model parameters. This staged training method enables the model to better learn the joint representation of vision and language, effectively retain the model's world knowledge ability, and alleviate the forgetting of the model's pure text ability.

[0072] In one embodiment, the low-rank adaptive decomposition method in the deepspeed zero2 large model fine-tuning acceleration framework is used to train the VILA multimodal large model and adjust the model parameters, and the trained LORA model parameters are merged with the initial parameters of the VILA multimodal large model as the model parameters of the trained wind turbine cabin fire warning model. Specifically, when the VILA multimodal large model is trained and the model parameters are adjusted using the low-rank adaptive decomposition method, the rank of lora is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is 2e-5, and the basic learning rate is set to 2e-4.

[0073] The VILA multimodal large model is different from the multimodal large model based on internal fusion of cross attention and internal fusion based on custom cross attention layer. This multimodal large model fuses the image modality and text modality at an early stage. This fusion method can effectively capture the multimodal representation of images and texts by mapping different modalities to a common semantic space, and shows unique advantages in multimodal representation learning, cross-modal generation tasks, and zero-sample downstream migration tasks.

[0074] This application uses the trained base model to build a wind turbine cabin fire warning model. Figure 3 As shown, it includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a trained base model and a target position decoder. Among them, the output of the image feature preprocessing module and the output of the text feature preprocessing module are respectively connected to the feature fusion module. The output of the feature fusion module is connected to the base model. The output of the base model is connected to the target position decoder via a linear layer. The output of the image feature preprocessing module is also connected to the target position decoder.

[0075] The image feature preprocessing module includes a visual encoder, a linear layer, and a resampling layer that are cascaded in sequence. The visual encoder uses the visual encoder ViT-L / 14 used in the DINOv2 open-world object detection model and the convolutional neural network ConvNeXt with an attention mechanism to capture spatial context information and local image information, respectively, and then extract visual features that may contain abnormal targets. The linear layer linearly transforms the high-dimensional visual features output by the visual encoder into features suitable for language model processing. The resampling layer uses a sampler sampler to downsample high-dimensional visual features to a low-dimensional space through a cross-attention mechanism, which can not only reduce the number of features, but also highlight important visual features and ignore minor visual features. The combination of linear layers and resampling layers can flexibly process visual features in different scenarios, balance computational efficiency and model performance, and more efficiently process complex visual analysis tasks.

[0076] The text feature preprocessing module includes a cascaded word segmentation encoding module and a vectorized representation module. The word segmentation encoding module uses the language tagger in the natural language processing application to treat each byte as a separate vocabulary unit, and then identifies the byte pairs with the highest frequency in the text based on statistical analysis and merges them into a new vocabulary unit. This process is repeated until the predetermined vocabulary size is reached or there are no more byte pairs to be merged. The vectorized representation module uses word2vec technology to map each subword into a one-dimensional vector embedding and converts it into a feature vector in the embedding space of the large language model. These feature vectors are injected into the base model as text information.

[0077] The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features.

[0078] The base model is trained based on the VILA-2.7b open source multimodal large model pre-trained on multiple benchmark datasets. The base model outputs a positioning feature identifier that combines image context semantics and world knowledge based on multimodal fusion features. The positioning feature identifier is fed into the target position decoder through a linear layer through a projection matrix to form a target position feature vector, and the positioning feature identifier is fed into the visual cue decoder through another linear layer through a projection matrix to form a visual cue feature vector.

[0079] The target position decoder uses the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. Please refer to Figure 4 The target position decoder shown in the structural diagram includes an organically combined multi-layer perceptron layer and a cosine similarity calculation module.

[0080] Furthermore, the constructed wind turbine cabin fire warning model also includes a visual cue decoder, the output of the base model is also connected to the visual cue decoder through a linear layer, and the output of the image feature preprocessing module and the output of the feature fusion module are also connected to the visual cue decoder respectively.

[0081] The visual cue decoder uses the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual cue decoder outputs an abnormal target search prompt heat map based on the input visual cue feature vector, image features and multimodal fusion features. Please refer to Figure 5 The structure diagram of the visual cue decoder is shown in Figure 1. The visual cue decoder is a lightweight network component. Each decoder layer performs the following steps: taking the multimodal fusion feature and the visual cue feature vector as the positioning feature, the self-attention layer first performs self-attention on the positioning feature, then the label-image cross-attention layer performs cross-attention on the query positioning feature and the image feature, then the multi-layer perceptron layer updates each positioning feature point by point, and finally the image-label cross-attention layer performs cross-attention from the image feature to the positioning feature. In order to ensure that the visual cue decoder can access key geometric information, when participating in the attention layer operation, the position encoding will be added to the image embedding vector, and the entire original positioning feature containing the position information will also be re-added to the image embedding vector. Through this clever design, the visual cue decoder prediction head can dynamically and real-time output segmentation masks, and has the ability to perceive ambiguity, can handle ambiguous prompts and generate multiple possible valid masks, and output abnormal target search prompt heat maps, which indicate the confidence of the existence of abnormal targets at different positions in the fused image to be monitored.

[0082] Step 5: Use a dual-spectrum camera to collect visible light images and infrared images of the wind turbine cabin to be monitored at the same time during operation, and use an image fusion module to fuse the images to obtain a fused image to be monitored. Input the fused image to be monitored and a character string for indicating recognition requirements into the wind turbine cabin fire warning model to obtain the recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored.

[0083] This step is similar to the above method of acquiring sample images when collecting visible light images and infrared images of the wind turbine cabin to be monitored at the same time, and also includes acquiring video streams, capturing image frames, and performing various image preprocessing and image registration operations, which will not be repeated here.

[0084] The method of using the image fusion module to fuse the visible light image and the infrared image at the same time to obtain the fused image to be monitored can refer to the process of obtaining the sample fused image in step 2 above, which will not be repeated here.

[0085] The character string used to indicate the recognition requirement can be customized, for example, the character string may be "Please locate the smoke target in the image", or other similar character strings that can indicate the recognition requirement may be used.

[0086] When the fused image to be monitored and the character string indicating the recognition requirement are input into the wind turbine cabin fire warning model, the fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module. The wind turbine cabin fire warning model built based on the multimodal large model understands and processes the input fused image to be monitored and the character string. The target position decoder outputs an abnormal target positioning frame based on the target position feature vector and the image feature. The abnormal target positioning frame is used to indicate the position, type, size and confidence of the abnormal target in the fused image to be monitored. The type of abnormal target is an overheated target, an oil leakage target or a smoke target.

[0087] One approach is to obtain the recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored based on the abnormal target positioning frame of the wind turbine cabin fire warning model. For the abnormal target positioning frame output by the wind turbine cabin fire warning model, when its confidence reaches the confidence threshold, it can be considered that there is an abnormal target for identifying the fire warning state at the abnormal target positioning frame, otherwise it is considered that there is no abnormal target at the abnormal target positioning frame, and finally the recognition and positioning results can be output including the position, type and size of all abnormal targets in the fused image to be monitored.

[0088] However, this approach is not ideal when identifying small-sized targets. For example, there will be a large number of missed detections and false detections when identifying small-sized smoke targets and small-sized oil leakage targets. Therefore, in another embodiment, the constructed wind turbine cabin fire warning model also includes a visual cue decoder. In addition to outputting an abnormal target positioning frame through a target position decoder, the positioning feature identifier output by the base model is input into the visual cue decoder through a linear layer and a projection matrix to form a visual cue feature vector. The visual cue decoder will output an abnormal target search prompt heat map based on the input visual cue feature vector, image features, and multimodal fusion features. The abnormal target search prompt heat map is used to indicate the confidence level of the existence of abnormal targets at different positions in the fused image to be monitored. After obtaining the abnormal target positioning frame and the abnormal target search prompt heat map output by the wind turbine cabin fire warning model, the abnormal target positioning frame and the abnormal target search prompt heat map are comprehensively considered to obtain the identification and positioning results, including:

[0089] When the confidence of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches the confidence threshold, the identification and positioning result including the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output.

[0090] When the confidence of the abnormal target positioning frame does not reach the confidence threshold, it is not directly determined that there is no abnormal target, but the fused image to be monitored is divided into multiple sub-regions, and the confidence of each sub-region containing abnormal targets is determined according to the abnormal target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of containing abnormal targets. For each traversed sub-region: the image of the fused image to be monitored in the sub-region is input into the wind turbine cabin fire warning model to obtain the abnormal target positioning frame for the sub-region. When the confidence of the abnormal target positioning frame reaches the confidence threshold, the recognition and positioning result of the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output, otherwise, the next sub-region is continued to be traversed until all sub-regions are traversed to determine that there is no abnormal target in the fused image to be monitored. This approach introduces active search mechanism, visual working memory and dynamic adjustment strategy. By simulating the human recognition mechanism of abnormal targets, it can make full use of the world knowledge and context understanding ability of the multimodal large model, and improve the accuracy of visual positioning and multimodal collaborative reasoning ability.

[0091] Step 6: obtaining a fire warning result for the wind turbine cabin to be monitored based on the identification and positioning result of the abnormal target in the wind turbine cabin to be monitored.

[0092] The identification and positioning results obtained through the above identification and detection can indicate the location, type, size and other information of the abnormal target. The corresponding fire level can be determined according to the content of the identification and positioning results and the corresponding fire warning results can be output. The fire level and fire warning results corresponding to different identification and positioning results can be customized.

[0093] For example, if we only focus on the gearbox in the wind turbine cabin, when the recognition and positioning results indicate that there are no abnormal targets in the monitored fusion image and it is in normal working state, no prompt will be given or the wind turbine cabin will be prompted to operate normally. When the recognition and positioning results indicate that there is an overheated target, the text "The position of the gearbox is [0.66, 0.74, 0.73, 0.76], the gearbox has an overheating problem, the infrared hot spot temperature of the gearbox is 120°C, there is no smoke target, and the fire level is 1" will pop up in the video screen, and the screen and text will be sent to the WeChat of the operation and maintenance personnel.

[0094] This application also provides an example to verify the effect, obtaining visible light images and infrared images of the wind turbine cabin in different working conditions at the same time, and performing image fusion to obtain the fused image to be monitored, thereby constructing a wind turbine cabin test set.

[0095] The open-source pre-trained VILA multimodal large model was directly used to test the visible light images in the wind turbine cabin test set, and the prediction accuracy was 79.23%. The open-source pre-trained VILA multimodal large model was directly used to test the infrared images in the wind turbine cabin test set, and the prediction accuracy was 80.15%. It can be seen that due to the limitations of the single-spectrum image representation features, and the pre-trained VILA multimodal large model did not dynamically adjust the search according to the target, the multimodal model produced hallucination defects similar to the large language model, so the prediction accuracy can only reach 80.15% at most.

[0096] Then, the method of the present application is adopted. First, the fire common sense graphic data set of the fused bispectral image is used to adjust the model parameters of the VILA multimodal large model, and then the fused image to be monitored in the wind turbine cabin test set is used for prediction. At this time, the prediction accuracy can be improved to 86.72%. Furthermore, after active search combined with the abnormal target search prompt heat map output by the wind turbine cabin fire warning model, the prediction accuracy can be further improved to 95.37%, which effectively solves the problem of false detection caused by the lack of historical memory of the multimodal large model, and can effectively filter out suspected abnormal targets, so that the accuracy is further improved.

[0097] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A wind turbine cabin fire warning method based on multi-modal large model fusion multi-spectral image, characterized in that: The wind turbine cabin fire early warning method comprises: The dual-spectrum camera is used to obtain sample images of the wind turbine cabin in different working states. The working state of the wind turbine cabin includes a normal working state and different fire warning states. The sample images of the wind turbine cabin in each working state include a sample visible light image and a sample infrared image. An image fusion module is used to fuse sample visible light images and sample infrared images in a group of sample images of the wind turbine cabin in each working state to obtain a corresponding sample fused image. The image fusion module performs multi-scale fuzzy transformation on the sample visible light images through a Gaussian kernel function to obtain a visible light basic image and a visible light detail image, and performs multi-scale fuzzy transformation on the sample infrared images through a Gaussian kernel function to obtain an infrared basic image and an infrared detail image; weighted average fusion is performed on the visible light basic image and the infrared basic image to obtain a fused basic image, image fusion is performed on the visible light detail image and the infrared detail image to obtain a fused detail image, and the fused basic image and the fused detail image are reconstructed to obtain a sample fused image of the wind turbine cabin in the current working state; Annotate the abnormal targets used to identify the fire warning state in the sample fusion images of the wind turbine cabin in each working state, and add the fire level description text used to describe the working state of the wind turbine cabin to the sample fusion images to construct a fire common sense image and text dataset; The fire common sense graphic data set is used to train the VILA multimodal large model and adjust the model parameters to obtain a base model through training, and a wind turbine cabin fire warning model is constructed based on the trained base model; A dual-spectrum camera is used to collect visible light images and infrared images of the wind turbine cabin to be monitored at the same time during operation, and an image fusion module is used to fuse the images to obtain a fused image to be monitored, and the fused image to be monitored and a character string for indicating recognition requirements are input into the wind turbine cabin fire warning model to obtain recognition and positioning results of abnormal targets in the wind turbine cabin to be monitored; A fire warning result for the wind turbine cabin to be monitored is obtained based on the identification and positioning result of the abnormal target in the wind turbine cabin to be monitored.

2. The wind turbine cabin fire early warning method according to claim 1, characterized in that: The image fusion of the visible light detail image and the infrared detail image to obtain the fused detail image comprises: A pre-trained residual network model is used to obtain the four-layer depth features of the visible light detail image, and a pre-trained residual network model is used to obtain the four-layer depth features of the infrared detail image. A multi-layer fusion strategy is used to fuse the four-layer depth features of the visible light detail image and the four-layer depth features of the infrared detail image to obtain the fused detail image.

3. The wind turbine cabin fire early warning method according to claim 1, characterized in that: The fire warning state of the wind turbine cabin includes at least one of an equipment overheating state, an oil leakage state and a fire state; The abnormal target used to identify the overheating state of the equipment in the sample fusion image of the wind turbine cabin in each working state is the overheating target. The overheating target is the equipment component in the wind turbine cabin where the infrared hot spot temperature reaches the temperature threshold; The abnormal target used to identify the oil leakage state in the sample fusion image of the wind turbine cabin in each working state is the oil leakage target, and the oil leakage target is the equipment component with oil leakage in the wind turbine cabin; The abnormal target used to identify the fire state in the sample fusion image of the wind turbine cabin in each working state is the smoke target.

4. The wind turbine cabin fire early warning method according to claim 3, characterized in that: Adding fire grade description text for describing the working status of the wind turbine cabin to the sample fusion image includes: Generate corresponding fire grade description text according to the state parameters of each equipment component of the wind turbine cabin displayed by the sample fusion image, the state parameters of the smoke target in the wind turbine cabin, and the fire grade according to the predetermined text rules; The state parameters of each equipment component include the name, installation location and operation status of the equipment component. The operation status of the equipment component is used to indicate whether the equipment component is in a normal operation state or an abnormal operation state. The abnormal operation state of each equipment component includes at least one of the infrared hot spot temperature reaching a temperature threshold and oil leakage. The state parameters of the equipment component whose infrared hot spot temperature reaches the temperature threshold also include the infrared hot spot temperature. The state parameters of the equipment component with oil leakage also include the oil leakage location. The state parameters of the smoke target in the wind turbine cabin include the position and size of the smoke target.

5. The wind turbine cabin fire early warning method according to claim 1, characterized in that: Using the fire common sense graphic data set to train the VILA multimodal large model and adjust the model parameters includes: The low-rank adaptive decomposition method in the deepspeed zero2 large model fine-tuning acceleration framework is used to train the VILA multimodal large model and adjust the model parameters. The trained LORA model parameters are merged with the initial parameters of the VILA multimodal large model as the model parameters of the trained wind turbine cabin fire warning model.

6. The wind turbine cabin fire early warning method according to claim 5, characterized in that: When using the low-rank adaptive decomposition method to train the VILA multimodal large model and adjust the model parameters, the rank of lora is set to 128, the alpha parameter is set to 256, the learning rate of the multimodal projection layer is 2e-5, and the basic learning rate is set to 2e-4.

7. The wind turbine cabin fire early warning method according to claim 3, characterized in that: The constructed wind turbine cabin fire warning model includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, the trained base model and a target position decoder; The fused image to be monitored is input into the image feature preprocessing module, and the character string indicating the recognition requirement is input into the text feature preprocessing module; The output of the image feature preprocessing module and the output of the text feature preprocessing module are connected to the feature fusion module respectively; The output of the feature fusion module is connected to the base model; the output of the base model is connected to the target position decoder via a linear layer; The output of the image feature preprocessing module is also connected to the target position decoder; where: The image feature preprocessing module includes a cascaded visual encoder, a linear layer, and a resampling layer. The visual encoder adopts the visual encoder ViT-L / 14 used in the DINOv2 open-world object detection model and the convolutional neural network ConvNeXt with an attention mechanism to capture spatial context information and image local information respectively. The text feature preprocessing module includes a word segmentation encoding module and a vectorization representation module which are cascaded in sequence. The word segmentation encoding module uses a language marker in natural language processing applications, and the vectorization representation module uses word2vec technology. The feature fusion module is used to perform feature splicing on the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features; The base model outputs a positioning feature identifier that combines image context semantics and world knowledge based on multimodal fusion features; the positioning feature identifier is formed into a target position feature vector through a linear layer through a projection matrix and input into a target position decoder; The target position decoder adopts the decoder structure of the detection head model architecture of the target detection model DETR based on the Transformer architecture. The target position decoder outputs an abnormal target positioning frame based on the input target position feature vector and image features. The abnormal target positioning frame is used to indicate the position, type, size and confidence of the abnormal target in the monitored fusion image. The type of abnormal target is an overheated target, an oil leakage target or a smoke target.

8. The wind turbine cabin fire early warning method according to claim 7, characterized in that: The constructed wind turbine cabin fire warning model also includes a visual clue decoder. The output of the base model is also connected to the visual clue decoder through a linear layer. The positioning feature identifier output by the base model is formed into a visual clue feature vector through a linear layer through a projection matrix and input into the visual clue decoder. The output of the image feature preprocessing module and the output of the feature fusion module are also connected to the visual clue decoder respectively. The visual clue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual clue decoder outputs an abnormal target search prompt heat map based on the input visual clue feature vector, image features and multimodal fusion features. The recognition and positioning results of the abnormal targets in the wind turbine cabin to be monitored also include: Acquire an abnormal target search prompt heat map output by the wind turbine cabin fire warning model, where the abnormal target search prompt heat map indicates the confidence of the presence of abnormal targets at different positions in the fused image to be monitored; When the confidence of the abnormal target positioning frame output by the wind turbine cabin fire warning model reaches the confidence threshold, an identification and positioning result including the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output; When the confidence of the abnormal target positioning frame does not reach the confidence threshold, the fused image to be monitored is divided into multiple sub-regions, and the confidence of the abnormal target contained in each sub-region is determined according to the abnormal target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of the abnormal target contained. For each traversed sub-region: the image of the fused image to be monitored in the sub-region is input into the wind turbine cabin fire warning model to obtain the abnormal target positioning frame for the sub-region. When the confidence of the abnormal target positioning frame reaches the confidence threshold, the identification and positioning result of the position, type and size of the abnormal target indicated by the abnormal target positioning frame is output, otherwise, the next sub-region is traversed until all sub-regions are traversed and it is determined that the fused image to be monitored does not contain abnormal targets.

Images