Photovoltaic power station fire warning method fusing visual search and multi-modal large model

Abstract

The application discloses a photovoltaic power station fire warning method fusing visual search and a multi-modal large model, and relates to the field of fire warning. The method uses a smoke target recognition model based on a multi-modal large model to preliminarily recognize and position a to-be-monitored image frame to obtain a smoke target positioning frame, a confidence degree and a smoke target search prompt heat map. When the confidence degree of the smoke target positioning frame does not reach a confidence degree threshold, the method further uses a visual search strategy to actively search based on the smoke target search prompt heat map, introduces an active search mechanism, a visual working memory and a dynamic adjustment strategy, simulates a human recognition mechanism for a smoke target, fully utilizes world knowledge and context understanding capability of the smoke target recognition model, enables the method to directly perform reasoning in a training-free environment, has a high recognition and positioning accuracy for the smoke target, and effectively realizes early fire warning of the photovoltaic power station and reduces implementation difficulty.

Description

TECHNICAL FIELD

[0001] The application relates to the field of fire early warning, in particular to a photovoltaic power station fire early warning method fusing visual search and a multi-modal large model. BACKGROUND

[0002] Photovoltaic power generation is a very promising important branch in the field of new energy power generation, and is highly concerned and valued by the state. However, after a long time of operation, photovoltaic components of photovoltaic power stations are prone to internal problems such as material aging and broken grids. In addition, large photovoltaic power stations are often built in the wild, so photovoltaic components are also prone to be affected by high temperature, sandstorms and other external environments. Photovoltaic components are prone to overheating under the influence of internal and external factors, which may cause a fire and threaten the safe and stable operation of equipment in the new power system.

[0003] Smoke is usually a feature of early fire, so accurate identification of smoke can assist the operation and maintenance personnel of photovoltaic power stations to carry out equipment fire safety work, minimize the impact of fire and protect the safety of operation and maintenance personnel. With the development of computer vision technology, the use of video to realize fire smoke early warning and monitoring has gradually been applied in many photovoltaic power stations. However, smoke itself is not easy to be detected in the video, and the working environment of photovoltaic power stations is complex, so the phenomena of false positives and false negatives are serious, and the early warning accuracy is not high. SUMMARY

[0004] In view of the above problems and technical needs, the application provides a photovoltaic power station fire early warning method fusing visual search and a multi-modal large model. The technical scheme of the application is as follows:

[0005] The photovoltaic power station fire early warning method fusing visual search and a multi-modal large model comprises:

[0006] A smoke target identification model is built, and the smoke target identification model comprises an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multi-modal large model module, a target position decoder and a visual clue decoder;

[0007] The working video stream of the photovoltaic power station to be monitored is acquired and image frames to be monitored are extracted in sequence, each image frame to be monitored is input into an image feature preprocessing module in the smoke target recognition model, a character string for indicating a recognition requirement is input into a text feature preprocessing module in the smoke target recognition model, an image feature extracted by the image feature preprocessing module and a text feature extracted by the text feature preprocessing module are fused by a feature fusion module and then input into a multimodal large model module, the multimodal large model module outputs a target position feature vector to a target position decoder through a linear layer, the multimodal large model module also outputs a visual clue feature vector to a visual clue decoder through a linear layer, the target position decoder outputs a smoke target positioning box and a confidence based on the target position feature vector and the image feature extracted by the image feature preprocessing module, and the visual clue decoder outputs a smoke target search prompt heat map based on the clue position feature vector, the image feature extracted by the image feature preprocessing module and the multimodal fusion feature output by the feature fusion module; the smoke target search prompt heat map indicates the confidence of different positions containing the smoke target.

[0008] When the confidence of the smoke target positioning box does not reach the confidence threshold, a visual search strategy is used to actively search the smoke target search prompt heat map until a smoke target positioning box reaching the confidence threshold is obtained, or until it is determined that the smoke target is not contained in the image frame to be monitored.

[0009] The smoke target recognition result is obtained according to the smoke target positioning box reaching the confidence threshold, and a fire warning result for the photovoltaic power station to be monitored is obtained according to the smoke target recognition result.

[0010] A further technical solution is that the active search based on the smoke target search prompt heat map by using the visual search strategy includes:

[0011] The image frame to be monitored is divided into a plurality of sub-regions, the confidence of each sub-region containing the smoke target is determined based on the smoke target search prompt heat map, and each sub-region is traversed in order from high to low according to the confidence of containing the smoke target, for each sub-region traversed: the image of the image frame to be monitored in the sub-region is input into the image feature preprocessing module in the smoke target recognition model, and the character string for indicating the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model, to obtain the smoke target positioning box and the confidence for the sub-region.

[0012] A further technical solution is that the smoke target recognition result is obtained according to the smoke target positioning box reaching the confidence threshold, and a fire warning result for the photovoltaic power station to be monitored is obtained according to the smoke target recognition result.

[0013] After each smoke target positioning box reaching the confidence threshold is enlarged, a target local image of the image frame to be monitored in the enlarged smoke target positioning box is extracted.

[0014] The image feature preprocessing module is used to extract the image feature of each target local image, and the smoke target recognition result is obtained based on the similarity between the image feature of each target local image and the fingerprint feature of the smoke target in the fire smoke feature fingerprint library.

[0015] A further technical solution is that the smoke target recognition result is obtained based on the similarity between the image feature of the target local image and the fingerprint feature of the smoke target in the fire smoke feature fingerprint library, which includes:

[0016] The KNN algorithm is used to calculate the similarity between the image feature of the target local image and the visual fingerprint feature of the smoke target in the fire smoke feature fingerprint library and obtain the visual classification result, and the KNN algorithm is used to calculate the similarity between the image feature of the target local image and the text fingerprint feature of the smoke target in the fire smoke feature fingerprint library and obtain the text classification result.

[0017] The smoke target recognition result is obtained based on the visual classification result and the text classification result.

[0018] A further technical solution is that the smoke target recognition result is obtained based on the visual classification result and the text classification result, which includes:

[0019] For each target local image, when the visual classification result indicates that the image feature of the target local image does not belong to any visual fingerprint feature classification, and the text classification result indicates that the image feature of the target local image does not belong to any text fingerprint feature classification, it is determined that the target local image does not include a smoke target, otherwise it is determined that the target local image includes a smoke target; the smoke target recognition result is obtained by integrating all target local images in the image frame to be monitored.

[0020] A further technical solution is that the fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes, and obtaining the smoke target recognition result based on the visual classification result and the text classification result further includes:

[0021] For each target local image, the size of the smoke target corresponding to the fingerprint feature in the visual fingerprint feature classification and the text fingerprint feature classification to which the image feature of the target local image belongs is determined, and the size of the smoke target contained in the target local image is determined according to the majority rule.

[0022] A further technical solution is that the photovoltaic power station fire warning method further includes:

[0023] A plurality of photovoltaic power station sample images containing smoke targets are obtained, the size of the smoke target in each photovoltaic power station sample image is labeled, and text description information describing the smoke feature of the smoke target in the photovoltaic power station sample image is added to each photovoltaic power station sample image.

[0024] For each photovoltaic power station sample image, image features of the photovoltaic power station sample image are extracted as visual fingerprint features by the image feature preprocessing module, and text features of the text description information of the photovoltaic power station sample image are extracted as text fingerprint features by the text feature preprocessing module;

[0025] The visual fingerprint features and the text fingerprint features of the various photovoltaic power station sample images are integrated to construct a fire smoke feature fingerprint library.

[0026] A further technical solution is that the smoke features described by the text description information added to each photovoltaic power station sample image include position, size, coverage area, color feature, optical feature, texture feature, physical and chemical feature, and fire degree feature.

[0027] A further technical solution is that obtaining the plurality of photovoltaic power station sample images containing smoke targets comprises:

[0028] Collecting historical images of the photovoltaic power station containing smoke targets when a fire occurs in the historical operation process of the photovoltaic power station, and preprocessing the historical images by using a defogging algorithm to obtain the photovoltaic power station sample images;

[0029] In addition, the simulation experiment platform is used to simulate the scene when the photovoltaic module generates smoke and obtain the photovoltaic power station sample images.

[0030] A further technical solution is that in the smoke target recognition model:

[0031] The image feature preprocessing module comprises an image encoder, a linear layer and a resampling layer connected in sequence, and the image encoder adopts a vit image encoder in the clip model;

[0032] The text feature preprocessing module comprises a word segmentation encoding module and a vectorization encoding module connected in sequence, and the word segmentation encoding module adopts a BPE word segmentation algorithm;

[0033] The multi-modal large model module adopts an LLaVA multi-modal model trained on a general data set;

[0034] The target position decoder adopts a target detection model DETR based on a Transformer architecture;

[0035] The visual clue decoder adopts an open world segmentation model SAM based on a Transformer architecture.

[0036] The beneficial technical effects of the present application are:

[0037] The application discloses a photovoltaic power station fire warning method fusing visual search and a multi-modal large model.

[0038] The method further combines prior knowledge in a fire smoke feature fingerprint library, compares the suspected smoke target with the fire smoke feature fingerprint library through a KNN unsupervised classification algorithm, effectively solves the false detection problem caused by the multi-modal model not having historical memory, and can further filter out suspected fire smoke targets, so that the accuracy of the smoke target is further improved, and the fire warning accuracy is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a method flowchart of a photovoltaic power station fire warning method of an embodiment of the application.

[0040] Figure 2 is a network structure diagram of a smoke target recognition model built by the application.

[0041] Figure 3 is a structure diagram of a target position decoder in Figure 2

[0042] Figure 4 is a structure diagram of a visual clue decoder in Figure 2

[0043] Figure 5 is a method flowchart of a photovoltaic power station fire warning method of another embodiment of the application. DETAILED DESCRIPTION

[0044] The specific embodiments of the application will be further described below with reference to the accompanying drawings.

[0045] ​​This application discloses a photovoltaic power station fire warning method that integrates visual search and multimodal large models. The photovoltaic power station fire warning method includes the following steps, please refer to Figure 1 The flowchart shown:

[0046] Step 1: Build a smoke target recognition model.

[0047] Please refer to Figure 2 As shown in the model structure, the smoke target recognition model includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multimodal large model module, a target position decoder and a visual clue decoder.

[0048] The image feature preprocessing module and the text feature preprocessing module are each connected to the feature fusion module. The output of the feature fusion module is connected to the multimodal large model module. The output of the multimodal large model module is connected to the object position decoder and the visual cue decoder via two linear layers. The output of the image feature preprocessing module is also connected to the object position decoder and the visual cue decoder. The output of the feature fusion module is also connected to the visual cue decoder.

[0049] The image feature preprocessing module is used to extract image features from the input image, the text feature preprocessing module is used to extract text features from the input text, and the feature fusion module is used to fuse image and text features to generate multimodal fusion features. The multimodal large model module is used to generate a positioning feature identifier based on the multimodal fusion features. The positioning feature identifier is transformed into a target position feature vector using a projection matrix through a linear layer and input into a target position decoder. The positioning feature identifier is transformed into a visual cue feature vector using a projection matrix through another linear layer and input into a visual cue decoder. The target position decoder outputs a smoke target positioning box and confidence score based on the initial image features and the target position feature vector. The visual cue decoder outputs a smoke target search hint heatmap based on the initial image features, the visual cue feature vector, and the multimodal fusion features.

[0050] The smoke target recognition model in this application is directly used and built based on existing model modules, so it can be quickly constructed. The module structure selected for each part includes:

[0051] (1) The image feature preprocessing module includes an image encoder, a linear layer, and a resampling layer connected in sequence.

[0052] The image encoder uses the ViT image encoder in the CLIP model.

[0053] The linear layer is implemented using an MLP multi-layer perceptron, which linearly transforms the high-dimensional visual features output by the image encoder into features suitable for language model processing.

[0054] The resampling layer uses a sampler to downsample high-dimensional visual features to a low-dimensional space through a cross-attention mechanism, which not only reduces the number of features but also highlights important visual features and ignores secondary visual features.

[0055] The combination of the linear layer and the resampling layer is used to flexibly process visual features in different scenarios, balance the calculation efficiency and model performance, and more efficiently process complex visual analysis tasks.

[0056] (2) The text feature preprocessing module includes a word segmentation and encoding module and a vectorization encoding module connected in sequence.

[0057] The word segmentation and encoding module uses the BPE word segmentation algorithm, which regards each byte as a separate vocabulary unit, and then identifies the most frequent byte pair in the text based on statistical analysis and merges it into a new vocabulary unit. This process is repeated until the predetermined vocabulary size is reached or no more byte pairs can be merged. Compared with other word segmentation algorithms, the BPE algorithm gradually constructs a vocabulary by iteratively merging the most common byte pairs in the text to effectively encode the text. On the one hand, it can adaptively capture common phrases and patterns in language, generating more rich and semantic vocabulary. On the other hand, it also provides a more compact and information-intensive text representation for the multi-modal large model module, which helps to improve the model's understanding and processing ability of text information.

[0058] The vectorization encoding module converts the segmented text into text features in the embedding space of the multi-modal large model module through encoding.

[0059] (3) The multi-modal large model module uses the LLaVA multi-modal model trained on general datasets. Specifically, the LLaVA-7B-v1.1 open-source multi-modal model trained on Objects365 v2, COCO-Stuff, LVIS-PACO, and other general detection and segmentation datasets, as well as refCOCO, refCOCO+, ref-COCOg, and other reference detection and segmentation datasets is used.

[0060] The multi-modal large model module is organically stacked and cascaded by a feature embedding sub-module, a self-attention sub-module and a cross-attention sub-module. The feature embedding sub-module is used to convert various modal features into feature vectors in an embedding space. The self-attention sub-module is used to learn the importance of different features in a single module. The cross-attention module is used to learn the coupling representation relationship between image, audio, depth and other modal information and text information, and to realize the feature alignment of image, audio, spatial depth and other modal information and text information. Through learning based on the attention mechanism, the multi-modal large model module can fully utilize its powerful cross-modal information mining capability to output a positioning feature identifier that combines image context semantics and world knowledge. The positioning feature identifier is transformed into a target position feature vector and a visual clue feature vector via two linear layers through a projection matrix.

[0061] (4) The target position decoder adopts the decoder structure of the target detection model DETR based on the Transformer architecture. The target position decoder includes a multi-layer perceptron layer and a cosine similarity calculation module. Please refer to the structure diagram shown in Figure 3 The image feature extracted by the image feature preprocessing module is input into the multi-layer perceptron layer and the cosine similarity calculation module in the target position decoder, and the target position feature vector output by the multi-modal large model module is input into the cosine similarity calculation module in the target position decoder. The multi-layer perceptron layer outputs a smoke target positioning frame, and the cosine similarity calculation module outputs the confidence thereof.

[0062] (5) The visual clue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual clue decoder is a lightweight network component. Please refer to Figure 4 ,

[0063] The multi-modal fusion feature and the visual clue feature vector are taken as positioning features. First, the self-attention layer performs self-attention operation on the positioning features. Then, the query-positioning feature and the image feature are cross-attention operated by the marker-image cross-attention layer. Next, the multi-layer perceptron layer updates each positioning feature point by point. Finally, the image-marker cross-attention layer performs cross-attention from the image feature to the positioning feature. In order to ensure that the visual clue decoder can access key geometric information, when participating in the attention layer operation, the position encoding is added to the image embedding vector, and the original positioning feature containing the position information is also re-added to the image embedding vector. Through this ingenious design, the visual clue decoder prediction head can dynamically and real-time output a segmentation mask, and has ambiguity perception ability, can process ambiguous prompts and generate multiple possible valid masks, and output a smoke target search prompt heat map.

[0064] Step 2, obtain the working video stream of the photovoltaic power station to be monitored and sequentially extract the image frames to be monitored. The working video stream can be obtained by on-site monitoring and collection of the photovoltaic power station, and then an image processing script is used to extract the image frames to be monitored according to a certain time interval.

[0065] After extraction, each image frame to be monitored is input into the image feature preprocessing module in the smoke target recognition model, and a string indicating the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model. Here, the string indicating the recognition requirement can be customized, such as the string "please locate the smoke target in the image", or other similar strings indicating the recognition requirement.

[0066] After the image frames to be monitored and the string are input into the smoke target recognition model as shown in Figure 2 , the image feature preprocessing module extracts the image features from the input image frames to be monitored, and the text feature preprocessing module extracts the text features from the input string. The feature fusion module fuses the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multi-modal fusion features and input them into the multi-modal large model module. The multi-modal large model module outputs the target position feature vector to the target position decoder and the visual clue feature vector to the visual clue decoder through a linear layer. The target position decoder outputs the smoke target positioning box and the confidence based on the target position feature vector and the image features extracted by the image feature preprocessing module. The visual clue decoder outputs the smoke target search prompt heat map based on the target position feature vector, the image features extracted by the image feature preprocessing module, and the multi-modal fusion features output by the feature fusion module. The obtained smoke target search prompt heat map indicates the confidence of different positions containing the smoke target.

[0067] Step 3, since the color feature of the smoke target, especially the early smoke target of a fire, is not obvious, it is easy to miss detection. For the smoke target positioning box output by the smoke target recognition model, when the confidence of the smoke target positioning box reaches the confidence threshold, it is determined that there is a smoke target at the position of the smoke target positioning box, and the smoke target search prompt heat map is no longer considered. However, when the confidence of the smoke target positioning box does not reach the confidence threshold, it is not directly determined that there is no smoke target, but further active search is performed based on the smoke target search prompt heat map using the visual search strategy until a smoke target positioning box reaching the confidence threshold is obtained, or until it is determined that the image frame to be monitored does not contain a smoke target. The confidence threshold can be set by the user, such as 0.7.

[0068] The method of active search includes: dividing the image frame to be monitored into multiple sub-regions, and the division method and granularity can be customized. Then, the confidence level of containing a smoke target in each sub-region is determined based on the smoke target search prompt heat map, and the statistical results of the confidence levels of containing a smoke target in different positions in each sub-region are counted as the confidence level of containing a smoke target in the sub-region. Then, the sub-regions are traversed in order from high to low according to the confidence level of containing a smoke target, that is, the sub-regions with higher confidence levels in the smoke target search prompt heat map are preferentially processed. For each sub-region traversed: input the image of the image frame to be monitored in the sub-region into the image feature preprocessing module in the smoke target recognition model, and input the string for indicating the recognition requirement into the text feature preprocessing module in the smoke target recognition model, to obtain the smoke target positioning box and the confidence level for the sub-region.

[0069] By adopting the active search and dynamic adjustment mechanism described above, the visual search strategy can guide the smoke target recognition model to not only rely on direct visual features, but also combine scene information and common sense reasoning in the image frame to be monitored, fully utilize the multi-modal collaborative reasoning and world knowledge ability of the multi-modal large model, and significantly outperform the traditional multi-modal large model which only relies on static image features.

[0070] Step 4: Obtain the smoke target recognition result according to the smoke target positioning box that reaches the confidence threshold.

[0071] In one embodiment, in order to further avoid false detection and improve the accuracy of smoke target recognition, the pre-constructed fire smoke feature fingerprint library is further combined to further screen and judge the screened smoke target positioning box. Please refer to Figure 5 In the embodiment shown, first, the candidate smoke target at the smoke target positioning box that reaches the confidence threshold screened above is extracted. However, since the original pre-trained multi-modal large model is labeled and trained on a rectangular box surrounding only the target, the context semantic information around the smoke target is ignored, and therefore the final output smoke target positioning box only contains the smoke target. In order to obtain the context semantic information of the smoke target, each smoke target positioning box that reaches the confidence threshold is expanded, and a target local image in the expanded smoke target positioning box in the image frame to be monitored is extracted. At this time, the target local image contains not only the smoke target, but also the surrounding environment of the smoke target.

[0072] Then, the image feature preprocessing module in the smoke target recognition model is used to extract the image features of each target local image, and the smoke target recognition result is obtained based on the similarity between the image features of each target local image and the fingerprint features of the smoke target in the fire smoke feature fingerprint library.

[0073] The fire smoke feature fingerprint library here is pre-constructed, which includes the fingerprint features of different smoke targets. In an embodiment, the fire smoke feature fingerprint library includes the visual fingerprint features and the text fingerprint features of each smoke target. Further, the fire smoke feature fingerprint library includes the fingerprint features of smoke targets of different sizes. The method for pre-constructing the fire smoke feature fingerprint library includes:

[0074] (1) Obtain a plurality of photovoltaic power station sample images containing smoke targets. There are mainly two channels for obtaining the photovoltaic power station sample images: collect historical images of the photovoltaic power station containing smoke targets when a fire occurs during the historical operation of the photovoltaic power station, and use a defogging algorithm to pre-process the historical images to obtain the photovoltaic power station sample images. And, use a simulation experiment platform to simulate the scene when the photovoltaic module generates smoke and obtain the photovoltaic power station sample images, such as setting an overcurrent by a person to make the connecting cable of the photovoltaic module continuously heat and generate smoke. The obtained photovoltaic power station sample images are generally cropped to the same size.

[0075] (2) Label the size of the smoke target in each photovoltaic power station sample image. Use the modified labelimg image labeling software, label the smoke target using a rectangular frame, and use a rectangular frame of a specified size (224x224 pixels) to label and frame the lower left corner of the smoke target labeling frame as the starting point. Then, label the size of the smoke target according to the proportion of the area of the smoke target to the area of the rectangular frame and add the corresponding label, such as small size, medium size and large size.

[0076] (3) Add text description information for describing the smoke features of the smoke target in each photovoltaic power station sample image. Including: using the blip image description model combined with the smoke features of the smoke target to make text description information of the smoke target to describe the smoke features in the form of text. The text description information added to each photovoltaic power station sample image describes the smoke features including the position, size, coverage area, color feature, optical feature, texture feature, physical and chemical feature of the smoke target, and the fire degree feature.

[0077] In one example, the text description information added to one sample image of a photovoltaic power station is: The image shows small-sized opaque early fire smoke emerging from the PV plant scene. The smoke is mixed with the PV plant scene in a small area of the image, meaning that the PV plant may be in the early stage of the fire.

[0078] In another example, the text description information added to another sample image of a photovoltaic power station is: The image shows medium-scale opaque early fire smoke emerging from the PV plant scene. The smoke is mixed with the PV plant scene in a large range of the image, which means that the PV plant is already in the early stage of the fire.

[0079] In yet another example, the text description information added to another sample image of a photovoltaic power station is: The image shows a large amount of opaque fire smoke emerging from the PV plant site, and the smoke almost covers the entire PV plant background, which means that a fire has occurred in the PV plant.

[0080] (4) For each sample image of a photovoltaic power station, first perform normalization processing, then extract the image features of the sample image of the photovoltaic power station as visual fingerprint features using the image feature preprocessing module in the above smoke target recognition model, and extract the text features of the text description information of the sample image of the photovoltaic power station as text fingerprint features using the text feature preprocessing module in the above smoke target recognition model.

[0081] (5) constructing a fire smoke feature fingerprint library by integrating the visual fingerprint features and the text fingerprint features of the sample images of the photovoltaic power station.

[0082] Regardless of how the fire smoke feature fingerprint library is constructed, the similarity between the fingerprint features of the smoke target in the fire smoke feature fingerprint library and the image features of the target local image can be used to further determine whether the target local image contains a smoke target, including:

[0083] First, the similarity between the image features of the target local image and the visual fingerprint features of the smoke target in the fire smoke feature fingerprint library is calculated using the KNN algorithm to obtain a visual classification result, and the similarity between the image features of the target local image and the text fingerprint features of the smoke target in the fire smoke feature fingerprint library is calculated using the KNN algorithm to obtain a text classification result.

[0084] Then, the smoke target recognition result is obtained according to the visual classification result and the text classification result. For each target local image, when the visual classification result indicates that the image features of the target local image do not belong to any visual fingerprint feature classification, and the text classification result indicates that the image features of the target local image do not belong to any text fingerprint feature classification, it is determined that the target local image does not include a smoke target, otherwise it is determined that the target local image contains a smoke target. Finally, the smoke target recognition result is obtained by integrating all target local images in the image frame to be monitored.

[0085] As described above, the fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes, so in another embodiment, in addition to being able to determine whether the target local image contains a smoke target according to the visual classification result and the text classification result, the size of the contained smoke target can also be obtained: for each target local image, the size of the smoke target corresponding to the fingerprint features in the visual fingerprint feature classification and the text fingerprint feature classification to which the image features of the target local image belong is determined, and the size of the smoke target contained in the target local image is determined according to the majority rule.

[0086] Step 5, obtaining a fire warning result for the photovoltaic power station to be monitored according to the smoke target recognition result. As described above, the smoke target recognition result obtained through the above identification detection can indicate the position and size of the smoke target, so the corresponding fire class can be determined according to the content of the smoke target recognition result and the corresponding fire warning result is output. The larger the size of the smoke target indicated by the smoke target recognition result, the higher the corresponding fire class, and the more important the position of the smoke target indicated by the smoke target recognition result, the higher the corresponding fire class. The fire class corresponding to different smoke target recognition results and the fire warning result can be set by the user.

[0087] For example, when the smoke target recognition result indicates that a small size smoke target is recognized, it is determined that there is a low level fire, such as popping up the text "Early smoke situation of fire on site, please send someone to verify immediately!" in the video screen, and sending the screen and text to the WeChat of the operation and maintenance personnel. For example, when the smoke target recognition result indicates that a medium size smoke target is recognized, an intermediate level fire warning result is output, such as popping up the text "More serious fire smoke situation on site, please send someone to verify immediately!" in the video screen. For example, when the smoke target recognition result indicates that a medium size smoke target is recognized, it is determined that there is a medium level fire, such as popping up the text "More serious fire smoke situation on site, please send someone to verify immediately!" in the video screen. For example, when the smoke target recognition result indicates that a large size smoke target is recognized, it is determined that there is a high level fire, and the text "Serious fire smoke situation on site, please handle immediately!" is popped up in the video screen. At this time, the system will send the screen and text to the WeChat of the operation and maintenance personnel, and start the fire alarm.

[0088] In one verification instance, when only the smoke target positioning frame output by the smoke target recognition model reaching the confidence threshold is used as the smoke target recognition result, the prediction accuracy is only 78.12%, because the smoke target recognition model only uses the pre-trained features of the entire image when extracting image features, without dynamic adjustment according to target search, resulting in similar illusion defects as large language models. Therefore, the smoke target recognition model has low prediction accuracy when performing zero-shot prediction without training, and cannot be applied to fire warning in photovoltaic power stations.

[0089] However, after further utilizing the visual search strategy combined with the smoke target search prompt heat map for active search, due to the introduction of active search mechanism, visual working memory and dynamic adjustment strategy, by simulating the recognition mechanism of human beings to the recognized target, the world knowledge and context understanding ability of the smoke target recognition model can be fully utilized, and the accuracy of visual positioning and multi-modal collaborative reasoning ability are improved. Therefore, the prediction accuracy can be improved to 93.27%.

[0090] However, at this time, it is still difficult to effectively distinguish between smoke and non-smoke targets. In order to further improve the recognition rate and reduce the false alarm rate of the warning system, further combining the prior knowledge in the fire smoke feature fingerprint library, the suspected smoke target is compared with the fire smoke feature fingerprint library through KNN unsupervised classification algorithm, effectively solving the misidentification problem caused by the lack of historical memory of multi-modal model, effectively filtering out suspected fire smoke targets, and further improving the accuracy, so the prediction accuracy can be further improved to 96.58%.

[0091] The above merely describes the preferred embodiments of the present application, and the present application is not limited to the above embodiments. It can be understood that other improvements and changes directly derived or thought by those skilled in the art without departing from the spirit and concept of the present application shall be considered to be included in the protection scope of the present application.

Claims

1. A photovoltaic power station fire warning method integrating visual search and multimodal large model, characterized by: The photovoltaic power station fire early warning method includes: Building a smoke target recognition model, which includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multimodal large model module, a target position decoder, and a visual cue decoder; Obtain a working video stream of the photovoltaic power station to be monitored and extract the image frames to be monitored in sequence, input each image frame to be monitored into the image feature preprocessing module in the smoke target recognition model, input a character string for indicating recognition requirements into the text feature preprocessing module in the smoke target recognition model, the feature fusion module fuses the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module, and inputs the fuses into the multimodal large model module, the multimodal large model module outputs a target position feature vector to the target position decoder through a linear layer, the multimodal large model module also outputs a visual cue feature vector to the visual cue decoder through a linear layer, the target position decoder outputs a smoke target positioning frame and confidence based on the target position feature vector and the image features extracted by the image feature preprocessing module, the visual cue decoder outputs a smoke target search prompt heat map based on the target position feature vector, the image features extracted by the image feature preprocessing module, and the multimodal fusion features output by the feature fusion module; the smoke target search prompt heat map indicates the confidence level of smoke targets at different positions; When the confidence of the smoke target positioning frame does not reach the confidence threshold, the visual search strategy is used to actively search based on the smoke target search prompt heat map until a smoke target positioning frame that reaches the confidence threshold is obtained, or until it is determined that the image frame to be monitored does not contain a smoke target; A smoke target recognition result is obtained according to the smoke target positioning frame that reaches the confidence threshold, and a fire warning result for the photovoltaic power station to be monitored is obtained according to the smoke target recognition result.

2. The photovoltaic power station fire early warning method according to claim 1, characterized in that: Active search based on smoke target search hint heat map using visual search strategy includes: The image frame to be monitored is divided into multiple sub-areas, and the confidence level of each sub-area containing a smoke target is determined based on the smoke target search prompt heat map. The sub-areas are traversed in order from high to low confidence level of the smoke target. For each traversed sub-area: the image of the image frame to be monitored in the sub-area is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model to obtain the smoke target positioning box and confidence level for the sub-area.

3. The photovoltaic power station fire early warning method according to claim 1, characterized in that: Obtaining a smoke target recognition result based on the smoke target positioning frame that reaches the confidence threshold includes: After enlarging each smoke target positioning frame that reaches the confidence threshold, extracting the target local image of the image frame to be monitored within the enlarged smoke target positioning frame; The image feature preprocessing module is used to extract the image features of each target local image, and the smoke target recognition result is obtained based on the similarity between the image features of each target local image and the fingerprint features of the smoke targets in the fire smoke feature fingerprint library.

4. The photovoltaic power station fire early warning method according to claim 3, characterized in that: The obtaining of the smoke target recognition result based on the similarity between the image feature of the target local image and the fingerprint feature of the smoke target in the fire smoke feature fingerprint library includes: Calculating the similarity between the image features of the target partial image and the visual fingerprint features of the smoke target in the fire smoke feature fingerprint library using the KNN algorithm and obtaining a visual classification result, and calculating the similarity between the image features of the target partial image and the text fingerprint features of the smoke target in the fire smoke feature fingerprint library using the KNN algorithm and obtaining a text classification result; The smoke target recognition result is obtained according to the visual classification result and the text classification result.

5. The photovoltaic power station fire early warning method according to claim 4, characterized in that: Obtaining the smoke target recognition result according to the visual classification result and the text classification result includes: For each target local image, when the visual classification result indicates that the image features of the target local image do not belong to any visual fingerprint feature classification, and the text classification result indicates that the image features of the target local image do not belong to any text fingerprint feature classification, it is determined that the target local image does not include a smoke target; otherwise, it is determined that the target local image contains a smoke target; the smoke target recognition result is obtained by comprehensively analyzing all target local images in the image frame to be monitored.

6. The photovoltaic power station fire early warning method according to claim 5, characterized in that: The fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes. The smoke target recognition result obtained based on the visual classification result and the text classification result also includes: For each target partial image, determine the size of the smoke target corresponding to the fingerprint feature in the visual fingerprint feature classification to which the image feature of the target partial image belongs and the text fingerprint feature classification, and determine the size of the smoke target contained in the target partial image according to the majority voting principle.

7. The photovoltaic power station fire early warning method according to claim 4, characterized in that: The photovoltaic power station fire early warning method further includes: Acquire several sample images of photovoltaic power stations containing smoke targets, mark the size of the smoke targets in each sample image of the photovoltaic power station, and add text description information for describing smoke characteristics of the smoke targets in the sample image of the photovoltaic power station to each sample image of the photovoltaic power station; For each photovoltaic power station sample image, an image feature preprocessing module is used to extract image features of the photovoltaic power station sample image as visual fingerprint features, and a text feature preprocessing module is used to extract text features of text description information of the photovoltaic power station sample image as text fingerprint features; The fire smoke feature fingerprint library is constructed by integrating the visual fingerprint features and text fingerprint features of sample images of various photovoltaic power stations.

8. The photovoltaic power station fire early warning method according to claim 7, characterized in that: The text description information added to each PV power station sample image is used to describe the smoke characteristics including the location, size, coverage area, color characteristics, optical characteristics, texture characteristics, physical and chemical characteristics of the smoke target and the characteristics representing the degree of fire.

9. The photovoltaic power station fire early warning method according to claim 7, characterized in that: The acquiring of a plurality of sample images of a photovoltaic power station containing smoke targets comprises: Collect historical images of a photovoltaic power station containing smoke targets when a fire occurred during its historical operation, and use a defogging algorithm to preprocess the historical images to obtain a sample image of the photovoltaic power station; In addition, the simulation experiment platform is used to simulate the scene when photovoltaic modules generate smoke and obtain sample images of photovoltaic power stations.

10. The photovoltaic power station fire early warning method according to claim 1, characterized in that: In the smoke target recognition model: The image feature preprocessing module includes an image encoder, a linear layer, and a resampling layer connected in sequence. The image encoder adopts the ViT image encoder in the CLIP model. The text feature preprocessing module includes a word segmentation encoding module and a vectorization encoding module connected in sequence. The word segmentation encoding module adopts the BPE word segmentation algorithm; The multimodal large model module uses the LLaVA multimodal model trained on a general dataset; The target position decoder adopts the decoder structure of the target detection model DETR based on the Transformer architecture; The visual cue decoder adopts the decoder structure of the open world segmentation model SAM based on the Transformer architecture.

Images