Photovoltaic power station fire early warning method fusing visual search and multi-mode large model

Abstract

The invention discloses a photovoltaic power station fire early warning method fusing visual search and a multi-mode large model, and relates to the field of fire early warning. According to the method, a smoke target recognition model built based on a multi-modal large model is utilized to perform preliminary recognition and positioning on a to-be-monitored image frame to obtain a smoke target positioning frame, a confidence coefficient and a smoke target search prompt heat map, and when the confidence coefficient of the smoke target positioning frame does not reach a confidence coefficient threshold value, the smoke target search prompt heat map is searched; furthermore, a visual search strategy is utilized to perform active search based on the smoke target search prompt heat map, an active search mechanism, a visual work memory and a dynamic adjustment strategy are introduced, and world knowledge and context understanding ability of a smoke target identification model can be fully utilized by simulating a smoke target identification mechanism of human beings; according to the method, reasoning can be directly carried out in an environment without training, high identification and positioning accuracy of a smoke target is achieved, early fire warning of the photovoltaic power station is effectively achieved, and the implementation difficulty is reduced.

Description

Technical Field

[0001] The present application relates to the field of fire warning, and in particular to a photovoltaic power station fire warning method integrating visual search and multimodal large models. Background Art

[0002] Photovoltaic power generation is an important and promising branch in the field of renewable energy power generation, and has received great attention and importance from the country. However, after a photovoltaic power station has been in operation for a long time, the photovoltaic components are prone to internal problems such as material aging and grid breakage. In addition, large photovoltaic power stations are often built outdoors, so photovoltaic components are also easily affected by external environments such as high temperatures and sandstorms. Photovoltaic components are prone to overheating under internal and external influences, leading to fires and threatening the safe and stable operation of equipment in new power systems.

[0003] Smoke is usually an early sign of a fire, so accurate identification of smoke can assist the operation and maintenance personnel of photovoltaic power stations in carrying out equipment fire safety work, minimize the impact of fire and ensure the safety of the operation and maintenance personnel. With the development of computer vision technology, the use of video to realize fire smoke early warning monitoring has gradually been applied in many photovoltaic power stations. However, smoke itself is not easy to detect in the video, and the working environment of photovoltaic power stations is complex, so the phenomenon of false alarms and missed alarms is relatively serious, and the accuracy of early warning is not high. Summary of the invention

[0004] In response to the above problems and technical requirements, this application proposes a photovoltaic power station fire warning method that integrates visual search and multimodal large models. The technical solution of this application is as follows:

[0005] A photovoltaic power station fire early warning method integrating visual search and multimodal large model, the photovoltaic power station fire early warning method comprising:

[0006] Build a smoke target recognition model, which includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multimodal large model module, a target position decoder, and a visual cue decoder;

[0007] The working video stream of the photovoltaic power station to be monitored is obtained and the image frames to be monitored are extracted in sequence, each image frame to be monitored is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model. The feature fusion module fuses the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module and inputs them into the multimodal large model module. The multimodal large model module outputs the target position feature vector to the target position decoder through the linear layer. The multimodal large model module also outputs the visual clue feature vector to the visual clue decoder through the linear layer. The target position decoder outputs the smoke target positioning frame and confidence based on the target position feature vector and the image features extracted by the image feature preprocessing module. The visual clue decoder outputs the smoke target search prompt heat map based on the clue position feature vector, the image features extracted by the image feature preprocessing module and the multimodal fusion features output by the feature fusion module; the smoke target search prompt heat map indicates the confidence that different positions contain smoke targets;

[0008] When the confidence of the smoke target positioning frame does not reach the confidence threshold, the smoke target search prompt heat map is actively searched using the visual search strategy until a smoke target positioning frame that reaches the confidence threshold is obtained, or until it is determined that the image frame to be monitored does not contain a smoke target;

[0009] A smoke target recognition result is obtained according to the smoke target positioning frame that reaches the confidence threshold, and a fire warning result of the photovoltaic power station to be monitored is obtained according to the smoke target recognition result.

[0010] A further technical solution is to use a visual search strategy to actively search based on a smoke target search prompt heat map, including:

[0011] The image frame to be monitored is divided into multiple sub-areas, and the confidence of each sub-area containing smoke targets is determined based on the smoke target search prompt heat map, and each sub-area is traversed in order from high to low confidence of containing smoke targets. For each traversed sub-area: the image of the image frame to be monitored in the sub-area is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model to obtain the smoke target positioning box and confidence for the sub-area.

[0012] A further technical solution is that the smoke target recognition result is obtained according to the smoke target positioning frame that reaches the confidence threshold, including:

[0013] After enlarging each smoke target positioning frame that reaches the confidence threshold, extract the target local image of the image frame to be monitored within the enlarged smoke target positioning frame;

[0014] The image feature preprocessing module is used to extract the image features of each target local image, and the smoke target recognition result is obtained based on the similarity between the image features of each target local image and the fingerprint features of the smoke target in the fire smoke feature fingerprint library.

[0015] A further technical solution is that the smoke target recognition result is obtained based on the similarity between the image features of the target local image and the fingerprint features of the smoke target in the fire smoke feature fingerprint library, including:

[0016] The similarity between the image features of the local target image and the visual fingerprint features of the smoke target in the fire smoke feature fingerprint library is calculated using the KNN algorithm to obtain a visual classification result, and the similarity between the image features of the local target image and the text fingerprint features of the smoke target in the fire smoke feature fingerprint library is calculated using the KNN algorithm to obtain a text classification result;

[0017] The smoke target recognition results are obtained based on the visual classification results and text classification results.

[0018] Its further technical solution is to obtain smoke target recognition results based on visual classification results and text classification results, including:

[0019] For each target local image, when the visual classification result indicates that the image features of the target local image do not belong to any visual fingerprint feature classification, and the text classification result indicates that the image features of the target local image do not belong to any text fingerprint feature classification, it is determined that the target local image does not include a smoke target; otherwise, it is determined that the target local image contains a smoke target; all target local images in the image frame to be monitored are combined to obtain the smoke target recognition result.

[0020] A further technical solution is that the fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes, and the smoke target recognition result obtained according to the visual classification result and the text classification result also includes:

[0021] For each target partial image, determine the size of the smoke target corresponding to the visual fingerprint feature classification to which the image feature of the target partial image belongs and the fingerprint feature in the text fingerprint feature classification, and determine the size of the smoke target contained in the target partial image according to the voting principle of minority obeys majority.

[0022] A further technical solution is that the photovoltaic power station fire warning method further includes:

[0023] Acquire a number of sample images of photovoltaic power stations containing smoke targets, annotate the size of the smoke targets in each sample image of the photovoltaic power station, and add text description information for describing the smoke characteristics of the smoke targets in the sample image of the photovoltaic power station to each sample image of the photovoltaic power station;

[0024] For each photovoltaic power station sample image, the image feature preprocessing module is used to extract the image feature of the photovoltaic power station sample image as the visual fingerprint feature, and the text feature preprocessing module is used to extract the text feature of the text description information of the photovoltaic power station sample image as the text fingerprint feature;

[0025] The fire smoke feature fingerprint library is constructed by integrating the visual fingerprint features and text fingerprint features of sample images of various photovoltaic power stations.

[0026] A further technical solution is to add text description information to each photovoltaic power station sample image to describe the smoke characteristics including the location, size, coverage area, color characteristics, optical characteristics, texture characteristics, physical and chemical characteristics of the smoke target and the characteristics representing the degree of fire.

[0027] A further technical solution is to obtain a number of sample images of photovoltaic power stations containing smoke targets, including:

[0028] Collect historical images of photovoltaic power stations containing smoke targets when fires occurred during historical operation, and use a defogging algorithm to preprocess the historical images to obtain sample images of photovoltaic power stations;

[0029] In addition, the simulation experiment platform is used to simulate the scene when photovoltaic modules produce smoke and obtain sample images of photovoltaic power stations.

[0030] A further technical solution is that in the smoke target recognition model:

[0031] The image feature preprocessing module includes an image encoder, a linear layer, and a resampling layer connected in sequence. The image encoder adopts the vit image encoder in the clip model.

[0032] The text feature preprocessing module includes a word segmentation encoding module and a vectorization encoding module connected in sequence. The word segmentation encoding module adopts the BPE word segmentation algorithm;

[0033] The multimodal large model module uses the LLaVA multimodal model trained on a common dataset;

[0034] The target position decoder uses the target detection model DETR based on the Transformer architecture;

[0035] The visual cue decoder adopts the open-world segmentation model SAM based on the Transformer architecture.

[0036] The beneficial technical effects of this application are:

[0037] The present application discloses a photovoltaic power station fire warning method integrating visual search and multimodal large model. The photovoltaic power station fire warning method uses a smoke target recognition model built based on a multimodal large model to perform preliminary recognition and positioning of a monitored image frame to obtain a smoke target positioning frame and a confidence level as well as a smoke target search prompt heat map. When the confidence level of the smoke target positioning frame does not reach a confidence threshold, a visual search strategy is further used to perform active search based on the smoke target search prompt heat map. An active search mechanism, visual working memory and a dynamic adjustment strategy are introduced. By simulating the human recognition mechanism of smoke targets, the world knowledge and context understanding ability of the smoke target recognition model can be fully utilized, so that the method can use the world knowledge and cross-modal ability learned by the multimodal large language model to directly perform reasoning in an environment without training, thereby solving the problem that the current data-driven fire smoke warning method requires a large amount of data training and also has a high visual positioning accuracy and multimodal collaborative reasoning ability. Therefore, the method has higher recognition and positioning accuracy for smoke targets, high warning accuracy and reduced implementation difficulty.

[0038] This method further combines the prior knowledge in the fire smoke feature fingerprint library and compares the similarity between the suspected smoke targets and the fire smoke feature fingerprint library through the KNN unsupervised classification algorithm, which effectively solves the false detection problem caused by the lack of historical memory of the multimodal model. It can further filter out the suspected fire smoke targets and further improve the accuracy of the smoke targets, thereby further improving the accuracy of fire warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a method flow chart of a photovoltaic power station fire warning method according to an embodiment of the present application.

[0040] Figure 2 This is the network structure diagram of the smoke target recognition model built in this application.

[0041] Figure 3 yes Figure 2 Schematic diagram of the structure of the target position decoder in .

[0042] Figure 4 yes Figure 2 Schematic diagram of the structure of the visual cue decoder in .

[0043] Figure 5 This is a method flow chart of a photovoltaic power station fire warning method according to another embodiment of the present application. DETAILED DESCRIPTION

[0044] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0045] This application discloses a photovoltaic power station fire warning method that integrates visual search and multimodal large models. The photovoltaic power station fire warning method includes the following steps, please refer to Figure 1 The flowchart shown:

[0046] Step 1: Build a smoke target recognition model.

[0047] Please refer to Figure 2 As shown in the model structure, the smoke target recognition model includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multimodal large model module, a target position decoder and a visual clue decoder.

[0048] The image feature preprocessing module and the text feature preprocessing module are connected to the feature fusion module respectively. The output of the feature fusion module is connected to the multimodal large model module. The output of the multimodal large model module is connected to the target position decoder and the visual clue decoder via two linear layers. The output of the image feature preprocessing module is also connected to the target position decoder and the visual clue decoder respectively. The output of the feature fusion module is also connected to the visual clue decoder.

[0049] The image feature preprocessing module is used to extract image features from the input image, the text feature preprocessing module is used to extract text features from the input text, and the feature fusion module is used to fuse image features and text features to obtain multimodal fusion features. The multimodal large model module is used to obtain a positioning feature identifier based on the multimodal fusion features. The positioning feature identifier is transformed into a target position feature vector through a projection matrix via a linear layer and input into the target position decoder. The positioning feature identifier is transformed into a visual cue feature vector through a projection matrix via another linear layer and input into the visual cue decoder. The target position decoder outputs the smoke target positioning box and confidence based on the initial image features and the target position feature vector. The visual cue decoder outputs a smoke target search prompt heat map based on the initial image features, the visual cue feature vector and the multimodal fusion features.

[0050] The smoke target recognition model of this application is directly used and built based on the existing model modules, so it can be quickly constructed. The module structure selected for each part includes:

[0051] (1) The image feature preprocessing module includes an image encoder, a linear layer, and a resampling layer connected in sequence.

[0052] The image encoder uses the ViT image encoder in the CLIP model.

[0053] The linear layer is implemented using an MLP multi-layer perceptron, which linearly transforms the high-dimensional visual features output by the image encoder into features suitable for language model processing.

[0054] The resampling layer uses a sampler to downsample high-dimensional visual features to a low-dimensional space through a cross-attention mechanism, which not only reduces the number of features but also highlights important visual features and ignores minor visual features.

[0055] The combination of linear layers and resampling layers can flexibly process visual features in different scenarios, balance computational efficiency and model performance, and process complex visual analysis tasks more efficiently.

[0056] (2) The text feature preprocessing module includes a word segmentation encoding module and a vectorization encoding module which are connected in sequence.

[0057] The word segmentation encoding module adopts the BPE word segmentation algorithm, which treats each byte as a separate vocabulary unit, and then identifies the byte pairs that appear most frequently in the text based on statistical analysis and merges them into a new vocabulary unit. This process is repeated until the predetermined vocabulary size is reached or there are no more byte pairs to be merged. Compared with other word segmentation algorithms, this BPE algorithm gradually builds a vocabulary by iteratively merging the most common byte pairs in the text to achieve effective encoding of the text. On the one hand, it can adaptively capture common phrases and patterns in the language and generate richer and more semantically informative vocabulary. On the other hand, it can also provide a more compact and information-intensive text representation method for the multimodal large model module, which helps to improve the model's understanding and processing capabilities of text information.

[0058] The vectorized encoding module converts the segmented text into text features in the embedding space of the multimodal large model module by encoding.

[0059] (3) The multimodal large model module uses the LLaVA multimodal model trained on general datasets. Specifically, it uses the LLaVA-7B-v1.1 open source multimodal model trained on general detection and segmentation datasets such as Objects365 v2, COCO-Stuff, and LVIS-PACO, as well as reference detection and segmentation datasets such as refCOCO, refCOCO+, and ref-COCOg.

[0060] The multimodal large model module is composed of a feature embedding submodule, a self-attention submodule and a cross-attention submodule, which are organically stacked and cascaded. Among them, the feature embedding submodule is used to convert various modal features into feature vectors of the embedding space. The self-attention submodule is used to learn the importance of different features in a single module. The cross-attention module is used to learn the coupling representation relationship between modal information such as image, audio, depth and text information, and realize the feature alignment of modal information such as image, audio, spatial depth and text information. Through learning based on the attention mechanism, the multimodal large model module can make full use of its powerful cross-modal information mining capabilities and output a positioning feature identifier that combines image context semantics and world knowledge. The positioning feature identifier is transformed into a target position feature vector and a visual clue feature vector through a projection matrix via two linear layers.

[0061] (4) The target position decoder adopts the decoder structure of the target detection model DETR based on the Transformer architecture. The target position decoder includes a multi-layer perceptron layer and a cosine similarity calculation module. Please refer to Figure 3 As shown in the structural diagram, the image features extracted by the image feature preprocessing module are input into the multi-layer perceptron layer and the cosine similarity calculation module in the target position decoder, and the target position feature vector output by the multimodal large model module is input into the cosine similarity calculation module in the target position decoder. The multi-layer perceptron layer outputs the smoke target positioning frame, and the cosine similarity calculation module outputs its confidence.

[0062] (5) The visual cue decoder uses the decoder structure of the open world segmentation model SAM based on the Transformer architecture. The visual cue decoder is a lightweight network component. Please refer to Figure 4 ,

[0063] The multimodal fusion feature and the visual cue feature vector are used as positioning features. First, the self-attention layer performs self-attention on the positioning features. Then, the label-image cross-attention layer performs cross-attention on the query positioning features and the image features. Then, the multi-layer perceptron layer updates each positioning feature point by point. Finally, the image-label cross-attention layer performs cross-attention from the image features to the positioning features. In order to ensure that the visual cue decoder can access key geometric information, when participating in the attention layer operation, the position encoding will be added to the image embedding vector, and the entire original positioning feature containing the position information will also be re-added to the image embedding vector. Through this clever design, the visual cue decoder prediction head can dynamically and real-time output segmentation masks, and has ambiguous perception capabilities, can handle ambiguous prompts and generate multiple possible valid masks, and output smoke target search prompt heat maps.

[0064] Step 2, obtain the working video stream of the photovoltaic power station to be monitored and extract the image frames to be monitored in sequence. The working video stream can be collected by on-site monitoring of the photovoltaic power station, and then use the image processing script to extract the image frames to be monitored according to a certain time interval.

[0065] After extraction, each image frame to be monitored is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model. The character string used to indicate the recognition requirement can be customized, such as the character string "Please locate the smoke target in the image", or other similar character strings that can indicate the recognition requirement.

[0066] The image frame and string input to be monitored are as follows: Figure 2 After the smoke target recognition model shown in the figure is constructed, the image feature preprocessing module extracts features from the input image frame to be monitored to obtain image features, and the text feature preprocessing module extracts features from the input character string to obtain text features. The feature fusion module fuses the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module to obtain multimodal fusion features and inputs them into the multimodal large model module. The multimodal large model module outputs the target position feature vector to the target position decoder and the visual clue feature vector to the visual clue decoder through the linear layer. The target position decoder outputs the smoke target positioning frame and confidence based on the target position feature vector and the image features extracted by the image feature preprocessing module. The visual clue decoder outputs the smoke target search prompt heat map based on the target position feature vector, the image features extracted by the image feature preprocessing module and the multimodal fusion features output by the feature fusion module. The obtained smoke target search prompt heat map indicates the confidence that different positions contain smoke targets.

[0067] Step 3: Since the color features of smoke targets, especially those in the early stages of a fire, are not obvious, it is easy to miss detection. For the smoke target positioning frame output by the smoke target recognition model, when the confidence of the smoke target positioning frame reaches the confidence threshold, it is determined that there is a smoke target at the location of the smoke target positioning frame, and the smoke target search prompt heat map is no longer considered. However, when the confidence of the smoke target positioning frame does not reach the confidence threshold, it is not directly determined that there is no smoke target, but the visual search strategy is further used to actively search based on the smoke target search prompt heat map until a smoke target positioning frame that reaches the confidence threshold is obtained, or until it is determined that the image frame to be monitored does not contain a smoke target. Among them, the confidence threshold can be customized, for example, it is generally set to 0.7.

[0068] The active search method includes: dividing the image frame to be monitored into multiple sub-areas, and the division method and granularity can be customized. Then, based on the smoke target search prompt heat map, the confidence that each sub-area contains a smoke target is determined, and the statistical results of the confidence that different positions in each sub-area contain smoke targets can be counted as the confidence that the sub-area contains smoke targets. Then, each sub-area is traversed in order from high to low in terms of the confidence that the smoke target is contained, that is, the sub-areas with higher confidence in the smoke target search prompt heat map are prioritized. For each traversed sub-area: the image of the image frame to be monitored in the sub-area is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model to obtain the smoke target positioning frame and confidence for the sub-area.

[0069] By adopting the above-mentioned active search and dynamic adjustment mechanism, the visual search strategy can guide the smoke target recognition model to rely not only on direct visual features, but also on scene information and common sense reasoning in the image frames to be monitored, making full use of the multimodal collaborative reasoning and world knowledge capabilities of the multimodal large model, which is significantly better than the traditional multimodal large model that only relies on static image features.

[0070] Step 4: Obtain the smoke target recognition result according to the smoke target positioning frame that reaches the confidence threshold.

[0071] In one embodiment, in order to further avoid false detection and improve the accuracy of smoke target recognition, the pre-built fire smoke feature fingerprint library is further combined to further screen and judge the selected smoke target positioning frame. Figure 5 In the embodiment shown, the candidate smoke targets at the smoke target positioning frame that has reached the confidence threshold and has been screened out as above are first extracted. However, since the original pre-trained multimodal large model is annotated and trained on a rectangular frame that only surrounds the target, the contextual semantic information around the smoke target is ignored, so the smoke target positioning frame that is finally output only contains the smoke target. In order to obtain the contextual semantic information of the smoke target, each smoke target positioning frame that has reached the confidence threshold is enlarged, and the target local image of the image frame to be monitored in the enlarged smoke target positioning frame is extracted. At this time, the target local image obtained not only contains the smoke target, but also includes the surrounding environment of the smoke target.

[0072] Then, the image feature preprocessing module in the above-mentioned smoke target recognition model is used to extract the image features of each target local image, and the smoke target recognition result is obtained based on the similarity between the image features of each target local image and the fingerprint features of the smoke target in the fire smoke feature fingerprint library.

[0073] The fire smoke feature fingerprint library is pre-built, including fingerprint features of different smoke targets. In one embodiment, the fire smoke feature fingerprint library includes visual fingerprint features and text fingerprint features of each smoke target. Furthermore, the fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes. The method of pre-building the fire smoke feature fingerprint library includes:

[0074] (1) First, obtain several sample images of photovoltaic power stations containing smoke targets. There are two main channels for obtaining sample images of photovoltaic power stations: collect historical images of photovoltaic power stations when fires occurred during historical operation and contained smoke targets, and use a defogging algorithm to preprocess the historical images to obtain sample images of photovoltaic power stations. Also, use a simulation experiment platform to simulate the scene when photovoltaic components produce smoke and obtain sample images of photovoltaic power stations. For example, by artificially setting overcurrent to make the connecting cables of photovoltaic components continue to heat up and produce smoke. The obtained sample images of photovoltaic power stations are generally cropped to the same size.

[0075] (2) Label the size of the smoke target in each PV power station sample image. Use the modified labelimg or other image labeling software to label the smoke target with a rectangular frame. At the same time, use a labeling frame of a specified size (224×224 pixels) and use the lower left corner of the smoke target labeling frame as the starting point for labeling and selection. Then, label the size of the smoke target according to the ratio of the smoke target area to the rectangular frame area and add corresponding labels, such as small, medium, and large.

[0076] (3) Text description information is added to each photovoltaic power station sample image to describe the smoke characteristics of the smoke target in the photovoltaic power station sample image. This includes: using the blip image description model and combining the smoke characteristics of the smoke target to produce text description information of the smoke target to describe the smoke characteristics in a textual manner. The text description information added to each photovoltaic power station sample image is used to describe the smoke characteristics including the location, size, coverage area, color characteristics, optical characteristics, texture characteristics, physical and chemical characteristics of the smoke target and the characteristics of the fire degree.

[0077] In one example, the text description information added to a sample image of a photovoltaic power station is: The imageshows small-sized opaque early fire smoke emerging from the PV plant scene. The smoke is mixed with the PV plant scene in a small area of ​​the image, meaning that the PV plant may be in the early stage of the fire.

[0078] In another example, the text description information added to another sample image of a photovoltaic power station is: The image shows medium-scale opaque early fire smoke emerging from the PV plant scene. The smoke is mixed with the PV plant scene in a large range of the image, which means that the PV plant is already in the early stage of the fire.

[0079] In yet another example, the text description information added to another sample image of a photovoltaic power plant is: The image shows a large amount of opaque fire smoke emerging from the PV plantsite, and the smoke almost covers the entire PV plant background, which means that a fire has occurred in the PV plant.

[0080] (4) For each photovoltaic power station sample image, normalization processing is first performed, and then the image features of the photovoltaic power station sample image are extracted as visual fingerprint features using the image feature preprocessing module in the above-mentioned smoke target recognition model, and the text features of the text description information of the photovoltaic power station sample image are extracted as text fingerprint features using the text feature preprocessing module in the above-mentioned smoke target recognition model.

[0081] (5) The fire smoke feature fingerprint library is constructed by integrating the visual fingerprint features and text fingerprint features of the sample images of each photovoltaic power station.

[0082] Regardless of how the fire smoke feature fingerprint library is specifically constructed, the similarity between the fingerprint features of the smoke target in the fire smoke feature fingerprint library and the image features of the target local image can be used to further determine whether the target local image contains a smoke target, including:

[0083] Firstly, the KNN algorithm is used to calculate the similarity between the image features of the target local image and the visual fingerprint features of the smoke target in the fire smoke feature fingerprint library, and the visual classification result is obtained. Also, the KNN algorithm is used to calculate the similarity between the image features of the target local image and the text fingerprint features of the smoke target in the fire smoke feature fingerprint library, and the text classification result is obtained.

[0084] Then, the smoke target recognition result is obtained based on the visual classification result and the text classification result. For each target partial image, when the visual classification result indicates that the image feature of the target partial image does not belong to any visual fingerprint feature classification, and the text classification result indicates that the image feature of the target partial image does not belong to any text fingerprint feature classification, it is determined that the target partial image does not include the smoke target, otherwise it is determined that the target partial image contains the smoke target. Finally, all the target partial images in the image frame to be monitored are combined to obtain the smoke target recognition result.

[0085] As described above, the fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes. In another embodiment, in addition to determining whether the target local image contains a smoke target, the size of the contained smoke target can also be obtained based on the visual classification results and the text classification results: for each target local image, determine the size of the smoke target corresponding to the fingerprint feature in the visual fingerprint feature classification to which the image feature of the target local image belongs and the text fingerprint feature classification, and determine the size of the smoke target contained in the target local image according to the majority voting principle.

[0086] Step 5, obtain the fire warning result of the photovoltaic power station to be monitored according to the smoke target recognition result. As mentioned above, the smoke target recognition result obtained through the above recognition and detection can indicate the position and size of the smoke target, and the corresponding fire level can be determined according to the content of the smoke target recognition result and the corresponding fire warning result can be output. The larger the size of the smoke target indicated by the smoke target recognition result, the higher the corresponding fire level; the higher the importance of the position of the smoke target indicated by the smoke target recognition result, the higher the corresponding fire level; the fire level and fire warning results corresponding to different smoke target recognition results can be customized.

[0087] For example, when the smoke target recognition result indicates that a small-sized smoke target is identified, it is determined that there is a low-level fire, such as a text pop-up on the video screen "Early smoke of fire appears on the scene, please send someone to verify immediately!", and the picture and text are sent to the WeChat of the operation and maintenance personnel. For another example, when the smoke target recognition result indicates that a medium-sized smoke target is identified, a medium-level fire warning result is output, such as a text pop-up on the video screen "Serious fire smoke appears on the scene, please send someone to verify immediately!", and the picture and text are sent to the WeChat of the operation and maintenance personnel. For another example, when the smoke target recognition result indicates that a medium-sized smoke target is identified, it is determined that there is a medium-level fire, such as a text pop-up on the video screen "Serious fire smoke appears on the scene, please send someone to verify immediately!". For another example, when the smoke target recognition result indicates that a large-sized smoke target is identified, it is determined that there is a high-level fire, and a text pop-up on the video screen "Serious fire smoke appears on the scene, please deal with it immediately!", at this time, the system will send the picture and text to the WeChat of the operation and maintenance personnel and start the fire alarm.

[0088] In a verification example, when only the smoke target positioning box that reaches the confidence threshold output by the smoke target recognition model is used as the smoke target recognition result, the prediction accuracy is only 78.12%. This is because the smoke target recognition model only uses the pre-trained features of the entire image when extracting image features, and does not dynamically adjust the search according to the target, causing the multimodal model to produce hallucination defects similar to those of the large language model. Therefore, the smoke target recognition model has a low prediction accuracy when performing zero-sample prediction without training, and cannot be applied to fire warnings in photovoltaic power plants.

[0089] After further using the visual search strategy combined with the smoke target search prompt heat map for active search, the introduction of active search mechanism, visual working memory and dynamic adjustment strategy can fully utilize the world knowledge and context understanding ability of the smoke target recognition model by simulating the recognition mechanism of humans for identifying targets, thereby improving the accuracy of visual positioning and multimodal collaborative reasoning ability. Therefore, the prediction accuracy can be increased to 93.27%.

[0090] Despite this, it is still impossible to effectively distinguish smoke and non-smoke targets at this time. In order to further improve the recognition rate and reduce the false alarm rate of the early warning system, the prior knowledge in the fire smoke feature fingerprint library is further combined, and the KNN unsupervised classification algorithm is used to compare the similarity between the suspected smoke target and the fire smoke feature fingerprint library, which effectively solves the false detection problem caused by the lack of historical memory of the multimodal model, and can effectively filter out the suspected fire smoke targets, so that the accuracy rate is further improved, so the prediction accuracy can be further increased to 96.58%.

[0091] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A photovoltaic power station fire warning method integrating visual search and multimodal large model, characterized in that: The photovoltaic power station fire early warning method comprises: Building a smoke target recognition model, the smoke target recognition model includes an image feature preprocessing module, a text feature preprocessing module, a feature fusion module, a multimodal large model module, a target position decoder and a visual clue decoder; Obtain the working video stream of the photovoltaic power station to be monitored and extract the image frames to be monitored in sequence, input each image frame to be monitored into the image feature preprocessing module in the smoke target recognition model, input the character string used to indicate the recognition requirement into the text feature preprocessing module in the smoke target recognition model, the feature fusion module fuses the image features extracted by the image feature preprocessing module and the text features extracted by the text feature preprocessing module and inputs them into the multimodal large model module, the multimodal large model module outputs the target position feature vector to the target position decoder through the linear layer, the multimodal large model module also outputs the visual clue feature vector to the visual clue decoder through the linear layer, the target position decoder outputs the smoke target positioning frame and confidence based on the target position feature vector and the image features extracted by the image feature preprocessing module, the visual clue decoder outputs the smoke target search prompt heat map based on the target position feature vector, the image features extracted by the image feature preprocessing module and the multimodal fusion features output by the feature fusion module; the smoke target search prompt heat map indicates the confidence that different positions contain smoke targets; When the confidence of the smoke target positioning frame does not reach the confidence threshold, an active search is performed based on the smoke target search prompt heat map using a visual search strategy until a smoke target positioning frame that reaches the confidence threshold is obtained, or until it is determined that the image frame to be monitored does not contain a smoke target; A smoke target recognition result is obtained according to the smoke target positioning frame that reaches the confidence threshold, and a fire warning result for the photovoltaic power station to be monitored is obtained according to the smoke target recognition result.

2. The photovoltaic power station fire early warning method according to claim 1, characterized in that: Active search based on smoke target search cue heat map using visual search strategies includes: The image frame to be monitored is divided into multiple sub-areas, and the confidence that each sub-area contains a smoke target is determined based on the smoke target search prompt heat map, and each sub-area is traversed in order from high to low in terms of the confidence that the smoke target is contained. For each traversed sub-area: the image of the image frame to be monitored in the sub-area is input into the image feature preprocessing module in the smoke target recognition model, and the character string used to indicate the recognition requirement is input into the text feature preprocessing module in the smoke target recognition model to obtain the smoke target positioning box and confidence for the sub-area.

3. The photovoltaic power station fire early warning method according to claim 1, characterized in that: The step of obtaining a smoke target recognition result according to the smoke target positioning frame reaching the confidence threshold comprises: After enlarging each smoke target positioning frame that reaches the confidence threshold, extracting the target local image of the image frame to be monitored within the enlarged smoke target positioning frame; The image feature preprocessing module is used to extract the image features of each target local image, and the smoke target recognition result is obtained based on the similarity between the image features of each target local image and the fingerprint features of the smoke target in the fire smoke feature fingerprint library.

4. The photovoltaic power station fire early warning method according to claim 3, characterized in that: The obtaining of the smoke target recognition result based on the similarity between the image feature of the target local image and the fingerprint feature of the smoke target in the fire smoke feature fingerprint library comprises: Using the KNN algorithm to calculate the similarity between the image features of the target local image and the visual fingerprint features of the smoke target in the fire smoke feature fingerprint library and obtain a visual classification result, and using the KNN algorithm to calculate the similarity between the image features of the target local image and the text fingerprint features of the smoke target in the fire smoke feature fingerprint library and obtain a text classification result; The smoke target recognition result is obtained according to the visual classification result and the text classification result.

5. The photovoltaic power station fire early warning method according to claim 4, characterized in that: The smoke target recognition result obtained according to the visual classification result and the text classification result includes: For each target local image, when the visual classification result indicates that the image features of the target local image do not belong to any visual fingerprint feature classification, and the text classification result indicates that the image features of the target local image do not belong to any text fingerprint feature classification, it is determined that the target local image does not include a smoke target; otherwise, it is determined that the target local image contains a smoke target; the smoke target recognition result is obtained by combining all target local images in the image frame to be monitored.

6. The photovoltaic power station fire early warning method according to claim 5, characterized in that: The fire smoke feature fingerprint library includes fingerprint features of smoke targets of different sizes. The smoke target recognition result obtained according to the visual classification result and the text classification result also includes: For each target partial image, determine the size of the smoke target corresponding to the fingerprint features in the visual fingerprint feature classification to which the image features of the target partial image belong and the text fingerprint feature classification, and determine the size of the smoke target contained in the target partial image according to the voting principle of minority obeys majority.

7. The photovoltaic power station fire early warning method according to claim 4, characterized in that: The photovoltaic power station fire early warning method further includes: Acquire a number of sample images of photovoltaic power plants containing smoke targets, mark the size of the smoke targets in each sample image of the photovoltaic power plant, and add text description information for describing smoke features of the smoke targets in the sample image of the photovoltaic power plant to each sample image of the photovoltaic power plant; For each photovoltaic power station sample image, an image feature preprocessing module is used to extract image features of the photovoltaic power station sample image as visual fingerprint features, and a text feature preprocessing module is used to extract text features of text description information of the photovoltaic power station sample image as text fingerprint features; The fire smoke feature fingerprint library is constructed by integrating the visual fingerprint features and text fingerprint features of sample images of various photovoltaic power stations.

8. The photovoltaic power station fire early warning method according to claim 7, characterized in that: The text description information added to each PV power station sample image is used to describe the smoke characteristics including the location, size, coverage area, color characteristics, optical characteristics, texture characteristics, physical and chemical characteristics of the smoke target and the characteristics representing the degree of fire.

9. The photovoltaic power station fire early warning method according to claim 7, characterized in that: The step of acquiring a plurality of sample images of photovoltaic power plants containing smoke targets comprises: Collect historical images of a photovoltaic power station when a fire occurs during historical operation and contains smoke targets, and use a defogging algorithm to preprocess the historical images to obtain a sample image of the photovoltaic power station; Furthermore, the simulation experiment platform is used to simulate the scene when photovoltaic modules produce smoke and obtain sample images of photovoltaic power stations.

10. The photovoltaic power station fire early warning method according to claim 1, characterized in that: In the smoke target recognition model: The image feature preprocessing module includes an image encoder, a linear layer, and a resampling layer connected in sequence. The image encoder adopts the ViT image encoder in the CLIP model. The text feature preprocessing module includes a word segmentation encoding module and a vectorization encoding module connected in sequence. The word segmentation encoding module adopts the BPE word segmentation algorithm; The multimodal large model module uses the LLaVA multimodal model trained on a common dataset; The target position decoder uses the decoder structure of the target detection model DETR based on the Transformer architecture; The visual cue decoder adopts the decoder structure of the open-world segmentation model SAM based on the Transformer architecture.

Images