A flame detection method based on a visual text large-scale pre-training model

Abstract

The application discloses a flame detection method based on a visual text large-scale pre-training model, and specifically comprises the following steps: obtaining a visual text large-scale pre-training model; fine-tuning the pre-training model on an image description generation dataset, a visual question answering dataset and a visual text retrieval dataset to obtain a description generation model, a visual question answering model and a visual text retrieval model respectively; integrating the three models by using a multi-expert mode to obtain a multi-expert flame detection model; and judging whether there is a flame in the image to be detected by fusing the output results of the three models, and finally completing the flame detection task with high quality. The application takes the visual text large-scale pre-training model as the core, and further proposes the multi-expert flame detection model, which has strong generalization ability, context understanding ability and reasoning ability, and can achieve good performance in the task scene without specific flame data training, and has a broader application prospect.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a flame detection method based on a large-scale pre-trained model of visual text. Background Technology

[0002] Flames are extremely dangerous and destructive, rapidly igniting flammable materials and spreading quickly, causing widespread damage. Economically, fires cause immense destruction and losses, incurring high repair and reconstruction costs, severely damaging business and personal property, and negatively impacting commerce and employment. Furthermore, environmentally, fires release large amounts of smoke, toxic gases, and dust, seriously affecting air quality and the environment, and exacerbating global warming and climate change. Therefore, preventing fires to reduce their hazards is crucial.

[0003] In recent years, flame detection methods based on object detection have attracted extensive research. These methods utilize mature object detection frameworks such as YOLOv5 to perform flame detection. However, in practice, object detection-based flame detection methods often face challenges such as a limited number of learnable samples, difficulty in collecting samples, and the complexity and variability of samples. This makes the trained models prone to overfitting and exhibits weak generalization ability. Consequently, object detection-based flame detection methods are difficult to apply to real-world scenarios.

[0004] Recently, the emergence of large-scale pre-trained visual-text models has alleviated the aforementioned challenges. Specifically, these models are trained on massive amounts of visual and textual data, possessing powerful contextual understanding and reasoning capabilities. They can better handle complex scenes and semantic contexts, exhibiting outstanding generalization ability, enabling them to achieve good performance even in tasks where specific data training is not required. Therefore, how to utilize large-scale pre-trained visual-text models for flame detection remains a challenge. Summary of the Invention

[0005] To address the above problems, this invention proposes a flame detection method based on a large-scale pre-trained model of visual text. The specific technical solution adopted in this invention is as follows:

[0006] A flame detection method based on a large-scale pre-trained model of visual text includes the following steps:

[0007] S1. Fine-tune the large-scale pre-trained visual text model on the image description generation dataset to obtain a description generation model for flame detection. The judgment process in the description generation model is as follows: input the image to be detected into the description generation model, the description generation model outputs N text descriptions, detect whether the N text descriptions contain keywords related to flame, and determine whether there is a flame in the image to be detected based on the detection results.

[0008] S2. Fine-tune the large-scale pre-trained visual text model on the visual question answering dataset to obtain a visual question answering model for flame detection. The judgment process in the visual question answering model is as follows: input the image to be detected and O text queries into the visual question answering model. Each text query is a question asking whether there is a flame in the image. The visual question answering model outputs O corresponding answer results. Determine whether there is a flame in the image to be detected based on the answer results.

[0009] S3. Fine-tune the large-scale pre-trained visual text model on the visual text retrieval dataset to obtain a visual text retrieval model for flame detection. The judgment process in the visual text retrieval model is as follows: input the image to be detected and Q text descriptions together into the visual text retrieval model. Each text description is a keyword related to flame. The visual text retrieval model outputs visual features and text features of each text description. Calculate the similarity between the visual features and each text feature. Use the threshold method to determine whether there is a flame in the image to be detected.

[0010] S4. The description generation model, visual question answering model, and visual text retrieval model are integrated using a multi-expert model to obtain a multi-expert flame detection model. The multi-expert flame detection model fuses the output results of the three models to finally determine whether there is a flame in the image to be detected.

[0011] Preferably, the large-scale pre-trained visual text model adopts the BLIP model.

[0012] Preferably, the keywords related to flame are synonyms of "fire" and "flame", or the names of substances that can emit light and are combustible.

[0013] Preferably, in step S1, the method for determining whether a flame exists in a given image based on the detection result is as follows: if any text description contains the keyword related to flame, then it is considered that a flame exists in the image to be detected.

[0014] Preferably, in step S2, the method for determining whether a flame exists in a given image based on the response results is as follows: if any response result indicates that a flame exists in the image, then it is considered that a flame exists in the image to be detected.

[0015] Preferably, in S3, for visual feature S and each text feature W i ∈{W1,W2,…,W Q The similarity X between the two i ∈{X1,X2,…,X Q The computational function of} is in the form of:

[0016]

[0017] Where · represents the dot product; * represents the product; ||·|| represents the L2 norm;

[0018] Preferably, in step S3, the method for determining whether there is a flame in the image to be detected using the threshold method is as follows: the similarity value between the visual features and each text feature is compared with a set similarity threshold. If any similarity value is higher than the set similarity threshold, then it is considered that there is a flame in the image to be detected.

[0019] Preferably, the similarity threshold is 0.2.

[0020] Preferably, in step S4, the method for the multi-expert flame detection model to fuse the output results of the three models is as follows: if the number of experts who determine that there is a flame exceeds a set threshold, then it is finally considered that there is a flame in the image to be detected.

[0021] Preferably, in step S4, the quantity threshold is 2.

[0022] Compared with the prior art, the present invention has the following characteristics:

[0023] 1) This invention takes a large-scale pre-trained visual text model as its core, fine-tunes it on image description generation dataset, visual question answering dataset and visual text retrieval dataset, and integrates the above three steps using a multi-expert mode, ultimately achieving the goal of accurately detecting flames.

[0024] 2) The multi-expert flame detection model proposed in this invention possesses strong generalization ability, contextual understanding ability, and reasoning ability, achieving good performance even in task scenarios where specific flame data training is not required. This invention can alleviate the problem of weak model generalization ability faced by past flame detection methods based on object detection, and has better practical application value. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a flame detection method based on a large-scale pre-trained model of visual text.

[0026] Figure 2 This is an overall architecture diagram of a flame detection method based on a large-scale pre-trained model of visual text. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0028] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0029] In a preferred embodiment of the present invention, a flame detection method based on a large-scale pre-trained model of visual text is provided, which can accurately predict whether there is a flame in an image, specifically including the following steps:

[0030] S1. Fine-tune the large-scale pre-trained visual text model on the image description generation dataset to obtain a description generation model for flame detection. The judgment process in the description generation model is as follows: input the image to be detected into the description generation model, the description generation model outputs N text descriptions, detect whether the N text descriptions contain keywords related to flame, and determine whether there is a flame in the image to be detected based on the detection results.

[0031] S2. Fine-tune the large-scale pre-trained visual text model on the visual question answering dataset to obtain a visual question answering model for flame detection. The judgment process in the visual question answering model is as follows: input the image to be detected and O text queries into the visual question answering model. Each text query is a question asking whether there is a flame in the image. The visual question answering model outputs O corresponding answer results. Determine whether there is a flame in the image to be detected based on the answer results.

[0032] S3. Fine-tune the large-scale pre-trained visual text model on the visual text retrieval dataset to obtain a visual text retrieval model for flame detection. The judgment process in the visual text retrieval model is as follows: input the image to be detected and Q text descriptions together into the visual text retrieval model. Each text description is a keyword related to flame. The visual text retrieval model outputs visual features and text features of each text description. Calculate the similarity between the visual features and each text feature. Use the threshold method to determine whether there is a flame in the image to be detected.

[0033] S4. The description generation model, visual question answering model, and visual text retrieval model are integrated using a multi-expert model to obtain a multi-expert flame detection model. The multi-expert flame detection model fuses the output results of the three models to finally determine whether there is a flame in the image to be detected.

[0034] It should be noted that in the embodiments of the present invention, N, O, and Q are hyperparameters representing quantities. The specific values ​​of the hyperparameters can be greater than or equal to 1, and can be fine-tuned according to the actual dataset being processed. In the embodiments of the present invention, no limitation is made.

[0035] It should be noted that the large-scale pre-trained visual text model used in the embodiments of this invention can be any open-source and general-purpose model. It can be selected according to actual needs, and after fine-tuning, any model suitable for image description, visual question answering, and visual text retrieval tasks can be used. Specifically, the BLIP model is used in this embodiment, and its implementation is existing technology. For details, please refer to the existing literature: Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022. Further details will not be elaborated upon here.

[0036] It should be noted that the image description generation dataset and visual text retrieval dataset used in the embodiments of this invention are derived from the existing technical literature Karpathy, A. and Li, F. Deep visual-semantic alignments for generating image descriptions. In CVPR, pp. 3128–3137, 2015. In the image description generation dataset, each training sample consists of several images and their corresponding real text descriptions. The real text descriptions are used as labels. The model inputs an image and outputs a predicted text description corresponding to the image. In the visual text retrieval dataset, each training sample consists of several images and their corresponding text descriptions. The text descriptions are used as labels. The model inputs an image and its corresponding text description and outputs the result of whether the input image and its corresponding text description match. Further details on this will not be elaborated upon.

[0037] It should be noted that the visual question answering dataset used in the embodiments of the present invention comes from the existing technical literature Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQAmatter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6325–6334, 2017. In the visual question answering dataset, each training sample is in the form of several images, a text query corresponding to the image, and a real answer result. The real answer result is used as the label value. The model inputs the image and the text query corresponding to the image and outputs the predicted answer result. This will not be elaborated further.

[0038] It should be noted that the number of training samples in the image description generation dataset, visual question answering dataset, and visual text retrieval dataset of this invention can be selected according to the actual situation, and is not limited in this invention embodiment.

[0039] It should be noted that in this embodiment of the invention, the pre-trained model is fine-tuned on the image description generation dataset, the visual question answering dataset, and the visual text retrieval dataset, respectively. During the fine-tuning process, it is necessary to adjust the learning rate, batch size, and number of training epochs. For details, please refer to the existing technical literature: Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022., which will not be elaborated further.

[0040] It should be noted that, in the embodiments of the present invention, the description generation model for flame detection, the visual question answering model for flame detection, and the visual text retrieval model for flame detection are specifically fine-tuned on a large-scale pre-trained BLIP model for visual text. For the specific implementation process of each model, please refer to existing technical literature. https: / / github.com / Salesforce / BLIP .

[0041] In step S1 of this embodiment of the invention, the keywords related to flame can be the names of substances that emit light and are combustible, or synonyms related to "fire" or "flame". In this embodiment of the invention, the specific form of the keywords related to flame is "fire" or "flame"; at the same time, the detection results can be used to determine whether a flame exists in a given image, that is, if any text description contains the keywords related to flame, then it is considered that a flame exists in the image to be detected.

[0042] In step S2 of this embodiment, each text query is a question asking whether a flame exists in the image. The question content can be a description with the same or similar meaning, such as "Is there a fire?", "Is there a flame?", or "Name of a substance that can emit light and burn?". The form of the question is not limited in this embodiment. This embodiment provides one text query: "Is there a flame / fire?". The visual question-answering model outputs the answer, which can be "Yes" or "No". The presence of a flame in the image to be detected is determined based on the answer. That is, if any answer indicates that a flame exists in the image, then a flame is considered to exist in the image to be detected.

[0043] In step S3 of this embodiment, the text description can be constructed as a sentence describing a flame, used to determine whether the image to be detected matches the text description. For example, if a flame exists in the image to be detected and the text description is "flame," then the two can be considered a match. In this embodiment, one text description is given as "flame, fire." Furthermore, for each visual feature S and each text feature W... i ∈{W1,W2,…,W Q The similarity X between visual features and text features described in each text description. i ∈{X1,X2,…,X Q}, similarity X i The computational function is in the form of:

[0044]

[0045] Where · represents the dot product; * represents the product; ||·|| represents the L2 norm;

[0046] In step S3 of this embodiment of the invention, the method for determining whether a flame exists in the image to be detected using the threshold method is as follows: the similarity value between the visual features and each text feature is compared with a set similarity threshold. If any similarity value is higher than the set similarity threshold, then the image to be detected is considered to contain a flame. In this embodiment of the invention, the similarity threshold is 0.2.

[0047] It should be noted that, in this embodiment of the invention, the description generation model, visual question answering model, and visual text retrieval model are integrated using a multi-expert model to obtain a multi-expert flame detection model, such as... Figure 2 As shown, specifically, the description generation model, visual question answering model, and visual text retrieval model, after fine-tuning the large-scale pre-trained BLIP model for visual text, are regarded as domain experts for each task, jointly determining whether a flame exists in a given image. The idea of ​​this multi-expert mode can be found in the following reference: Gormley IC, Frühwirth-Schnatter S. Mixture of experts models[J]. Handbook of mixture analysis,2019:271-307. In particular, new experts can be added or removed as needed.

[0048] It should be noted that, in this embodiment of the invention, the method for fusing the output results of the three models by the multi-expert flame detection model is as follows: if the number of experts who determine that a flame exists exceeds a set threshold, then the image to be detected is ultimately considered to contain a flame. In this embodiment of the invention, the threshold is 2.

[0049] Example

[0050] The implementation method of this embodiment is as described above, and the specific steps will not be elaborated in detail again. The following only shows its effect based on the case data.

[0051] In this embodiment of the invention, a flame detection image dataset is constructed from 9372 images containing flames and 11497 images not containing flames. In this embodiment, the images containing and not containing flames are drawn from various real-life scenarios and are used to determine the prediction accuracy of the multi-expert flame detection model in both flame and non-flame scenarios.

[0052] The BLIP model was adopted and fine-tuned on image description generation datasets, visual question answering datasets, and visual text retrieval datasets. To demonstrate the performance of the multi-expert flame detection model, this embodiment of the invention uses only one text description generated by the model in the image description task, only one text query "Is there a flame / fire?" in the image question answering task, and only one text description "flame,fire" in the image retrieval task, with a similarity threshold set to 0.2.

[0053] When one or more experts believe that a flame exists, the image to be detected is considered to contain a flame. This embodiment of the invention achieves an accuracy rate of 99.7% on images containing flames and 80.2% on images where no flame exists. When two or more experts believe that a flame exists, the image to be detected is considered to contain a flame. This embodiment of the invention achieves an accuracy rate of 98.3% on images containing flames and 97.9% on images where no flame exists. When three experts believe that a flame exists, the image to be detected is considered to contain a flame. This embodiment of the invention achieves an accuracy rate of 85.2% on images containing flames and 99.8% on images where no flame exists.

[0054] Traditional flame detection methods train neural network models using hundreds or thousands of flame images. The detection performance of this approach is highly dependent on the quality of the flame image dataset; a high-quality dataset is required for good results. However, constructing such datasets presents challenges such as a limited number of learnable flame samples, difficulty in collecting such samples, and the complexity and variability of existing samples. This makes models trained using this method prone to overfitting and weak generalization. In contrast, large-scale pre-trained visual text models are trained on real-world datasets containing millions or even hundreds of millions of images, exhibiting extremely strong generalization capabilities. Therefore, this invention uses a large-scale pre-trained visual text model as its core. It fine-tunes the model on image description generation datasets, visual question answering datasets, and visual text retrieval datasets, and integrates these three steps using a multi-expert model. Combined with deep learning technology, it ultimately achieves accurate flame detection. The proposed multi-expert flame detection model possesses strong generalization, contextual understanding, and reasoning abilities. This invention alleviates the weak generalization problem faced by past object detection-based flame detection methods and has greater practical application value.

[0055] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A flame detection method based on a large-scale pre-trained model of visual text, characterized in that, Includes the following steps: S1. Fine-tune the large-scale pre-trained visual text model on the image description generation dataset to obtain a description generation model for flame detection. The judgment process in the description generation model is as follows: input the image to be detected into the description generation model, the description generation model outputs N text descriptions, detect whether the N text descriptions contain keywords related to flame, and determine whether there is a flame in the image to be detected based on the detection results. S2. Fine-tune the large-scale pre-trained visual text model on the visual question answering dataset to obtain a visual question answering model for flame detection. The judgment process in the visual question answering model is as follows: the image to be detected and O text queries are input into the visual question answering model. Each text query is a question asking whether there is a flame in the image. The visual question answering model outputs O corresponding answer results. The presence of a flame in the image to be detected is determined based on the answer results. S3. Fine-tune the large-scale pre-trained visual text model on the visual text retrieval dataset to obtain a visual text retrieval model for flame detection. The judgment process in the visual text retrieval model is as follows: input the image to be detected and Q text descriptions together into the visual text retrieval model. Each text description is a keyword related to flame. The visual text retrieval model outputs visual features and text features of each text description. Calculate the similarity between the visual features and each text feature. Use the threshold method to determine whether there is a flame in the image to be detected. S4. The description generation model, visual question answering model, and visual text retrieval model are integrated using a multi-expert model to obtain a multi-expert flame detection model. The multi-expert flame detection model fuses the output results of the three models to finally determine whether there is a flame in the image to be detected.

2. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, The large-scale pre-trained visual text model adopts the BLIP model.

3. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, The keywords related to flame are "fire", "flame", "fire" or "flame".

4. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, In step S1, the method for determining whether a flame exists in a given image based on the detection result is as follows: if any text description contains the keyword related to flame, then it is considered that a flame exists in the image to be detected.

5. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, In S2, the method for determining whether there is a flame in a given image based on the answer result is as follows: if any answer result shows that there is a flame in the image, then it is considered that there is a flame in the image to be detected.

6. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, In S3, for visual features and each text feature The similarity between the two The computational function is in the form of: in, Represents the dot product; Indicates product; This represents the L2 norm.

7. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, In step S3, the method of determining whether there is a flame in the image to be detected using the threshold method is as follows: the similarity value between the visual features and each text feature is compared with a set similarity threshold. If any similarity value is higher than the set similarity threshold, then it is considered that there is a flame in the image to be detected.

8. The flame detection method based on a large-scale pre-trained visual text model according to claim 7, characterized in that, The similarity threshold is 0.

2.

9. The flame detection method based on a large-scale pre-trained visual text model according to claim 1, characterized in that, In S4, the method for the multi-expert flame detection model to fuse the output results of the three models is as follows: if the number of experts who determine that there is a flame exceeds the set threshold, then it is finally considered that there is a flame in the image to be detected.

10. A flame detection method based on a large-scale pre-trained visual text model according to claim 9, characterized in that, In S4, the quantity threshold is 2.

Images