Fire extinguishing system based on visual content correlation

Abstract

The invention relates to a fire extinguishing system based on visual content correlation, belongs to the technical field of fire detection, and solves the problems that the false alarm rate is high, the response speed is low, complex scenes cannot be recognized and an effective fire extinguishing mode cannot be provided due to the fact that an existing fire extinguishing system depends on a single sensor. The system comprises an image acquisition module used for acquiring an image of a fire scene; the image analysis module is used for segmenting the image and obtaining a preliminary description text of the fire scene; the visual content correlation extraction module is used for extracting the image features and the text features of the preliminary description text, and selecting the features of the segmented image most related to the text content for fusion based on the similarity between the image features and the text features; and the multi-modal reasoning module is used for converting the fused image features into accurate text description of a fire scene by using a multi-modal large model, and reasoning with a fire knowledge base by using the multi-modal large model to obtain a final fire extinguishing scheme.

Description

Technical Field

[0001] The present invention relates to the technical field of fire detection, and in particular to a fire extinguishing system based on visual content correlation. Background Art

[0002] Fire is a very destructive disaster, which not only causes huge property losses, but also seriously threatens people's lives. With the development of society, the demand for fire safety monitoring in the public domain is growing.

[0003] Traditional flame detection methods mainly rely on single-mode sensors such as smoke detectors and temperature sensors, which have limitations such as high false alarm rate, slow response speed, and inability to identify complex scenes. In recent years, with the rapid development of technologies such as computer vision and deep learning, breakthrough progress has been made in flame detection using target detection algorithms. However, machine vision and deep learning models can only detect whether there is a flame within the camera range and locate the location of the flame. Simply identifying the location of the flame can only serve as an early warning. In the process of fire extinguishing, in different fire scenarios, using the wrong fire extinguishing method will lead to the intensification of the fire, casualties, and property losses. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a fire extinguishing system based on visual content relevance, so as to solve the problems of high false alarm rate, slow response speed, inability to recognize complex scenes and inability to provide effective fire extinguishing methods caused by the existing fire extinguishing system relying on a single sensor.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] The present invention provides a fire extinguishing system based on visual content correlation, comprising:

[0007] An image acquisition module, used for acquiring images containing the fire scene;

[0008] An image parsing module is used to segment the image and obtain a preliminary description text of the fire scene in the image;

[0009] A visual content relevance extraction module is used to extract the image features of each segmented image and the text features of the preliminary description text respectively, and based on the similarity of each of the image features and the text features, select the features of several segmented images that are most relevant to the text content for fusion to obtain relevance enhanced image features;

[0010] The multimodal reasoning module is used to use the multimodal large model to convert the correlation enhanced image features into an accurate text description of the fire scene, perform similarity matching with a pre-built fire knowledge base, obtain the matched fire knowledge base text, and then use the multimodal large model to reason with the accurate text description again to obtain the final fire extinguishing plan.

[0011] Furthermore, the image analysis module includes an image segmentation module and a preliminary description text generation module; wherein,

[0012] The image segmentation module is used to decompose the image at different scales to obtain a plurality of sub-images;

[0013] The preliminary description text generation module is used to obtain a preliminary description text about the fire scene based on the image using a multimodal large model.

[0014] Furthermore, the visual content correlation extraction module includes a feature extraction module, a to-be-fused image screening module and a feature fusion module;

[0015] The feature extraction module uses a pre-trained image encoding module and a text encoding module to extract image features of each sub-image and text features of the preliminary description text respectively; wherein the text features include features of each word after the preliminary description text is segmented and overall features of the preliminary description text;

[0016] The image screening module to be fused calculates similarity between the image features of each sub-image and the features of each word through a pre-trained similarity calculation module, and constructs similarities between each sub-image and the preliminary description text based on the calculated similarities between each sub-image and each word for sorting, and selects the first k sub-image features with the highest similarity as the image features to be fused;

[0017] The feature fusion module is used to use a pre-trained similarity calculation module to calculate the similarity between each of the image features to be fused and the overall features of the preliminary description text, and use the obtained similarity as the weight of each of the image features to be fused, and perform weighted fusion on each of the image features to be fused to obtain the correlation enhanced image feature.

[0018] Furthermore, similarity calculation is performed between the image features of each sub-image and the features of each word, including:

[0019] For each sub-image, the similarity between its image feature and the feature of each word is calculated to obtain a similarity vector of each sub-image;

[0020] The variance of the similarity vector of each sub-image is calculated respectively, and the obtained value is used as the similarity between each sub-image and the preliminary description text.

[0021] Furthermore, the image encoding module, text encoding module and similarity calculation module are trained using the following method:

[0022] Constructing a training data set for similarity calculation; the training data set includes a number of fire scene images and corresponding language description texts as sample data and the similarities of the corresponding fire scene images and language description texts as labels;

[0023] Loading each of the fire scene images into the image encoding module to obtain an image feature encoding vector;

[0024] Loading the corresponding language description text into the text encoding module to obtain a text feature encoding vector;

[0025] The image feature coding vector and the text feature coding vector are used to obtain similarity through the similarity calculation module;

[0026] The similarity labels corresponding to the sample data are loaded, and the image encoding module, the text encoding module and the similarity calculation module are trained simultaneously using the contrast loss function. The model parameters are optimized by minimizing the loss function to obtain the trained image encoding module, the text encoding module and the similarity calculation module.

[0027] Furthermore, the multimodal reasoning module includes a correlation feature reasoning module, a fire extinguishing knowledge retrieval module and a fire extinguishing decision module;

[0028] The correlation feature inference module is used to input the correlation enhanced image features into the multimodal large model to obtain an accurate text description of the fire scene;

[0029] The fire extinguishing knowledge retrieval module is used to measure the similarity between the features of the precise text description of the fire scene and the features of each text entry in the pre-built fire knowledge base, so as to obtain the features of the text entry most relevant to the fire scene description;

[0030] The fire extinguishing decision module is used to use a multimodal large model to reason with the features of the text items most relevant to the fire scene description and the precise text description to obtain a final fire extinguishing plan.

[0031] Furthermore, in the fire extinguishing knowledge retrieval module,

[0032] Using a pre-trained text encoding module, extracting features from the precise text description of the fire scene to obtain precise text description features;

[0033] Using a pre-trained text encoding module, extracting features from each text entry in the fire knowledge base to obtain features of each text entry;

[0034] The precise text description feature is respectively calculated for similarity with the scene feature of each text item, and the scene feature of the text item with the highest similarity value is obtained as the scene feature of the text item most relevant to the fire scene description.

[0035] Furthermore, in the fire extinguishing decision module, the features of the text entries most relevant to the fire scene description are spliced ​​with the precise text description features, and the spliced ​​features are inferred through a multimodal large model to obtain a final fire extinguishing plan.

[0036] Furthermore, the cosine similarity calculation formula is used to calculate the similarity between features.

[0037] Furthermore, each text entry in the fire knowledge base includes a fire extinguishing method category, a usage scenario of the fire extinguishing method of the current category, and precautions for the fire extinguishing method of the current category.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. The present invention is not limited by a single image modality and complexity in the process of flame and environment recognition. Only one scene image data is needed to perform fire monitoring and recommend fire extinguishing methods for open world scenes, so as to cope with fires in the public safety field, timely discover and use the best fire extinguishing method to prevent the further spread of the fire and reduce the risk.

[0040] 2. The present invention utilizes the visual content correlation method to perform multi-scale decomposition on the scene data, thereby completing the analysis of the scene image at different scales, so that the small scale may pay more attention to details, and the large scale may capture more overall structural information.

[0041] 3. The present invention uses a large-scale pre-trained image feature extraction model and a text feature extraction model to extract scene image features and semantic features respectively, and pairs the image features with the semantic features to establish the consistency of the image and semantic information in the implicit space.

[0042] 4. The present invention constructs a knowledge base and, in the process of determining the occurrence of a fire, proposes the best fire extinguishing method by querying the knowledge base to prevent the further spread of the fire, thereby reducing the risk.

[0043] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0045] Figure 1 Schematic diagram of the structure of a fire extinguishing system based on visual content correlation in an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of the structure of an image encoding module in an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the structure of the text encoding module in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0049] A specific embodiment of the present invention discloses a fire extinguishing system based on visual content correlation, such as Figure 1 As shown, it includes an image acquisition module, an image parsing module, a visual content relevance extraction module and a multimodal reasoning module.

[0050] Furthermore, the image acquisition module is used to acquire images containing the fire scene.

[0051] Specifically, an image of the fire scene is obtained through the surveillance camera of the fire scene, and the image is preprocessed including noise removal and image enhancement to facilitate subsequent image recognition.

[0052] The image analysis module is used to segment the image and obtain a preliminary description text of the fire scene in the image.

[0053] Furthermore, the image analysis module includes an image segmentation module and a preliminary description text generation module; wherein,

[0054] The image segmentation module is used to decompose the image at different scales to obtain a plurality of sub-images.

[0055] Specifically, the input image data is decomposed into information at different scales, and each scale contains enough local views to obtain different degrees of global and local information to capture different details and features of the fire scene. Exemplarily, the image is divided into 21 images of different scales according to the scales of 4×4, 2×2 and 1×1, among which the size of 4×4 is 1 / 16 of the whole image, which can obtain significant recognition features.

[0056] The preliminary description text generation module is used to obtain a preliminary description text about the fire scene based on the image using a multimodal large model.

[0057] Specifically, the Multimodal Large Language Models (MLLM) is an artificial intelligence model based on Large Language Models (LLM) and Large Vision Models (LVM), which can process and understand a variety of different types of data inputs, such as text, images, audio and video. The Multimodal Large Model achieves an integrated representation of multimodal data through a unified encoding library and joint embedding space, so that different modalities can be seamlessly processed. Through large-scale data pre-training, the model learns how to jointly understand and generate information across multiple modes.

[0058] The acquired image containing the fire scene is passed through the multimodal large model to obtain a preliminary description text about the fire scene. For example, the obtained preliminary description text may be "a fire occurred in the chemical pharmacy plant area, and the plant has caught fire."

[0059] Furthermore, a visual content relevance extraction module is used to respectively extract the image features of each segmented image and the text features of the preliminary description text, and based on the similarity of each of the image features and text features, select the features of several segmented images that are most relevant to the text content for fusion to obtain relevance enhanced image features.

[0060] Specifically, the visual content relevance extraction module converts the image and preliminary description text data into feature representations that can be compared and associated with each other, and then selects the image features most relevant to the text content for fusion to enhance the relevance.

[0061] Furthermore, the visual content relevance extraction module includes a feature extraction module, a to-be-fused image screening module and a feature fusion module.

[0062] The feature extraction module uses a pre-trained image encoding module and a text encoding module to respectively extract the image features of each sub-image and the text features of the preliminary description text; wherein the text features include the features of each word after the preliminary description text is segmented and the overall features of the preliminary description text.

[0063] Specifically, the image encoding module is used to extract the image features of each sub-image and convert the visual information into a feature vector that can be subsequently processed; Figure 2As shown, the image encoding module of this embodiment uses the VisionTransformer (ViT) model, which divides the image into small patches (patches) and uses the encoder part of the Transformer to extract features for classification.

[0064] The text encoding module is used to extract text features of the preliminary description text, convert text data into a numerical form that can be understood and processed by a machine, and capture the semantic information and overall context of words in the text; Figure 3 As shown, this embodiment uses a text feature extraction model (BERT), which is a pre-trained language model based on the Transformer architecture. It learns the deep semantic representation of text through bidirectional context information, which is very effective in understanding the complex semantic relationships of text.

[0065] It should be noted that the preliminary description text is segmented to decompose the continuous text into discrete words or phrases, so that the model can recognize and process the semantic units of the text, which serves as the basis for feature extraction and semantic understanding, thereby improving the accuracy and efficiency of text processing.

[0066] Furthermore, the image screening module to be fused calculates the similarity between the image features of each sub-image and the features of each word through a pre-trained similarity calculation module, and constructs the similarity between each sub-image and the preliminary description text based on the calculated similarity between each sub-image and each word, and sorts them, and selects the top k sub-image features with the highest similarity as the image features to be fused.

[0067] Specifically, in addition to the fire part, surrounding buildings and environment that are of concern, the acquired images of the fire scene also contain some irrelevant information, such as people, animals, natural landscapes and other parts that are irrelevant to the fire scene. Therefore, based on the similarity of the word segmentation between each sub-image and the preliminary description text, the sub-image most relevant to the fire scene description can be effectively screened out, thereby reducing interference and noise in subsequent processing and improving the accuracy and timeliness of fire detection.

[0068] Furthermore, in the similarity calculation module, the image features of the sub-images are similarly calculated with the text features of each extracted word to obtain the similarity between each sub-image and the text features of each word, thereby forming a similarity vector for each sub-image.

[0069] Specifically, the similarity calculation is to use the cosine similarity method to calculate the similarity between the image features and the text features in a trained similarity calculation module.

[0070] Furthermore, based on the similarity vectors of the sub-images, the similarity vector variance is calculated as the similarity between the sub-images and the preliminary description text.

[0071] Specifically, variance is a statistic used to measure the degree of dispersion between the similarity vector of a single sub-image and its average similarity. A large variance may mean that the sub-image is highly correlated with the description text in some dimensions, but not so correlated in other dimensions. This diversity may provide richer information and help capture the multifaceted features of the description text.

[0072] For example, there are M sub-images, and the preliminary description text is divided into N words. For each sub-image, its similarity vector S i Indicated as S i =[s i1 ,s i2 ,…,s iN ](i∈[1,M]); where S i represents the similarity vector of the i-th sub-image; s i1 represents the similarity between the ith sub-image and the first word; s iN Represents the similarity between the i-th sub-image and the N-th word segmentation.

[0073] The similarity vector variance of each sub-image is calculated using the following formula:

[0074]

[0075] Among them, Var i represents the variance of the similarity vector of the i-th sub-image; s ij Represents the similarity between the i-th sub-image and the j-th word.

[0076] The similarity vector variance Var of the above M sub-images i Sort in descending order and select the first k sub-image features with the highest similarity as the image features to be fused, ensuring that the fused features are both discriminative and can provide rich information.

[0077] Furthermore, in the multimodal learning process, the model needs to establish connections between different modalities. Therefore, the image encoding module, text encoding module and similarity calculation module are trained so that the model can better understand and capture the semantic relationship between image and text, and can more accurately evaluate the similarity between image features and text features, thereby improving the accuracy of retrieval and matching in practical applications.

[0078] Specifically, the following method is used to train the image encoding module, text encoding module and similarity calculation module:

[0079] A training data set for similarity calculation is constructed; the training data set includes a number of fire scene images and corresponding language description texts as sample data and the similarities of the corresponding fire scene images and language description texts as labels.

[0080] Specifically, the several fire scene images may use fire conditions, smoke, and fire detection data sets; the corresponding language description texts are multi-scale serialized text data; the similarities between the test set and the corresponding language description texts are manually annotated as labels to guide model training.

[0081] Loading each of the fire scene images into the image encoding module to obtain an image feature encoding vector;

[0082] Loading the corresponding language description text into the text encoding module to obtain a text feature encoding vector;

[0083] The image feature coding vector and the text feature coding vector are used to obtain similarity through the similarity calculation module;

[0084] The similarity labels corresponding to the sample data are loaded, and the image encoding module, the text encoding module and the similarity calculation module are trained simultaneously using the contrast loss function. The model parameters are optimized by minimizing the loss function to obtain the trained image encoding module, the text encoding module and the similarity calculation module.

[0085] Specifically, the goal of the contrastive loss function is to make the distance between the encoding vectors of similar samples closer and the distance between the encoding vectors of dissimilar samples farther. Using the contrastive loss function for training can not only train the image encoding module and the text encoding module to recognize the similarity between images and texts, but also train the similarity calculation module to accurately measure the similarity between image features and text features.

[0086] Furthermore, the feature fusion module is used to use a pre-trained similarity calculation module to calculate the similarity of each image feature to be fused and the overall feature of the preliminary description text, and use the similarity obtained as the weight of each image feature to be fused, and perform weighted fusion on each image feature to be fused to obtain the correlation enhanced image feature. Specifically, the sub-image with higher similarity to the preliminary description text will obtain a higher weight, so that it will occupy a more important position in the fusion process.

[0087] Furthermore, the multimodal reasoning module is used to use a multimodal large model to convert the correlation enhanced image features into an accurate text description of the fire scene, perform similarity matching with a pre-built fire knowledge base, obtain the matching fire knowledge base text, and then use the multimodal large model to reason with the accurate text description again to obtain the final fire extinguishing plan.

[0088] Furthermore, the multimodal reasoning module includes a correlation feature reasoning module, a fire extinguishing knowledge retrieval module and a fire extinguishing decision module.

[0089] The correlation feature reasoning module is used to input the correlation enhanced image features into the multimodal large model to obtain an accurate text description of the fire scene.

[0090] Specifically, by matching with each word segment of the preliminary description text, selecting a sub-image with high correlation according to the degree of matching, and performing weighted fusion based on the similarity with the overall text features of the preliminary description text, the correlation enhanced image feature that integrates multiple sub-images and emphasizes the strong correlation with the preliminary description text is obtained. In the process of using multimodal large model reasoning, the accuracy of decision-making and prediction can be improved.

[0091] After receiving the correlation-enhanced image features, the multimodal large model converts these features into precise text descriptions. In this process, the model uses its knowledge obtained through pre-training on a large number of image and text pairs to convert the visual information of the fire scene into a semantically accurate text description. For example, the precise text description can be "a sudden fire broke out in the chemical pharmacy plant area, the fire was fierce, the plant was engulfed in flames, and a large number of flammable and explosive chemicals were stored in the plant. Once the fire got out of control, the consequences would be disastrous."

[0092] Furthermore, the fire extinguishing knowledge retrieval module is used to measure the similarity between the features of the precise text description of the fire scene and the features of each text entry in the pre-built fire knowledge base to obtain the features of the text entry most relevant to the fire scene description.

[0093] Specifically, the pre-built fire knowledge base is a database containing a large amount of information about different types of fires and their response measures. This knowledge base is designed to assist firefighters in making quick and accurate decisions when facing different fire situations. It contains large-scale rescue knowledge data sets of various types collected from various disaster rescue documents, books, public information, etc.

[0094] Furthermore, each text entry in the fire knowledge base includes a fire extinguishing method category, a usage scenario of the fire extinguishing method of the current category, and precautions for the fire extinguishing method of the current category.

[0095] Specifically, the fire knowledge base constructed in this embodiment is constructed based on fire types. For example, the following are some examples of fire extinguishing methods in the fire knowledge base:

[0096] 1. Fire extinguishing method: water extinguishing

[0097] Applicable scenarios: Suitable for solid combustible fires (Class A fires) such as wood, paper, cloth, etc.

[0098] Example: Use a fire hose and water spray gun to spray water at the fire source to reduce the fire temperature and extinguish the flames.

[0099] Precautions: Avoid using on electrical fires (Class C fires) or oil fires (Class B fires) because water can cause oil fires to splash or conduct electricity.

[0100] 2. Fire extinguishing method: dry powder fire extinguishing

[0101] Applicable scenarios: Applicable to various types of fires, including solid combustibles (Class A), liquid or gas fires (Class B and Class C), and electrical equipment fires (Class E).

[0102] Example: When using a dry powder fire extinguisher, pull out the safety pin, hold the nozzle, aim at the root of the fire and spray the dry powder to cover the fire source and isolate it from oxygen.

[0103] Note: Dry powder fire extinguishers may cause secondary damage to precision equipment and are difficult to clean up after use.

[0104] 3. Fire extinguishing method: Carbon dioxide fire extinguishing

[0105] Applicable scenarios: Suitable for electrical equipment fires (Class E) and liquid fires (Class B), especially suitable for extinguishing fires of precision instruments and valuable equipment.

[0106] Example: When using a carbon dioxide fire extinguisher, pull out the safety pin, hold the nozzle, aim at the fire source and spray carbon dioxide to reduce the oxygen concentration around the fire source.

[0107] Note: Be careful to avoid frostbite when using, as carbon dioxide cools rapidly when sprayed.

[0108] 4. Fire extinguishing method category: Foam fire extinguishing

[0109] Applicable scenarios: Suitable for liquid fires (Class B), especially oil fires such as gasoline, diesel, etc.

[0110] Example: When using a foam fire extinguisher, the foam is sprayed onto the surface of the fire to form a covering layer, isolating the air and cooling the fire.

[0111] Note: Foam extinguishers are not suitable for electrical fires because the foam may conduct electricity.

[0112] 5. Fire extinguishing method: Sand fire extinguishing

[0113] Applicable scenarios: Suitable for small-scale oil fires (Class B), especially when there is no fire extinguisher.

[0114] Example: Use sand to cover the fire source, isolate the air, and prevent the fire from spreading.

[0115] Note: Sand fire extinguishing is suitable for small-scale fires, and it is necessary to ensure that the sand is dry. Wet sand may not be effective.

[0116] 6. Fire extinguishing method: automatic water sprinkler

[0117] Applicable scenarios: Suitable for fire prevention and automatic fire extinguishing in buildings, especially large commercial buildings, factories, etc.

[0118] Example: When a fire detector detects a fire source, the automatic sprinkler system will be activated to spray water mist or water stream to extinguish the fire.

[0119] Note: Automatic sprinkler systems require regular maintenance to ensure they work properly.

[0120] 7. Fire extinguishing method: gas fire extinguishing (such as heptafluoropropane)

[0121] Applicable scenarios: Suitable for fire protection in important places such as computer rooms, archives rooms, museums, etc., especially electrical fires and precision equipment fires.

[0122] Example: When a fire occurs, the gas fire extinguishing system will automatically release inert gases such as heptafluoropropane to reduce the oxygen concentration around the fire source and extinguish the flames.

[0123] Note: Gas fire extinguishing system poses a certain risk of suffocation to the human body. When using it, ensure that personnel evacuate safely.

[0124] 8. Fire extinguishing method: Cover with wet cloth

[0125] Applicable scenarios: Suitable for small-scale solid combustible fires (Class A fires), such as small fires on kitchen stoves.

[0126] Example: Use a wet cloth or towel to cover the fire source to isolate the air and prevent the fire from spreading.

[0127] Note: Covering with wet cloth is suitable for small-scale fires. If the fire is large, you should use a fire extinguisher or call the police in time.

[0128] 9. Fire extinguishing method category: suffocation fire extinguishing

[0129] Applicable scenarios: Suitable for small-scale liquid fires (Class B fires) or gas fires (Class C fires).

[0130] Example: Use non-combustible materials (such as asbestos cloth, wet blanket) to cover the fire source, isolate the air, and cause the flame to extinguish due to lack of oxygen.

[0131] Note: The asphyxiation method is suitable for small-scale fires. When the fire is large, you should use a fire extinguisher or call the police in time.

[0132] 10. Fire extinguishing method: Chemical suppression fire extinguishing

[0133] Applicable scenarios: Suitable for chemical fires, especially those with the risk of spontaneous combustion or explosion.

[0134] Example: Using chemical suppressants (such as Halons) to extinguish chemical fires to suppress the combustion chain reaction.

[0135] Note: Chemical inhibitors may be harmful to the environment and relevant safety regulations must be followed when using them.

[0136] Furthermore, the fire extinguishing knowledge retrieval module is used to measure the similarity between the features of the precise text description of the fire scene and the features of each text entry in the pre-built fire knowledge base to obtain the features of the text entry most relevant to the fire scene description.

[0137] Specifically, by matching the text entries most relevant to the fire scene description from the knowledge base and the most relevant knowledge entries, the subsequent multimodal large model reasoning process is made more targeted, thereby improving rescue efficiency, accuracy and response speed.

[0138] Furthermore, a pre-trained text encoding module is used to extract features of the precise text description of the fire scene to obtain precise text description features.

[0139] Specifically, the pre-trained text encoding module is used to pre-process the precise text description of the fire scene and then extract the feature vector of the text.

[0140] Use the pre-trained text encoding module to extract features from each text entry in the fire knowledge base to obtain the features of each text entry.

[0141] Similarly, the pre-trained text encoding module mentioned above performs preprocessing and feature extraction on each text entry in the fire knowledge base to generate a corresponding feature vector.

[0142] The precise text description feature is respectively calculated for similarity with the scene feature of each text item, and the scene feature of the text item with the highest similarity value is obtained as the scene feature of the text item most relevant to the fire scene description.

[0143] Specifically, the cosine similarity calculation formula is used to calculate the similarity of the precise text description feature and the scene feature of each text entry in the fire knowledge base, and the text entry with the highest similarity is obtained.

[0144] For example, when the precise text description is "A sudden fire broke out in the chemical pharmacy plant area. The fire was fierce and the plant was engulfed in flames. A large amount of flammable and explosive chemicals were stored in the plant. Once the fire got out of control, the consequences would be disastrous", the text entry with the highest similarity was "10. Fire extinguishing method category: chemical suppression fire extinguishing".

[0145] Furthermore, the fire extinguishing decision module is used to use a multimodal large model to reason with the features of the text items most relevant to the fire scene description and the precise text description to obtain a final fire extinguishing plan.

[0146] Furthermore, the features of the text items most relevant to the fire scene description are spliced ​​with the features of the precise text description, and the spliced ​​features are inferred through a multimodal large model to obtain a final fire extinguishing plan.

[0147] Specifically, the feature vector of the text entry most relevant to the description of the fire scene is spliced ​​with the feature vector of the precise text description, which can integrate the two types of information, so that the model can simultaneously consider the specific description of the fire scene and the professional fire-fighting knowledge in the knowledge base. The multimodal large model will conduct in-depth learning and reasoning on the spliced ​​feature vectors, consider the specific conditions of the fire scene and the fire-fighting knowledge in the knowledge base, and infer the final fire-fighting plan; the final fire-fighting plan can include specific fire-fighting methods, operating steps, safety precautions, etc., which can be directly used to guide on-site fire-fighting operations or provide real-time auxiliary decision support for firefighters.

[0148] For example, the precise text description "A sudden fire broke out in the chemical pharmacy plant area. The fire was fierce and the plant was engulfed in flames. A large number of flammable and explosive chemicals were stored in the plant. Once the fire got out of control, the consequences would be disastrous" was combined with the most relevant text entry "10. Fire extinguishing method category: chemical suppression fire extinguishing Applicable scenarios: Applicable to chemical fires, especially those chemicals with the risk of spontaneous combustion or explosion. Example: Use chemical inhibitors (such as halogenated fire extinguishing agents) to extinguish chemical fires and suppress the combustion chain reaction. Note: Chemical inhibitors may be harmful to the environment and must be used in accordance with relevant safety regulations." The features are then spliced ​​and input into the multimodal large model. The model uses its knowledge obtained through pre-training on a large number of image and text pairs to perform in-depth reasoning and analysis to infer the final fire extinguishing plan: "The fire in the chemical pharmacy plant area is extremely dangerous. People should be evacuated immediately, the police should be called, and professional firefighters should be waited for firefighting and rescue. Do not blindly use fire extinguishing agents such as water, foam, and dry powder to avoid violent reactions with chemicals, resulting in explosions or aggravation of the fire, resulting in more serious consequences. Use special fire extinguishing agents for different chemicals."

[0149] In summary, a fire extinguishing system based on visual content correlation according to an embodiment of the present invention has the following beneficial effects:

[0150] 1. The present invention is not limited by a single image modality and complexity in the process of flame and environment recognition. Only one scene image data is needed to perform fire monitoring and recommend fire extinguishing methods for open world scenes, so as to cope with fires in the public safety field, timely discover and use the best fire extinguishing method to prevent the further spread of the fire and reduce the risk.

[0151] 2. The present invention utilizes the visual content correlation method to perform multi-scale decomposition on the scene data, thereby completing the analysis of the scene image at different scales, so that the small scale may pay more attention to details, and the large scale may capture more overall structural information.

[0152] 3. The present invention uses a large-scale pre-trained image feature extraction model and a text feature extraction model to extract scene image features and semantic features respectively, and pairs the image features with the semantic features to establish the consistency of the image and semantic information in the implicit space.

[0153] 4. The present invention constructs a knowledge base and, in the process of determining the occurrence of a fire, proposes the best fire extinguishing method by querying the knowledge base to prevent the further spread of the fire, thereby reducing the risk.

[0154] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A fire extinguishing system based on visual content correlation, characterized in that: include: An image acquisition module, used for acquiring images containing the fire scene; An image parsing module is used to segment the image and obtain a preliminary description text of the fire scene in the image; A visual content relevance extraction module is used to extract the image features of each segmented image and the text features of the preliminary description text respectively, and based on the similarity of each of the image features and the text features, select the features of several segmented images that are most relevant to the text content for fusion to obtain relevance enhanced image features; The multimodal reasoning module is used to use the multimodal large model to convert the correlation enhanced image features into an accurate text description of the fire scene, perform similarity matching with a pre-built fire knowledge base, obtain the matched fire knowledge base text, and then use the multimodal large model to reason with the accurate text description again to obtain the final fire extinguishing plan.

2. The system according to claim 1, characterized in that: The image analysis module includes an image segmentation module and a preliminary description text generation module; wherein, The image segmentation module is used to decompose the image at different scales to obtain a plurality of sub-images; The preliminary description text generation module is used to obtain a preliminary description text about the fire scene based on the image using a multimodal large model.

3. The system according to claim 2, characterized in that: The visual content correlation extraction module includes a feature extraction module, a to-be-fused image screening module and a feature fusion module; The feature extraction module uses a pre-trained image encoding module and a text encoding module to extract image features of each sub-image and text features of the preliminary description text respectively; wherein the text features include features of each word after the preliminary description text is segmented and overall features of the preliminary description text; The image screening module to be fused calculates similarity between the image features of each sub-image and the features of each word through a pre-trained similarity calculation module, and constructs similarities between each sub-image and the preliminary description text based on the calculated similarities between each sub-image and each word for sorting, and selects the first k sub-image features with the highest similarity as the image features to be fused; The feature fusion module is used to use a pre-trained similarity calculation module to calculate the similarity between each of the image features to be fused and the overall features of the preliminary description text, and use the obtained similarity as the weight of each of the image features to be fused, and perform weighted fusion on each of the image features to be fused to obtain the correlation enhanced image feature.

4. The system according to claim 3, characterized in that: Calculating similarity between the image features of each sub-image and the features of each word includes: For each sub-image, the similarity between its image feature and the feature of each word is calculated to obtain a similarity vector of each sub-image; The variance of the similarity vector of each sub-image is calculated respectively, and the obtained value is used as the similarity between each sub-image and the preliminary description text.

5. The system according to claim 4, characterized in that: Use the following method to train the image encoding module, text encoding module and similarity calculation module: Constructing a training data set for similarity calculation; the training data set includes a number of fire scene images and corresponding language description texts as sample data and the similarities of the corresponding fire scene images and language description texts as labels; Loading each of the fire scene images into the image encoding module to obtain an image feature encoding vector; Loading the corresponding language description text into the text encoding module to obtain a text feature encoding vector; The image feature coding vector and the text feature coding vector are used to obtain similarity through the similarity calculation module; The similarity labels corresponding to the sample data are loaded, and the image encoding module, the text encoding module and the similarity calculation module are trained simultaneously using the contrast loss function. The model parameters are optimized by minimizing the loss function to obtain the trained image encoding module, the text encoding module and the similarity calculation module.

6. The system according to claim 3, characterized in that: The multimodal reasoning module includes a correlation feature reasoning module, a fire extinguishing knowledge retrieval module and a fire extinguishing decision module; The correlation feature inference module is used to input the correlation enhanced image features into the multimodal large model to obtain an accurate text description of the fire scene; The fire extinguishing knowledge retrieval module is used to measure the similarity between the features of the precise text description of the fire scene and the features of each text entry in the pre-built fire knowledge base, so as to obtain the features of the text entry most relevant to the fire scene description; The fire extinguishing decision module is used to use a multimodal large model to reason with the features of the text items most relevant to the fire scene description and the precise text description to obtain a final fire extinguishing plan.

7. The system according to claim 6, characterized in that: In the fire extinguishing knowledge retrieval module, Using a pre-trained text encoding module, extracting features from the precise text description of the fire scene to obtain precise text description features; Using a pre-trained text encoding module, extracting features from each text entry in the fire knowledge base to obtain features of each text entry; The precise text description feature is respectively calculated for similarity with the scene feature of each text item, and the scene feature of the text item with the highest similarity value is obtained as the scene feature of the text item most relevant to the fire scene description.

8. The system according to claim 7, characterized in that: In the fire-fighting decision module, the features of the text items most relevant to the fire scene description are spliced ​​with the precise text description features, and the spliced ​​features are inferred through a multimodal large model to obtain a final fire-fighting plan.

9. The system according to claim 8, characterized in that: The cosine similarity calculation formula is used to calculate the similarity between features.

10. The system according to any one of claims 6 to 9, characterized in that: Each text entry in the fire knowledge base includes a fire extinguishing method category, a usage scenario of the fire extinguishing method of the current category, and precautions for the fire extinguishing method of the current category.

Images