Emergency fire protection hidden danger troubleshooting method based on multi-mode AI large model identification technology

Abstract

The invention discloses an emergency fire-fighting hidden danger troubleshooting method based on a multi-modal AI large model identification technology, and relates to the field of artificial intelligence. The multi-modal hidden danger knowledge base is constructed by collecting multi-modal data such as images, texts and related laws and regulations actually recorded in the hidden danger checking process, the recognition and analysis ability of hidden danger information is improved based on the deep learning and multi-modal large model technology, the multi-hop reasoning and thinking chain technology is utilized, and the hidden danger checking efficiency is improved. A complex hidden danger positioning problem can be disassembled into a series of interpretable reasoning steps, and a user is helped to understand a hidden danger judgment basis. According to the method, the hidden danger in the image is analyzed from multiple angles and levels, different hypotheses or conditions can be verified step by step, so that the judgment accuracy is improved, the information of each dimension is comprehensively synthesized for comprehensive evaluation, and the comprehensiveness and accuracy of hidden danger investigation are ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an emergency fire hazard investigation method based on multimodal AI large model recognition technology. Background Art

[0002] With the continuous advancement of industrialization, the factory production environment has become more and more complex, and fire safety hazards have become an important factor threatening production safety and employee life safety. Traditional fire hazard detection methods mainly rely on manual inspection or single-mode data analysis, which often have the following shortcomings:

[0003] 1. Insufficient multimodal data fusion: Fire hazards usually exist in the form of images, texts and other data, but existing technologies have weak processing capabilities for multimodal data, making it difficult to effectively fuse them;

[0004] 2. Lack of accuracy and comprehensiveness in hidden danger identification: Traditional methods are based on predefined hidden danger classifications and are difficult to deal with new types of hidden dangers or those with significant changes. The diversity and ambiguity of hidden dangers make traditional methods difficult to adapt to complex scenarios in practical applications;

[0005] 3. Limited application of domain knowledge: The identification of fire hazards requires the combination of knowledge in the fields of fire safety regulations, equipment operation specifications, etc. The existing technology fails to fully and effectively apply these unstructured domain knowledge to the hazard detection model;

[0006] Therefore, how to achieve comprehensive and accurate identification of fire hazards and generate intelligent rectification suggestions based on laws and regulations has become an urgent problem to be solved by current technology. Summary of the invention

[0007] The purpose of the present invention is to provide an emergency fire hazard inspection method based on multimodal AI large model recognition technology. By constructing a multimodal hazard knowledge base and combining deep learning with multimodal large model technology, the recognition ability of hazard information is improved. Through multi-hop reasoning and knowledge retrieval enhanced generation technology, the present invention can accurately analyze fire hazards from multiple dimensions, and automatically generate rectification suggestions and legal basis, thereby improving the intelligence and efficiency of hazard inspection to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: an emergency fire hazard investigation method based on multimodal AI large model recognition technology, comprising the following steps:

[0009] Step 1: Data processing: collect historical hidden danger inspection data including images as multimodal data, organize the data uniformly and label the safety hidden dangers according to the corresponding hidden danger levels;

[0010] Step 2: Image processing: adding an adaptive image coding mechanism to the image data to directly process images of various resolutions and aspect ratios;

[0011] Step 3: Fine-tune the large model using the LoRa method. Use the collected and processed data to introduce the LoRa layer into each layer of the basic model, set the training parameters, and freeze all other parameters in the basic model except the LoRa layer. Only update the parameters in the LoRa layer, so that the model can adapt to specific tasks more quickly and form a more comprehensive judgment system for the large model in the field of emergency firefighting.

[0012] Step 4: Vectorize text data. Convert text data in historical hidden danger investigation data into numerical vector representation based on deep learning embedding model. Store this knowledge in the knowledge base in vector form, making the retrieval process more efficient.

[0013] Step 5: Hidden danger reasoning and retrieval. Design a multi-hop prompt template to perform multi-hop reasoning on hidden dangers for the input image or video. Guide the multi-dimensional analysis of hidden danger factors through a staged chain of thinking with different attention points. Comprehensively consider the information of each dimension, retrieve content similar to the hidden danger reasoning results in the knowledge base, re-examine the hidden danger reasoning results from multiple angles, and generate rectification opinions at the same time.

[0014] Step 6: Generate a response. Based on the results of hidden danger reasoning and the content of knowledge base retrieval, the big model is given the context to generate safety hazards, the legal basis for the corresponding safety hazards, and the rectification opinions for the corresponding safety hazards.

[0015] Preferably, the historical hidden danger inspection data in step one also includes text data of hidden danger report documents, equipment operating specifications, industry standards and relevant laws and regulations, and the text data is organized into structured electronic data, wherein the text data of the laws and regulations are organized into csv format, with one law as one cell, and at the same time, the safety hazards existing in each image in the image data of the historical hidden danger inspection are labeled with hidden danger levels and described, and a mapping relationship between each law and regulation and the safety hazard is established.

[0016] Preferably, the specific processing method of the picture by the adaptive image coding mechanism in step 2 is:

[0017] First, the image is scaled to the specified size, encoded using VIT, and the encoded images are merged and flattened;

[0018] Secondly, the image is divided into several parts according to the set specifications and sizes, and VIT is used to encode them separately, and the encoded images are merged and flattened;

[0019] Finally, the two flattened encoded images are vectorized and concatenated to obtain the final image encoding.

[0020] Preferably, in step 4, the specific method of converting the text data in the historical hidden danger investigation data into a numerical vector representation based on a deep learning embedding model comprises the following steps:

[0021] S101: Segment the structured text data into meaningful text blocks, where the text blocks may be words or sub-word units, and common stop words are removed;

[0022] S102: restore the text block to its basic form, reduce the diversity of vocabulary, and add corresponding part-of-speech tags;

[0023] S103: Convert the processed text block into a word vector using a word embedding model, store it in a vector data knowledge base, create an index for the knowledge base, and enable the knowledge base to perform retrieval and matching.

[0024] Preferably, the retrieval matching of the knowledge base specifically includes the following method: converting the user input query into a vector identical to the knowledge base, using the cosine similarity calculation method to search the knowledge base for the text data vector most similar to the query vector, and sorting the retrieved results from large to small according to the similarity score.

[0025] Preferably, the hidden danger reasoning and retrieval in step 5 specifically includes the following steps:

[0026] S201: Identify the content in the image data, and describe the predicted hidden danger content in multiple dimensions and steps according to the identified image content and the constructed multi-hop prompt framework;

[0027] S202: Input the predicted hidden danger content described into the knowledge base for similarity matching, output the content with the highest relevance score as the search content, and determine the laws and regulations corresponding to each safety hazard

[0028] S203: Retrieve professional content from the knowledge base and generate corresponding rectification opinions for the identified safety hazards according to corresponding laws and regulations.

[0029] As a preferred method, the specific method of inputting the predicted hidden danger content into the knowledge base for similarity matching is:

[0030] The CrossEncoder model is used to combine the step-by-step description of the predicted hidden danger content with the corresponding safety hidden danger text blocks retrieved from the knowledge base for encoding. The model generates a relevance score based on this pair of combined inputs, sorts them according to the model's score, and outputs the text block with the highest score as the final result retrieval content.

[0031] Preferably, the multi-hop prompt framework includes the following parts:

[0032] Problem statement: Describe the problem that needs to be solved;

[0033] Initial information: Provide necessary background information of the image content;

[0034] Reasoning steps: List the descriptions of the corresponding areas in the image content step by step according to the image content, and each step is equipped with corresponding logical basis;

[0035] Auxiliary information: introducing additional knowledge or data support;

[0036] Reasoning results: Output the reasoning results of each step and give a clear answer.

[0037] Preferably, a noise filtering mechanism is set in the retrieval and matching process of the knowledge base, and the specific method is: noise filtering is performed on text data and image data before retrieval, and only high-quality data is retained for indexing and storage. After the user submits the query, the retrieval results are dynamically filtered to remove irrelevant information, and the noise filtering mechanism is continuously optimized based on user feedback.

[0038] In summary, the beneficial effects of the present invention are:

[0039] 1. The present invention conducts a comprehensive investigation on the current status of emergency firefighting, combines the characteristics of factory fire hazards and the characteristics of multimodal data, collects multimodal data such as images, texts, relevant laws and regulations actually recorded in the process of hazard investigation, and uniformly organizes and annotates these data to establish a multimodal hazard knowledge base. It also proposes to improve the recognition and analysis capabilities of hazard information through deep learning and multimodal large model technology, and use multimodal thinking chains and retrieval enhancement generation technology to realize the multimodal data-driven accurate hazard identification function, helping users to quickly and accurately discover potential safety hazards in the production environment.

[0040] 2. The present invention utilizes multi-hop reasoning and thinking chain technology to decompose the complex hidden danger location problem into a series of explainable steps, making the decision-making process of the model more transparent, helping humans understand the basis for each hidden danger judgment, and analyzing the hidden dangers in the picture from multiple angles and levels, rather than relying solely on a single piece of information for judgment. Each reasoning step can gradually verify different assumptions or conditions, thereby enhancing the accuracy of the final judgment, and comprehensively considering the information of each dimension to make a comprehensive judgment.

[0041] Multi-hop reasoning guides the model to focus on different details through a phased chain of thinking. In each step of reasoning, the model not only pays attention to the core elements of the current reasoning step, but also analyzes and verifies the details again based on the conclusions of the previous steps. This gradual refinement process helps to improve the model's sensitivity to subtle hidden dangers in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 It is a schematic diagram of the process framework structure of an emergency fire hazard investigation method based on multimodal AI large model recognition technology of the present invention;

[0044] Figure 2 A schematic diagram of the division of laws and regulations for an emergency fire hazard investigation method based on multimodal AI large model recognition technology of the present invention;

[0045] Figure 3 A schematic diagram for describing safety hazards of an emergency fire hazard troubleshooting method based on multimodal AI large model recognition technology according to the present invention;

[0046] Figure 4 It is a schematic diagram of the image coding process framework of an emergency fire hazard investigation method based on multimodal AI large model recognition technology of the present invention;

[0047] Figure 5 This is a schematic diagram showing image coding of an emergency fire hazard inspection method based on multimodal AI large model recognition technology of the present invention. DETAILED DESCRIPTION

[0048] The present invention will now be further described in detail in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. These drawings are all simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic way, and therefore they only show the structures related to the present invention.

[0049] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0050] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0051] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0052] In the present invention, unless otherwise clearly defined and specified, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of at least two elements or the interaction relationship between at least two elements, unless otherwise clearly defined. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] Intelligent identification of hidden dangers based on multimodal large models is highly dependent on the amount of data in the hidden danger knowledge base and its coverage of different hidden danger categories. For new hidden dangers or non-significant hidden dangers not covered in the knowledge base, the existing hidden danger identification system often cannot give clear risk warnings, or only gives general suggestions such as "manual review is recommended". This will not only reduce the trust of factory managers in the hidden danger identification system, but may also lead to potential hidden dangers not being handled in a timely manner, increasing the risk of accidents.

[0054] Therefore, an embodiment provided by the present invention is: an emergency fire hazard investigation method based on multimodal AI large model recognition technology, which utilizes the factory's existing hazard report documents, equipment operation specifications, fire safety rules and regulations, and relevant laws and regulations to generate diagnoses and suggestions corresponding to the hazards from multimodal data, thereby further improving the accuracy and adaptability of hazard identification;

[0055] Combine the following Figure 1-5 This embodiment is described in detail, including the following steps:

[0056] Step 1: Data Processing

[0057] Collect historical hidden danger inspection data, including hidden danger report documents, equipment operation specifications, industry standards and relevant laws and regulations, text data and image data, as multimodal data, organize the data uniformly and mark the safety hidden dangers of corresponding hidden danger levels. By using the actual recorded data in the hidden danger inspection process, the big model can better solve the problems in the actual inspection scenario and better fit the real use scenario.

[0058] Specifically, the image data mainly comes from: obtaining images and videos from surveillance cameras, mobile devices (such as smartphones and tablets), drones and other devices, capturing scenes related to safety hazards, such as equipment failure, unsafe working conditions, blocked fire passages, etc., adding corresponding hazard descriptions and hazard level labels to the safety hazards in the images, and each picture or video clip should contain an accurate timestamp to track the time of the event. If possible, geotags can also be added to determine the location where the hazard occurred.

[0059] Text data can come from hidden danger report documents, checklists, maintenance logs, feedback from inspectors, accident investigation reports, etc. Text data should be converted into structured electronic data as much as possible to facilitate machine reading and processing;

[0060] It should be noted that the more important sources of text data for laws and regulations are national and local safety standards, industry guidelines, internal corporate safety policies, etc. It is necessary to keep the latest versions of laws, regulations and rules and regulations, and record the history of any updates or revisions, and organize the text data of laws and regulations into csv format, with one law as one cell.

[0061] Image data annotation includes:

[0062] Object detection: Label objects in images, such as equipment parts, workers, protective equipment, etc. For videos, labeling can be performed by frame or time period.

[0063] Scene understanding: Identify and annotate specific scenes shown in images or videos, such as workshop layout, hazardous areas, etc.

[0064] Behavior recognition: For video data, people’s behaviors can also be annotated, such as not wearing a helmet, illegal operation, etc.

[0065] Text data annotation includes:

[0066] Entity recognition: marking key entities in text, such as names of people, places, equipment names, time, quantity, etc.;

[0067] Relationship extraction: find the relationship between entities in the text, such as cause and effect, sequence, etc.

[0068] Sentiment analysis: assessing the sentiment expressed in text, especially when it comes to prosecutorial feedback;

[0069] Legal and regulatory data annotation includes

[0070] Clause analysis: Analyze laws, regulations and rules and regulations one by one to clarify their scope of application and requirements.

[0071] Compliance checkpoints: Identify specific clauses that can be used directly for potential hazard detection and rectification suggestions.

[0072] Association mapping: Establish a mapping relationship between regulatory provisions and actual hidden dangers to help quickly locate relevant regulations.

[0073] The second step is adaptive processing of image processing

[0074] An adaptive image coding mechanism is added to the image data to directly process images of various resolutions and aspect ratios. Traditional image processing models usually require the input image to have a fixed resolution and aspect ratio, or the image needs to be scaled and cropped before processing. This will lead to information loss or deformation, especially when the image content is relatively complex or important details are located at the edge of the image. In order to support the input of images with arbitrary resolutions and aspect ratios and ensure that the high-dimensional features of the image are not lost, the present invention designs an adaptive image coding mechanism, which can flexibly adapt to image data in various practical scenarios without human intervention. This enables the system to process images or videos from various sources, greatly improving the versatility and adaptability of the model. This mechanism can directly process images of various resolutions and aspect ratios, avoiding unnecessary image deformation or loss of important details, thereby improving the accurate recognition of image details.

[0075] Image and video data in the real world are highly uncertain and may come from different cameras, devices or sensors in different formats. Supporting inputs of multiple resolutions and aspect ratios allows the model to flexibly respond to different data sources, thereby improving the reliability and robustness of the model in various scenarios, ensuring that accurate safety hazard analysis can be provided in various complex scenarios. Users do not need to care about the size, resolution and aspect ratio of the image in actual operation, which greatly simplifies the usage process. Whether uploading high-resolution HD pictures or low-quality surveillance videos, the system can automatically adapt and perform effective analysis. This design improves the user experience, especially for non-professional users or staff who do not have image processing skills, greatly reducing the usage threshold.

[0076] It is worth mentioning that, in this embodiment, the specific processing method of the adaptive image coding mechanism for the picture is:

[0077] First, the image is scaled to the specified size, encoded using VIT, and the encoded images are merged and flattened; secondly, the image is divided into several parts according to the set specification size, encoded separately using VIT, and the encoded images are merged and flattened; finally, the two flattened encoded images are vectorized and spliced ​​to obtain the final image encoding.

[0078] Images of various resolutions and aspect ratios can be processed directly, avoiding unnecessary image deformation or loss of important details, thereby improving the accurate recognition of image details. In reality, the resolution and aspect ratio of images vary, especially in images taken by cameras, mobile devices, surveillance cameras, drones and other devices, the image quality and ratio vary greatly. The present invention can support images from a variety of different sources, avoiding the forced unification of images in the preprocessing link, so that images taken by different devices can be efficiently processed, enhancing the applicability of the system, and being able to process security monitoring images or videos from different sources and formats. Traditional methods usually require a large number of preprocessing steps, such as unifying the size, cropping or scaling, which not only increase the workload, but also may cause the loss or deformation of image information. Supporting image input of any resolution and aspect ratio greatly simplifies the process, reduces the manual processing of data, and makes the system more efficient and automated during the processing process.

[0079] Step 3: Fine-tune the large model using LoRa

[0080] Using the collected and processed data, the large model is fine-tuned using LoRa, making it more suitable for emergency firefighting. LoRa is a parameter-efficient fine-tuning method that adapts to specific tasks by adjusting only a small number of parameters in the model (i.e., low-rank matrices), thereby greatly reducing the computing resources and time required for training.

[0081] First, the LoRa layer is introduced: the LoRa layer is introduced in each layer of the basic model. The LoRa layer consists of two low-rank matrices, which are used to adjust the row space and column space of the original weight matrix respectively. In this way, it can adapt to new tasks by learning a small number of parameters without changing the original weights.

[0082] Set hyperparameters: Determine the rank of the LoRa layer. Usually, a smaller rank (such as 4 or 8) can achieve better results. You can also set other training parameters such as learning rate and batch size.

[0083] Freeze most parameters: Except for the LoRa layer, freeze all other parameters in the base model and only update the parameters in the LoRa layer, which can significantly reduce the computing resources and time required for training.

[0084] The second is to design specific tasks:

[0085] Classification tasks: For example, identifying different types of safety hazards or emergency response levels based on descriptions.

[0086] Question and answer system: Develop a question and answer system that can answer questions about emergency procedures, laws and regulations, etc.

[0087] Text generation: Train models to generate detailed emergency response plans, incident reports, or training content.

[0088] Multimodal tasks: Combining image and text data enables the model to understand and analyze the relationship between visual information (such as photos of the fire scene) and text information (such as accident descriptions).

[0089] Finally, training and optimization

[0090] Training process: Use the prepared data set to fine-tune the model. Since LoRa only needs to train a small number of parameters, the training speed is fast and not prone to overfitting.

[0091] Monitoring indicators: Pay close attention to evaluation indicators such as loss function, accuracy, F1 score, etc. during training to ensure continuous improvement of model performance.

[0092] Validation and testing: Evaluate the model’s performance on independent validation and test sets to ensure its generalization ability.

[0093] By combining multimodal data such as images, text, and laws and regulations, the large model fine-tuned by LoRA can understand different types of information at the same time. For example, images can provide visual clues to hidden dangers, text data (such as safety inspection reports) can provide detailed descriptions and background information, and laws and regulations provide the model with a standard basis for legality and compliance. This multimodal data fusion enables the model to perform more comprehensive and accurate analysis. Traditional methods usually rely on only a single modality (such as using only text or images for analysis), which may ignore important information in the image or text. Multimodal learning allows the model to better understand different types of data and improves the effectiveness of the model in actual tasks. Through multimodal learning, the model can more comprehensively capture potential hidden dangers in images and compare them with relevant texts and regulations to derive more accurate rectification suggestions, efficiently integrate these heterogeneous knowledge sources, and form a more comprehensive judgment system;

[0094] One advantage of LoRA fine-tuning is that it can effectively reduce the need for labeled data. By utilizing a pre-trained large model, the LoRA fine-tuning method can quickly improve model performance based on a small amount of labeled data. Especially in applications for hidden danger detection, due to the large amount and diversity of data, the workload of fully labeling all data is very large, and LoRA fine-tuning makes this process more efficient.

[0095] Step 4: Build a knowledge base

[0096] The text data in the historical hidden danger investigation data is converted into numerical vector representation based on the deep learning embedding model, and this knowledge is stored in the knowledge base in the form of vectors, making the retrieval process more efficient;

[0097] In this embodiment, the specific method of converting the text data in the historical potential hazard investigation data into a numerical vector representation based on a deep learning embedding model includes the following steps:

[0098] First is text preprocessing. The structured text data is segmented into meaningful text blocks, which can be words or sub-word units. Common stop words (such as "de", "shi", etc.) are removed, as these words usually contribute less to semantics.

[0099] Secondly, the text blocks are restored to their basic forms (such as changing verbs back to their original forms) to reduce the lexical diversity, and at the same time, corresponding part-of-speech tags are added, which helps to understand the text structure more precisely.

[0100] Finally, the processed text blocks are converted into word vectors using a word embedding model: piccolo-large-zh, and stored in a vector number knowledge base. Common vector databases include Faiss, Annoy, HNSWLib, etc., which support fast approximate nearest neighbor search (ANN). Here we choose Faiss and create an index for the vector database to accelerate the retrieval process. The index can select different algorithms according to the application scenario, such as tree structures, hash tables, inverted indexes, etc.

[0101] The model can quickly find the regulations, standards, or inspection histories most relevant to the current potential hazard or problem from the knowledge base through similarity measurement in the vector space. The vectorized knowledge base uses a digital vector storage method and has high scalability. As laws, regulations, and industry standards increase, the knowledge base can be easily expanded without affecting the retrieval efficiency or model performance. The model can adapt to more regulatory data and adjust decision-making strategies according to the newly added information. Through the vectorized knowledge base, the model can generate rectification suggestions with legal basis for each potential hazard, ensuring that each potential hazard investigation is based on the same standard, avoiding the differences in human judgment, and ensuring the consistency and fairness of the application of regulations.

[0102] It is also worth mentioning that in this embodiment, the retrieval matching of the above-mentioned knowledge base specifically includes the following methods:

[0103] When a user enters a query, first convert it into the same vector representation as the knowledge base, search for the document vector most similar to the query vector in the vector database. The distance measurement methods used include cosine similarity, Euclidean distance, etc. Sort the retrieved results from largest to smallest according to the similarity score and return the most relevant documents or paragraphs.

[0104] Step 5: Potential hazard inference and retrieval

[0105] Design a multi-hop prompt template to perform multi-hop reasoning on hidden dangers in the input image or video. Guide the multi-dimensional analysis of hidden danger factors through a staged chain of thinking with different attention points to details. Comprehensively consider the information of each dimension, retrieve content similar to the hidden danger reasoning results in the knowledge base, re-examine the hidden danger reasoning results from multiple angles, verify the rationality of the reasoning results, and rely on similar content in the knowledge base for self-correction, and re-examine the hidden danger reasoning results from multiple angles;

[0106] The specific method is as follows:

[0107] 1. Identify the types of problems. For emergency firefighting, these may include but are not limited to the following types of problems:

[0108] Hazard identification: Identify potential safety hazards based on the description;

[0109] Regulatory Lookup: Find laws, regulations or industry standards relevant to a specific situation.

[0110] 2. Build a multi-hop prompt framework: The multi-hop prompt framework includes the following key parts:

[0111] Problem statement: Clearly describe the problem that needs to be solved;

[0112] Initial information: Provide necessary background information of the image content;

[0113] Reasoning steps: List the descriptions of the corresponding areas in the image content step by step according to the image content, and each step is equipped with corresponding logical basis;

[0114] Auxiliary information: introduce additional knowledge or data support when necessary;

[0115] Reasoning results: Output the reasoning results of each step and give a clear answer.

[0116] 3. Design a multi-hop prompt template

[0117] #Multi-hop prompt template

[0118] ##Problem Statement

[0119] [Briefly describe the problem to be solved]

[0120] ## Initial Information

[0121] [Provide necessary background information, known conditions or context]

[0122] ###Inference steps

[0123] ####Step 1: [Description of the first step]

[0124] -**Operation**: [specific operation or query]

[0125] -**Result**: [Record the result of the first step]

[0126] ####Step 2: [Description of the second step]

[0127] -**Operation**: [specific operation or query]

[0128] -**Result**: [Record the result of step 2] ...

[0130] ####Step N: [Description of step N]

[0131] -**Operation**: [specific operation or query]

[0132] -**Result**: [Record the result of step N]

[0133] ##Auxiliary Information

[0134] [Introduce additional knowledge, data or references]

[0135] ##Inference Results

[0136] [Enter the content description of each step into the knowledge base to retrieve the corresponding safety hazards and rectification suggestions]

[0137] 4. Optimization and expansion

[0138] Dynamic adjustment: Based on actual application scenarios and user feedback, the prompt template is continuously optimized, new reasoning steps are added, or existing steps are adjusted.

[0139] Automatic generation: Develop tools or scripts to automatically generate multi-hop prompts to reduce the workload of manual writing.

[0140] Knowledge graph: Combined with knowledge graph technology, relevant regulations, standards and cases are represented in a structured manner to enhance the accuracy and depth of reasoning.

[0141] User interaction: Allowing users to input additional information or ask questions during the inference process allows the model to adjust and improve based on new information.

[0142] It is worth mentioning that, in this embodiment, after the prompt framework template of multi-hop reasoning is completed, the multi-hop reasoning and retrieval of hidden dangers in the image data specifically includes the following steps:

[0143] Identify the content in the image data, and describe the predicted hidden danger content in multiple dimensions and steps based on the identified image content and the constructed multi-hop prompt framework;

[0144] Input the predicted hidden dangers described into the knowledge base for retrieval to determine the laws and regulations corresponding to the safety hazards;

[0145] The generated safety hazards are input into the knowledge base for similarity matching. The step-by-step description of the predicted hazard content and the corresponding safety hazard text blocks retrieved from the knowledge base are combined and encoded through the CrossEncoder model. The model generates a relevance score based on this pair of combined inputs, sorts them according to the model's score, and outputs the retrieval content with the highest score as the final result, thereby obtaining the laws and regulations corresponding to each safety hazard in the image data;

[0146] Finally, professional content is obtained from the knowledge base, and corresponding rectification opinions are generated for the identified safety hazards based on the corresponding laws and regulations.

[0147] It should be noted that, in this embodiment, a noise filtering mechanism is set in the retrieval and matching process of the knowledge base, and noise filtering is performed on text data and image data before retrieval, and only high-quality data is retained for indexing and storage. After the user submits a query, the retrieval results are dynamically filtered to remove irrelevant information, and the noise filtering mechanism is continuously optimized according to user feedback;

[0148] The noise filtering mechanism can effectively identify and exclude content that is irrelevant or of low quality to the current reasoning task, thereby ensuring that the model only receives valuable knowledge. For example, for the task of safety hazard investigation, noise filtering can exclude regulations or standards that are not relevant to specific hazard scenarios, making the retrieval results more in line with the task requirements. Especially in complex tasks (such as hazard detection), incorrect knowledge points may cause the model to produce misleading reasoning results, which in turn affects the final decision. Through noise filtering, it can ensure that the model obtains more accurate and high-quality inputs and improve the reliability of reasoning results. Through the noise filtering mechanism, the amount of irrelevant information input to the model can be reduced, thereby reducing the computational burden of subsequent reasoning stages. When the data input to the large model is more streamlined and relevant, the model can focus on key information during processing and improve reasoning efficiency. The noise filtering mechanism can ensure that in cross-domain applications, the model is not disturbed by noise information between different domains. This is especially important for the fusion of multimodal data (such as images, text, videos, etc.), because different types of data may contain different noise sources. Through noise filtering, the system can ensure that cross-domain information fusion is more efficient and accurate.

[0149] Step 6: Generate a response. Finally, based on the results of multi-hop reasoning and the content of the knowledge base retrieval, the large model is given context to generate safety hazards, the legal basis for the corresponding safety hazards, and the corresponding safety hazard rectification opinions:

[0150] The following is the image data captured by all inspectors through their mobile phones in Area A of a production workshop at 15:00 on December 4, 2024. The content is: there is a large combustion boiler in the workshop, there is a small amount of coal slag on the ground, and the surrounding environment is messy.

[0151] Therefore, a multi-hop prompt template is designed to perform multi-hop reasoning on the hidden dangers of the above image data.

[0152] #Hazard Identification

[0153] ##Problem Statement

[0154] Based on the site description provided, identify potential safety hazards, including but not limited to in-depth analysis of equipment, materials, environment, behavior, etc.

[0155] ## Initial Information

[0156] - On-site description: There is a large combustion boiler in the workshop, with a small amount of coal slag on the ground and a messy surrounding environment.

[0157] - Time: 15:00, December 4, 2024

[0158] -Location: Production Workshop Area A

[0159] - **Time Alignment**: Ensure that timestamps of all sensor data are synchronized.

[0160] ####Step 1: Analyze images or videos using a fine-tuned multimodal large model

[0161] -**Operation**: Identify environmental information, equipment information, behavior information, material information, and safety information in pictures or videos.

[0162] -**Result**: Recognition results of production pictures or videos, Environment: dim workshop environment; Equipment: a combustion boiler; Behavior: no human behavior; Materials: cluttered; Safety: no safety warning information was found.

[0163] ####Step 2: Based on the information generated in the first step, identify the security risks in the image or video from multiple dimensions

[0164] -**Operation**: Based on the environment, equipment, personnel behavior, materials, etc. generated in the first step, determine whether there are any safety hazards in the diagram.

[0165] -**Result**: Some safety hazards were found. Environment: lack of proper lighting in the surrounding area; Equipment: obvious corrosion and wear on the equipment surface; Behavior: no human behavior; Materials: coal slag randomly piled on the ground; Safety: lack of safety warning information.

[0166] ####Step 3: Input the description of the potential safety hazard into the knowledge base for retrieval, filter out the irrelevant knowledge, and output the corresponding effective legal and regulatory basis:

[0167] 1. Environment: lack of proper lighting around;

[0168] 2. Equipment: There is obvious corrosion and wear on the surface of the equipment;

[0169] 3. Materials: There are coal slags randomly piled on the ground;

[0170] 4. Safety: lack of safety warning information;

[0171] The content described in the above image or video is then input into the knowledge base for similarity matching;

[0172]

[0173]

[0174] 1. "Construction Safety Management Code 4.8.5", "Urban Road Maintenance Operation Code 12.0.8", "Technical Standards for Temporary Electricity Safety at Construction Sites of Buildings and Municipal Engineering Projects 9.1.1";

[0175] 2. "Standardization Specifications for Safety Production of Machinery Manufacturing Enterprises 4.2.17.3", "Article 6 of the Standards for Determining Major Accident Hazards in Industrial and Trade Enterprises", "Safety Requirements for Cupolas and Cupola Chargers 5.6.2";

[0176] 3. Article 16 of the Hangzhou City Dust Pollution Prevention and Control Management Measures, Article 6 of the Standards for Determining Major Accident Hazards in Industrial and Trade Enterprises, and 4.2.17.3 of the Safety Standardization Specifications for Machinery Manufacturing Enterprises;

[0177] 4. "Article 6 of the Standards for Determining Major Accident Hazards in Industrial and Trade Enterprises" and "4.2.7 of the Standardization Specifications for Safety Production in Machinery Manufacturing Enterprises";

[0178] ####Step 4: Obtain professional knowledge from the knowledge base and determine the legal and regulatory basis for safety hazards.

[0179] -**Operation**: Check the content of relevant laws and regulations, confirm the legal basis for common safety hazards, and infer other possible safety hazards.

[0180] -**Result**: The security risks finally found are:

[0181] 1. According to the requirements of 4.8.5 of the "Code for Safety Management of Construction", the lack of proper lighting around the image is a safety hazard.

[0182] 2. According to the requirements of 4.2.17.3 of the "Standardized Specifications for Safety Production of Mechanical Manufacturing Enterprises", the boiler equipment in the image has obvious corrosion and wear on the surface, which is a safety hazard.

[0183] 3. According to Article 16 of the "Hangzhou Urban Dust Pollution Prevention and Control Management Measures", the randomly piled coal slag on the ground in the image poses a safety hazard.

[0184] 4. According to Article 6 of the “Standards for Determining Major Accident Hazards in Industrial and Commercial Enterprises”, the lack of safety signs in the image is a safety hazard.

[0185] ####Step 5: Reasoning in step 3, obtain professional knowledge from the knowledge base and generate corresponding rectification opinions for the identified safety hazards.

[0186] -**Operation**: Check the relevant laws and regulations and safety hazards, and generate rectification suggestions.

[0187] -**Result**: Generate rectification suggestions for each hidden danger:

[0188] ##Auxiliary Information

[0189] - "Construction Safety Management Code"

[0190] - "Standardization Specifications for Safety Production in Machinery Manufacturing Enterprises"

[0191] - "Hangzhou City Dust Pollution Prevention and Control Management Measures"

[0192] - "Standards for Determining Major Accident Hazards in Industrial and Trade Enterprises"

[0193] ##Inference results: Generate the final answer through the large model.

[0194] -**Operation**: Refer to the requirements of the corresponding data structure of the business system and generate the final answer content text based on the language generation capabilities of the large model.

[0195] ####Step 6: Generate the final answer:

[0196] 1. Identification results: There is obvious corrosion and wear on the surface of the furnace, especially around the furnace door.

[0197] Corrective suggestions: Please check the daily maintenance records of the boiler furnace structure to ensure that there are no safety hazards in the furnace structure.

[0198] Legal basis: 4.2.17.3 of the Standardization Specification for Safety Production in Machinery Manufacturing Enterprises.

[0199] 2. Identification result: There are coal slag and debris on the ground, which poses a risk of slipping or tripping.

[0200] Corrective suggestions: Please clean up the coal slag and debris on the ground in time to ensure that the ground is clean and tidy to prevent staff from slipping or tripping.

[0201] Legal basis: Article 16 of the Hangzhou Urban Dust Pollution Prevention and Control Management Measures.

[0202] 3. Identification results: There are no obvious safety signs or emergency stop devices.

[0203] Corrective suggestions: Please set up corresponding safety signs and equipment emergency stop devices as required.

[0204] Legal basis: Article 6 of the "Standards for Determining Major Accident Hazards in Industrial and Commercial Enterprises".

[0205] 4. Identification result: Lack of proper lighting around the furnace may affect operational safety.

[0206] Corrective suggestions: Please keep the surrounding light clean and not affect the safety of operation.

[0207] Legal basis: "Construction Safety Management Code" 4.8.5.

[0208] By retrieving content similar to the current reasoning results in the knowledge base, more relevant background information or case support can be provided to the large model. These similar knowledge points can be used as a reference for the model, allowing the model to make more professional and accurate judgments on the existing basis. The knowledge base contains information such as industry regulations, standards, and best practices related to hidden danger detection. The model can retrieve content similar to the reasoning results and combine it with domain-specific expertise for further analysis. This approach can enhance the sensitivity of the model in specific areas (such as safety hazard identification), enabling it to capture more details and potential hazards.

[0209] By introducing content similar to the current reasoning results, the model can re-examine the problem from multiple angles and verify whether the reasoning results are reasonable. This approach enhances the model's fault tolerance and enables it to correct potential misjudgments, especially when faced with ambiguous or uncertain information. The model can rely on similar content in the knowledge base to correct itself.

[0210] The above is only a specific implementation of the invention, but the protection scope of the invention is not limited to it. Any changes or substitutions that are not conceived through creative work should be included in the protection scope of the invention. Therefore, the protection scope of the invention should be based on the protection scope defined in the claims.

Claims

1. An emergency fire hazard investigation method based on multimodal AI large model recognition technology, characterized by: The following steps are involved: Step 1: Data processing: collect historical hidden danger inspection data including images as multimodal data, organize the data uniformly and label the safety hidden dangers according to the corresponding hidden danger levels; Step 2: Image processing: adding an adaptive image coding mechanism to the image data to directly process images of various resolutions and aspect ratios; Step 3: Fine-tune the large model using the LoRa method. Use the collected and processed data to introduce the LoRa layer into each layer of the base model, set the training parameters, and only update the parameters in the LoRa layer, so that the model can adapt to specific tasks more quickly. Step 4: Vectorize text data. Convert text data in historical hidden danger investigation data into numerical vector representation based on deep learning embedding model. Store this knowledge in the knowledge base in vector form, making the retrieval process more efficient. Step 5: Hidden danger reasoning and retrieval. Design a multi-hop prompt template to perform multi-hop reasoning on hidden dangers on the input image data. Guide the multi-dimensional analysis of hidden danger factors through a staged chain of thinking with different attention points. Comprehensively consider the information of each dimension, retrieve content similar to the hidden danger reasoning results in the knowledge base, and re-examine the hidden danger reasoning results from multiple angles. Step 6: Generate a response. Use the results of hidden danger reasoning and the content retrieved from the knowledge base as context to generate a response for the large model.

2. According to claim 1, a method for checking emergency fire hazards based on multimodal AI large model recognition technology is characterized by: The historical hidden danger inspection data in the step one also includes text data of hidden danger report documents, equipment operation specifications, industry standards and relevant laws and regulations. The text data is organized into structured electronic data, wherein the text data of the laws and regulations are organized into csv format, with each law as a cell. At the same time, the safety hazards existing in each image in the image data of the historical hidden danger inspection are labeled with hidden danger levels and hidden danger descriptions, and a mapping relationship between each law and regulation and the safety hazard is established.

3. According to claim 2, a method for checking emergency fire hazards based on multimodal AI large model recognition technology is characterized in that: The specific processing method of the picture by the adaptive image coding mechanism in step 2 includes the following steps: First, the image is scaled to the specified size, encoded using VIT, and the encoded images are merged and flattened; Secondly, the image is divided into several parts according to the set specifications and sizes, and VIT is used to encode them separately, and the encoded images are merged and flattened; Finally, the two flattened encoded images are vectorized and concatenated to obtain the final image encoding.

4. According to claim 3, a method for checking emergency fire hazards based on multimodal AI large model recognition technology is characterized in that: The specific method of converting the text data in the historical hidden danger investigation data into a numerical vector representation based on a deep learning embedding model in step 4 includes the following steps: S101: Segment structured text data into meaningful text blocks and remove common stop words; S102: restore the text block to its basic form, reduce the diversity of vocabulary, and add corresponding part-of-speech tags; S103: Convert the processed text block into a word vector using a word embedding model, store it in a vector data knowledge base, create an index for the knowledge base, and enable the knowledge base to perform retrieval and matching.

5. According to claim 4, a method for checking emergency fire hazards based on multimodal AI large model recognition technology is characterized in that: The retrieval matching of the knowledge base specifically includes the following methods: converting the user input query into a vector identical to the knowledge base, using a cosine similarity calculation method to search for a text data vector most similar to the query vector in the knowledge base, and sorting the retrieved results according to the similarity score.

6. According to claim 5, a method for checking emergency fire hazards based on multimodal AI large model recognition technology is characterized in that: The hidden danger reasoning and retrieval in step 5 specifically includes the following steps: S201: Identify the content in the image data, and describe the predicted hidden danger content in multiple dimensions and steps according to the identified image content and the constructed multi-hop prompt framework; S202: Input the predicted hidden danger content described into the knowledge base for similarity matching, output the content with the highest relevance score as the search content, and determine the laws and regulations corresponding to each safety hazard S203: Retrieve professional content from the knowledge base and generate corresponding rectification opinions for the identified safety hazards according to corresponding laws and regulations.

7. The method for checking emergency fire hazards based on multimodal AI large model recognition technology according to claim 6 is characterized in that: The specific method of inputting the predicted hidden danger content into the knowledge base for similarity matching is: The CrossEncoder model is used to combine the step-by-step description of the predicted hidden danger content with the corresponding safety hidden danger text blocks retrieved from the knowledge base for encoding. The model generates a relevance score based on this pair of combined inputs, sorts them according to the model's score, and outputs the text block with the highest score as the final result retrieval content.

8. The method for checking emergency fire hazards based on multimodal AI large model recognition technology according to claim 7 is characterized in that: A noise filtering mechanism is set up in the retrieval and matching process of the knowledge base. The specific method is: pre-processing the text data and image data before retrieval to remove irrelevant information, retaining only high-quality data for indexing and storage, and dynamically filtering the retrieval results to remove irrelevant information after the user submits the query, while continuously optimizing the noise filtering mechanism based on user feedback.

9. The method for checking emergency fire hazards based on multimodal AI large model recognition technology according to claim 8 is characterized in that: The content of the response generated in step six includes a description of the safety hazard, the laws and regulations corresponding to the safety hazard, and rectification suggestions corresponding to the safety hazard, and each rectification suggestion corresponds to the relevant laws and regulations.

Images